From 59d24e5e438186182f5caa0cfd0133370ecdd7ab Mon Sep 17 00:00:00 2001 From: mvdbeek Date: Mon, 4 May 2026 16:26:24 +0200 Subject: [PATCH] Add tool source storage admin and dev documentation - doc/source/admin/tool_source_storage.rst: operator-facing guide covering backend choice, populate_store usage, watch mode, and the per-conf composite story. - doc/source/dev/tool_source_storage.rst: developer-facing architecture overview of the store, LazyToolBox, and the batch endpoint integration. --- doc/source/admin/tool_source_storage.rst | 148 ++++++++----- doc/source/dev/tool_source_storage.rst | 261 +++++++++++++++-------- 2 files changed, 266 insertions(+), 143 deletions(-) diff --git a/doc/source/admin/tool_source_storage.rst b/doc/source/admin/tool_source_storage.rst index 59348a52f1f..cdda15b55c8 100644 --- a/doc/source/admin/tool_source_storage.rst +++ b/doc/source/admin/tool_source_storage.rst @@ -1,56 +1,74 @@ Tool Source Storage =================== +Galaxy can cache parsed tool sources to improve startup time and reduce memory usage +when serving batch API endpoints. This is especially useful for large Galaxy installations +with thousands of tools. + Overview -------- -By default, Galaxy parses every tool at startup and keeps all of them in -memory. For installations with many tools this: +By default, Galaxy loads all tools into memory at startup. For installations with many tools, +this can: -- Slows down Galaxy startup significantly -- Consumes large amounts of memory in every Galaxy process +- Slow down Galaxy startup significantly +- Consume large amounts of memory -Tool source storage addresses this by doing the parsing work once, ahead of -time: +The tool source storage system addresses these issues by: -1. Tool sources are pre-parsed (with macros expanded) and stored in a - configurable database backend -2. A lightweight index over the stored tools supports fast tool listings and - search without touching tool files - -A toolbox that consumes this store to load tools on demand is planned as -follow-up work; this document covers the store, the populator, and the index -that it will build on. +1. Pre-parsing and storing tool sources in a configurable backend +2. Maintaining a lightweight index in memory for fast API responses +3. Loading full Tool objects on-demand with LRU caching Configuration ------------- Tool source storage is configured in ``galaxy.yml``. The following options are available: -Default Store -^^^^^^^^^^^^^ +Backend Selection +^^^^^^^^^^^^^^^^^ .. code-block:: yaml galaxy: - # SQLAlchemy URI for storing tool sources. - tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite + # Backend for storing tool sources: 'database' or 'sqlalchemy' + tool_source_store: database -The store lives in a standalone database - a SQLite file under -``/tool_sources.sqlite`` by default - separate from Galaxy's main -database. It is a rebuildable cache: it can be deleted at any time and -recreated by re-running the population script. +**Database Backend** (default) -Multi-host deployments must point every Galaxy process (web workers *and* -job handlers) at the same store — typically a SQLite file on a shared -filesystem: +Stores tool sources in the Galaxy database. Best for: + +- Single-server deployments +- Installations where tools don't change frequently +- Simplest setup (no additional infrastructure) + +**SQLAlchemy Backend** + +Stores tool sources in a separate SQLAlchemy-managed database (typically a +SQLite file). Useful for shipping read-only tool source bundles via per-conf +``tool_source_stores`` entries (see CVMFS recipe below). .. code-block:: yaml galaxy: - tool_source_database_connection: sqlite:////shared/galaxy/tool_sources.sqlite + tool_source_store: sqlalchemy + tool_source_disk_path: /path/to/tool_sources.sqlite # SQLite shortcut -Any other SQLAlchemy-supported database (e.g. PostgreSQL) works as well. +Toolbox Selection +^^^^^^^^^^^^^^^^^ + +.. code-block:: yaml + + galaxy: + # Opt in to the LazyToolBox. Off by default; setting this to true is + # required to activate per-conf store="..." routing. + use_lazy_toolbox: true + +The LazyToolBox is opt-in: leave ``use_lazy_toolbox`` unset (or false) and +Galaxy uses the traditional eager ToolBox even when the store is populated +or when a tool_conf carries a ``store="..."`` attribute. Set +``use_lazy_toolbox: true`` to activate lazy loading and per-conf store +routing. Per-conf Store Routing (CVMFS Recipe) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -61,21 +79,24 @@ read-only SQLite bundle on CVMFS alongside a tool_conf, so worker processes can resolve every tool in that conf with local-cached lookups instead of one network round-trip per JSON file. -Declare the named stores under the top-level ``tool_source_stores`` key in -``galaxy.yml``. Each entry takes a SQLAlchemy ``url`` and optional -``read_only`` flag. SQLite is the typical choice for CVMFS bundles (single -self-contained file), but any SQLAlchemy-supported database works: +Declare the named stores under the new top-level ``tool_source_stores`` +key in ``galaxy.yml``. The ``sqlalchemy`` backend takes either a SQLAlchemy +``url`` or a ``path`` shortcut that builds a SQLite URL. SQLite is the +typical choice for CVMFS bundles (single self-contained file), but any +SQLAlchemy-supported database works: .. code-block:: yaml galaxy: - tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite + tool_source_store: database # the writable default tool_source_stores: cvmfs_main: - url: sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true + backend: sqlalchemy + path: /cvmfs/example.org/tools/sources.sqlite read_only: true site_shared: - url: sqlite:///file:/shared/galaxy/tool_sources.sqlite?mode=ro&uri=true + backend: sqlalchemy + url: postgresql://galaxy_ro@db.example.org/tool_sources read_only: true Then point the tool_conf at it via the root element's ``store`` attribute @@ -110,9 +131,20 @@ it, ``populate_store.py`` populates **every writable store** referenced from a tool_conf in the same run. Once the bundle is in place on CVMFS (or any read-only mount), restart -Galaxy. The ``read_only: true`` flag prevents Galaxy from writing through that -store. For SQLite connection-level read-only, use ``mode=ro&uri=true`` in the -SQLite URI as shown above. +Galaxy. + +Cache Configuration +^^^^^^^^^^^^^^^^^^^ + +.. code-block:: yaml + + galaxy: + # Maximum Tool objects in the LazyToolBox LRU cache (default: 500) + lazy_toolbox_cache_size: 500 + +The ``lazy_toolbox_cache_size`` determines how many fully-loaded Tool objects +are kept in memory by the LazyToolBox. A typical Galaxy installation has +500-2000 tools. If you frequently use many different tools, increase this value. Populating the Tool Source Store -------------------------------- @@ -127,14 +159,6 @@ Basic Usage $ python scripts/tool_source/populate_store.py --config /path/to/galaxy.yml -Deployments that install Galaxy from packages get the same command as the -``galaxy-populate-tool-source-store`` console script (shipped with the -``galaxy-app`` package), so no Galaxy source checkout is needed: - -.. code-block:: console - - $ galaxy-populate-tool-source-store --config /path/to/galaxy.yml - This will: 1. Discover tools from your tool configs (uses the same logic as Galaxy startup) @@ -164,6 +188,7 @@ Command Line Options --verbose, -v Verbose output --watch, -w Watch tool directories and send reload notifications --watch-polling Use polling observer (for NFS/CVMFS/network FS) + --debounce SECS Debounce time for watch mode (default: 2.0) Examples ^^^^^^^^ @@ -200,10 +225,10 @@ script on a schedule: Watch Mode (Live Updates) ^^^^^^^^^^^^^^^^^^^^^^^^^ -As an alternative to cron, you can run the population script in watch mode to -keep the store continuously up to date. This uses ``watchdog`` to monitor tool -directories for changes and automatically updates the store, then sends a -notification via Kombu to trigger cache reloads in all Galaxy processes. +For development environments or installations where tools change frequently, you can run +the population script in watch mode. This uses ``watchdog`` to monitor tool directories +for changes and automatically updates the store, then sends a notification via Kombu +to trigger cache reloads in all Galaxy processes. .. code-block:: console @@ -213,12 +238,13 @@ Watch mode options: - ``--watch, -w`` - Enable watch mode - ``--watch-polling`` - Use polling observer (required for network filesystems like NFS/CVMFS) +- ``--debounce SECS`` - Debounce time for file changes (default: 2.0 seconds) Example with polling for network filesystem: .. code-block:: console - $ python scripts/tool_source/populate_store.py -c galaxy.yml --watch --watch-polling + $ python scripts/tool_source/populate_store.py -c galaxy.yml --watch --watch-polling --debounce 5.0 **Requirements:** @@ -235,9 +261,9 @@ When a tool XML file changes, watch mode will: This is useful for: -- Installations using shared storage where tools may be updated externally -- CI/CD pipelines that deploy tool updates - Development environments where tools are being actively edited +- CI/CD pipelines that deploy tool updates +- Installations using shared storage where tools may be updated externally Troubleshooting --------------- @@ -253,17 +279,24 @@ Tools not appearing in the index 2. Check for parsing errors in the Galaxy log -Populating an existing installation ------------------------------------ +High memory usage +^^^^^^^^^^^^^^^^^ -To set up tool source storage on an existing Galaxy installation: +1. Reduce ``lazy_toolbox_cache_size`` to cache fewer Tool objects +2. Ensure ``use_lazy_toolbox: true`` is set in ``galaxy.yml`` + +Migration from Traditional Toolbox +---------------------------------- + +To migrate an existing Galaxy installation to use tool source storage: 1. Add the configuration to ``galaxy.yml``: .. code-block:: yaml galaxy: - tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite + tool_source_store: database + use_lazy_toolbox: true 2. Run the population script: @@ -272,3 +305,6 @@ To set up tool source storage on an existing Galaxy installation: $ python scripts/tool_source/populate_store.py -c /path/to/galaxy.yml 3. Restart Galaxy + +The traditional toolbox will continue to work as a fallback if the tool source +store is not populated or if a specific tool is not found in the store. diff --git a/doc/source/dev/tool_source_storage.rst b/doc/source/dev/tool_source_storage.rst index fc64e1cb2d5..575998e98c5 100644 --- a/doc/source/dev/tool_source_storage.rst +++ b/doc/source/dev/tool_source_storage.rst @@ -1,12 +1,9 @@ Tool Source Storage Architecture ================================ -This document describes the architecture of the tool source storage subsystem: -the store backends, the populator, and the index they build. For operator-facing -setup and configuration, see :doc:`/admin/tool_source_storage`. - -A toolbox that consumes this store to load tools on demand is planned as -follow-up work; the pieces documented here are the storage layer it will build on. +This document describes the architecture of the tool source storage subsystem +and the LazyToolBox. For operator-facing setup and configuration, see +:doc:`/admin/tool_source_storage`. Goals ----- @@ -15,38 +12,38 @@ The traditional ``ToolBox`` parses every tool XML at startup, builds full ``Tool`` objects, and keeps them all in memory. With thousands of tools that scales poorly: slow boot, large per-process RSS, and expensive worker reloads. -The tool source storage subsystem moves that parsing work out of the request -path: +The tool source storage subsystem moves that work out of the request path: - A separate process (``populate_store.py``) parses tools once and persists the canonical, macro-expanded source plus a lightweight metadata index. -- The store and index are laid out so a consumer can load only the index at - startup and materialize ``Tool`` objects on demand, instead of parsing the - full tree in-process. That consumer is the planned follow-up toolbox. +- Galaxy processes load only the index at startup and materialize ``Tool`` + objects on demand, with LRU eviction. +- Batch endpoints (``/api/tools``, ``/api/tools/tests_summary``, + ``/api/tool_panels`` …) answer from the index instead of iterating the + full toolbox. Module Layout ------------- :: - lib/galaxy/tools/source_store/ - __init__.py Public re-exports - interface.py ToolSourceStore ABC and StoredToolSource - factory.py Store construction from Galaxy configuration - sqlalchemy.py SqlAlchemyToolSourceStore (any SQLAlchemy URL) + lib/galaxy/tool_source_store/ + __init__.py ToolSourceStore ABC, StoredToolSource, build_tool_source_store() + database.py DatabaseToolSourceStore (uses tool_source + tool_index tables) + sqlalchemy.py SqlAlchemyToolSourceStore (any SA URL; sqlite shortcut) composite.py CompositeToolSourceStore (per-conf routing, merged index) index.py ToolIndex, ToolIndexEntry (the lightweight metadata) - search.py ToolWhooshIndex (Whoosh search index built from a ToolIndex) - discover.py discover_tools() — conf walk without booting a ToolBox - populator.py Population + watch logic (parse, store, index, broadcast) - freshness.py Optional external freshness probes - watcher.py Filesystem watch support + models.py Pydantic response models for the API layer - scripts/tool_source/populate_store.py Thin CLI wrapper over populator.main + lib/galaxy/tools/lazy_toolbox.py LazyToolBox (subclass of ToolBox) + lib/galaxy/tool_util/toolbox/ + base.py (small hook to support lazy mode) + lib/galaxy/tool_util/id_util.py Cheap tool-ID extraction (regex, no XML parser) -The same ``populator.main`` is registered as the -``galaxy-populate-tool-source-store`` console script in the ``galaxy-app`` -package metadata (``packages/app/pyproject.toml``). + lib/galaxy/webapps/galaxy/services/tools.py Batch endpoints (index-aware) + + scripts/tool_source/populate_store.py Population & watch script + scripts/tool_source/_discover.py Tool-file discovery (used by populate_store) Data Model ---------- @@ -55,37 +52,35 @@ Two persistence concepts: **StoredToolSource** — the canonical macro-expanded XML/YAML for a tool, keyed by SHA-256 of the expanded content. Multiple versions of the same -``tool_id`` coexist as separate hashes. The store keeps its own schema in a -standalone database (a SQLite file by default, any SQLAlchemy URL for shared -deployments) — deliberately outside Galaxy's database: the store is a -rebuildable cache and does not participate in Galaxy's migrations or session -lifecycle. +``tool_id`` coexist as separate hashes. The DB backend persists these in the +``tool_source`` table; the Redis and disk backends use their own layout. -**ToolIndex** — a Pydantic model containing one default ``ToolIndexEntry`` per tool -plus its versioned and panel-placement projections, -holding everything a store consumer needs (id, name, description, panel section, +**ToolIndex** — a single dataclass containing one ``ToolIndexEntry`` per tool, +holding everything the batch APIs need (id, name, description, panel section, labels, EDAM, requirements, container info, test counts, hidden/disabled, shed metadata). The index is serialized and gzip-compressed as a blob. -The schema is auto-created on first open; ``tool_index`` holds a single -row per index version. +The DB backend gets a new ``tool_index`` table (migration +``f5a73c8b9d12_add_tool_index_table``) with a single row per index version. +Redis stores the blob under a known key; disk stores it as a file. Backend Abstraction ------------------- -``ToolSourceStore`` (in ``tools/source_store/interface.py``) is an ABC defining: +``ToolSourceStore`` (in ``tool_source_store/__init__.py``) is an ABC defining: - ``store/get/exists/delete/list_all/get_by_tool_id/count`` — per-tool source operations, all keyed by content hash. - ``store_index/load_index/update_index_entry`` — index operations. - ``get_stats()`` — backend-specific stats (count, size, backend name). -``build_tool_source_store(config)`` is the only entry point used -by Galaxy. It builds the default store from -``config.tool_source_database_connection`` and uses the same SQLAlchemy-backed -store implementation for all configured URIs. ``ConfigurationError`` is raised -for missing required settings and is allowed to propagate up so -misconfiguration fails fast at startup. +``build_tool_source_store(app)`` is the only entry point used by Galaxy. It +inspects ``config.tool_source_store`` and lazily imports the chosen backend +so deployments only pay for the dependencies they use (no SQLAlchemy import +on a Redis-only deploy, no ``redis`` client on a DB-only deploy). +``ConfigurationError`` is raised for unknown backends or missing required +settings; it is allowed to propagate up so misconfiguration fails fast at +startup. The ABC defines a ``read_only: bool`` class attribute (default ``False``). ``ReadOnlyStoreError`` is raised by mutating methods of stores that opted @@ -100,8 +95,9 @@ or ``store: ...`` key (YAML), ``build_tool_source_store`` instantiates the referenced named stores from ``config.tool_source_stores`` and wraps them with the writable default in a :class:`CompositeToolSourceStore`. -The composite implements the same ``ToolSourceStore`` interface, so store -consumers stay completely unaware of the multi-store layout: +The composite implements the same ``ToolSourceStore`` interface, so the +LazyToolBox, services, and queue worker stay completely unaware of the +multi-store layout: - **Reads** iterate ``[per-conf members..., default]`` in order; first hit wins. ``count`` and ``list_all`` dedupe across members. @@ -123,12 +119,11 @@ case. The ``sqlalchemy`` backend (``sqlalchemy.py``) was added to make this useful for CVMFS: a single self-contained ``.sqlite`` file, opened with its own SQLAlchemy ``MetaData`` (independent of ``galaxy.model``) so the -file is portable, and openable with a SQLite URI such as -``sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true`` -for read-only mounts. Despite the name, the backend is not sqlite-specific - -pass any SQLAlchemy URL (Postgres, MySQL, ...). Auto schema creation runs on -first open; on remote backends operators may prefer to manage migrations -explicitly. +file is portable, and openable with ``mode=ro&uri=true`` for read-only +mounts. Despite the name, the backend is not sqlite-specific — pass any +SQLAlchemy ``url`` (Postgres, MySQL, …) instead of ``path``. Auto schema +creation runs on first open; on remote backends operators may prefer to +manage migrations explicitly. Per-conf populator routing ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -142,34 +137,65 @@ run; ``--target NAME`` restricts to a single store and raises read-only in default mode are silently skipped (the bundle is treated as authoritative for those entries). +LazyToolBox +----------- + +``LazyToolBox`` extends ``ToolBox`` rather than reimplementing it, so the rest +of Galaxy can keep using the same ``trans.app.toolbox`` interface. The key +override is ``__init__``: instead of calling ``_init_tools_from_configs`` and +loading every tool, it calls ``_init_lazy_toolbox`` which: + +1. Sets up the same internal dicts (``_tools_by_id``, ``_tools_by_uuid``, + etc.) that ``AbstractToolBox.__init__`` would. +2. Walks the tool conf files just enough to build the panel structure + (sections, labels) and a ``tool_id → (section_id, section_name)`` map. +3. Loads the ``ToolIndex`` from the store. If the store has tool sources but + no index, the index is rebuilt from the stored sources. +4. Populates ``_tools_by_id`` with stub entries from the index so + ``has_tool()`` and similar lookups work without touching the store. + +Full ``Tool`` objects are built on demand and kept in an ``LRUCache`` of +``lazy_toolbox_cache_size`` entries (default 500). Cache hits and misses are +guarded by an ``RLock`` for thread safety. + +The toolbox auto-enables: if the operator hasn't set ``use_lazy_toolbox``, +``galaxy.app._use_lazy_toolbox()`` enables the lazy implementation when the +configured store contains at least one tool. This means a fresh deploy +without the populator run keeps the eager behavior, and only switches once +``populate_store.py`` succeeds. + +Tool ID extraction +^^^^^^^^^^^^^^^^^^ + +``galaxy.tool_util.id_util`` provides ``extract_tool_id_from_xml`` and +``extract_tool_id_from_file``: regex-based ID lookup that reads only the +first ~2 KB of the XML. This avoids paying for full XML parsing during +panel-structure discovery, where we just need the ID to map a file entry +back to an index entry. + Discovery ---------- +^^^^^^^^^ -``galaxy.tools.source_store.discover.discover_tools`` walks tool config files -and yields ``DiscoveredTool`` records without booting a full ``ToolBox``. It is -used by: +``scripts.tool_source._discover.discover_tools`` walks tool config files +and yields ``DiscoveredTool`` records. It is used by: -- the populator to find tools to parse and store. -- watch mode to know which directories to monitor. -- callers that compare on-disk confs against the indexed tool set. +- ``populate_store.py`` to find tools to parse and store. +- ``populate_store.py --watch`` to know which directories to monitor. +- (Indirectly) the LazyToolBox panel-structure code path. -Pulling discovery out of ``ToolBox`` was deliberate: the populator must run -*without* a full app (or even a running Galaxy), and the watch mode must run in -a long-lived loop with no Galaxy process at all. +Pulling discovery out of ``ToolBox`` was deliberate: the population script +must run *without* a full app (or even a running Galaxy), and the watch +mode must run in a long-lived loop with no Galaxy process at all. Population Script ----------------- -``scripts/tool_source/populate_store.py`` is a thin CLI wrapper over -``galaxy.tools.source_store.populator.main``. It loads only the Galaxy -config and calls ``build_tool_source_store(config)``. Converter discovery builds -the datatypes registry, but the standalone process does not initialize the Galaxy -model. Tools are parsed in a +``scripts/tool_source/populate_store.py`` runs out of process. It builds a +minimal app context (datatypes registry + SQLAlchemy model + config) and +calls ``build_tool_source_store`` with that context. Tools are parsed in a ``ThreadPoolExecutor`` (``--parallel``, default 4 workers); each tool is -matched to its source path and carried forward when its raw file hash is unchanged -(``--incremental``, the default). Once the JSON index is committed the -populator rebuilds the Whoosh search index (``search.py``) so ranked tool -search stays in sync with the stored sources. +hashed and skipped if an entry with the same hash already exists +(``--incremental``, the default). Watch mode (``--watch``) uses ``watchdog`` to monitor every directory yielded by ``discover_tools``. File events are debounced (default 2 s), the changed @@ -178,23 +204,84 @@ files are re-parsed, the store is updated, and a single exchange. ``--watch-polling`` switches to ``PollingObserver`` for NFS/CVMFS/network filesystems where inotify is unreliable. -The broadcast is the populator's half of the contract: it publishes -``reload_tool_source_cache`` so peer processes can drop their stale index -view. The control-task handler that consumes the message lands with the -follow-up toolbox. +The control task handler lives in ``galaxy.queue_worker.reload_tool_source_cache`` +and is wired into the ``control_message_to_task`` map. Each Galaxy process +that receives the message: + +1. Calls ``LazyToolBox.invalidate_index_cache()`` (drops the in-memory + index reference so the next access reloads from the store). +2. Calls ``ToolSourceStore.invalidate_index_cache()`` on the store itself. + +Note that the LRU cache of fully constructed ``Tool`` objects is not +flushed by reload — only the index is invalidated. Stale ``Tool`` instances +are evicted naturally as new ones are loaded. + +Batch Endpoint Integration +-------------------------- + +``ToolsService`` (``services/tools.py``) checks at runtime whether +``trans.app.toolbox`` is a ``LazyToolBox`` (via ``_get_lazy_toolbox``) and +routes accordingly: + +- ``list_tools(in_panel=False)`` → ``index.list_all()`` mapped to API dicts. +- ``search_tools`` → ``index.search()`` returning IDs. +- ``get_tests_summary`` → ``index.get_tests_summary()``. +- ``get_all_requirements`` → ``index.get_all_requirements()``. +- ``get_panel_views`` → ``index.get_panel_views()`` for views, plus the + toolbox's own ``default_panel_view``. + +In every case there is a fallback path that iterates the eager toolbox, so +the same code works on deployments that have not opted into lazy mode. + +``ToolAPICache`` (``api_cache.py``) sits in front of the gzip-compressed JSON +representations of these batch responses, keyed by the canonical request +URL and TTL'd by ``tool_api_cache_ttl`` (default 300 s, ``0`` disables). + +API Layer +--------- + +``api/tool_sources.py`` exposes three FastAPI ``Router.cbv`` controllers: + +- ``ToolSourcesAPI`` — store inspection (admin-only). +- ``ToolIndexAPI`` — index browsing (mostly public, ``/stats`` is admin). +- ``ToolCacheAPI`` — LazyToolBox cache stats and clear (admin-only). + +Static suffix routes (``/stats``, ``/search``) deliberately precede the +``/{id}`` and ``/{hash}`` parameterized routes to avoid path shadowing — +this is enforced by route declaration order, not by FastAPI itself. + +``services/tool_sources.py`` is the thin service layer over the store and +the index. It depends on ``trans.app.tool_source_store``; the LazyToolBox +is only consulted for cache stats/clear. + +App Wiring +---------- + +``galaxy.app.UniverseApplication.__init__`` calls +``_init_tool_source_store`` early and registers the result as a singleton +under ``ToolSourceStore``. The toolbox is then chosen based on +``_use_lazy_toolbox()`` (explicit config override, otherwise auto-detect). +The store is exposed as ``app.tool_source_store`` and is ``Optional`` only +to satisfy type checkers — in practice the build either succeeds or raises +``ConfigurationError``. Design Notes ------------ -**Why a separate index instead of always querying the store?** A consumer -needs O(N) access to N entries; doing that against the backing store on every -request is a latency hit. Keeping the index in-process and only paying for -invalidation on reload is the better tradeoff. +**Why a separate index instead of always-querying-the-store?** Batch +endpoints need O(N) access to N entries; doing that against Redis or the +disk is a latency hit on every request. Keeping the index in-process and +only paying for invalidation on reload is the better tradeoff. **Why an out-of-process populator?** Parsing tools and computing macro expansions is expensive and shouldn't block worker startup. Keeping the populator separate also lets it run on a single host while many web workers -share the resulting store. +share the resulting store (especially with the Redis backend). + +**Why subclass ToolBox instead of building a parallel hierarchy?** +``trans.app.toolbox`` is referenced from hundreds of call sites that expect +the full ToolBox interface. Subclassing keeps the Liskov-substitution +property and lets unmodified callers benefit from lazy loading transparently. **Why hash-keyed storage?** Content-addressed storage gives us cheap deduplication across versions and shed installations, and idempotent @@ -204,12 +291,12 @@ effectively a no-op. Testing ------- -- Store unit tests: ``test/unit/app/tools/source_store/`` exercises each backend - through the ``ToolSourceStore`` interface (``test_stores.py``, - ``test_sqlite_store.py``, ``test_composite_store.py``, - ``test_index_versions.py``). -- Populator/discovery unit tests: ``test/unit/scripts/tool_source/`` - (``test_populate_store.py``, ``test_discover.py``, - ``test_build_index_entry.py``, ``test_whoosh_dir.py``). These use fakes - (not mocks) of ``ToolSourceStore`` so behavior is verified against the real - interface. +- Unit tests: ``test/unit/tool_source_store/test_stores.py`` exercises each + backend through the ``ToolSourceStore`` interface. +- Integration tests: ``test/integration/test_tool_source_storage.py`` spins + up Galaxy with each backend and verifies end-to-end behavior. +- Populator tests: ``test/unit/scripts/tool_source/test_populate_store.py`` + uses fakes (not mocks) of ``ToolSourceStore`` so behavior is verified + against the real interface. +- API tests: ``lib/galaxy_test/api/test_tool_sources.py``. +- Benchmarks: ``python -m galaxy.tool_source_store.benchmarks --iterations 100``.