mirror of
https://github.com/galaxyproject/galaxy.git
synced 2026-09-24 16:30:27 +08:00
Add tool source storage admin and dev documentation
- doc/source/admin/tool_source_storage.rst: operator-facing guide covering backend choice, populate_store usage, watch mode, and the per-conf composite story. - doc/source/dev/tool_source_storage.rst: developer-facing architecture overview of the store, LazyToolBox, and the batch endpoint integration.
This commit is contained in:
mvdbeek
@@ -1,56 +1,74 @@
|
||||
Tool Source Storage
|
||||
===================
|
||||
|
||||
Galaxy can cache parsed tool sources to improve startup time and reduce memory usage
|
||||
when serving batch API endpoints. This is especially useful for large Galaxy installations
|
||||
with thousands of tools.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
By default, Galaxy parses every tool at startup and keeps all of them in
|
||||
memory. For installations with many tools this:
|
||||
By default, Galaxy loads all tools into memory at startup. For installations with many tools,
|
||||
this can:
|
||||
|
||||
- Slows down Galaxy startup significantly
|
||||
- Consumes large amounts of memory in every Galaxy process
|
||||
- Slow down Galaxy startup significantly
|
||||
- Consume large amounts of memory
|
||||
|
||||
Tool source storage addresses this by doing the parsing work once, ahead of
|
||||
time:
|
||||
The tool source storage system addresses these issues by:
|
||||
|
||||
1. Tool sources are pre-parsed (with macros expanded) and stored in a
|
||||
configurable database backend
|
||||
2. A lightweight index over the stored tools supports fast tool listings and
|
||||
search without touching tool files
|
||||
|
||||
A toolbox that consumes this store to load tools on demand is planned as
|
||||
follow-up work; this document covers the store, the populator, and the index
|
||||
that it will build on.
|
||||
1. Pre-parsing and storing tool sources in a configurable backend
|
||||
2. Maintaining a lightweight index in memory for fast API responses
|
||||
3. Loading full Tool objects on-demand with LRU caching
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Tool source storage is configured in ``galaxy.yml``. The following options are available:
|
||||
|
||||
Default Store
|
||||
^^^^^^^^^^^^^
|
||||
Backend Selection
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
galaxy:
|
||||
# SQLAlchemy URI for storing tool sources.
|
||||
tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite
|
||||
# Backend for storing tool sources: 'database' or 'sqlalchemy'
|
||||
tool_source_store: database
|
||||
|
||||
The store lives in a standalone database - a SQLite file under
|
||||
``<data_dir>/tool_sources.sqlite`` by default - separate from Galaxy's main
|
||||
database. It is a rebuildable cache: it can be deleted at any time and
|
||||
recreated by re-running the population script.
|
||||
**Database Backend** (default)
|
||||
|
||||
Multi-host deployments must point every Galaxy process (web workers *and*
|
||||
job handlers) at the same store — typically a SQLite file on a shared
|
||||
filesystem:
|
||||
Stores tool sources in the Galaxy database. Best for:
|
||||
|
||||
- Single-server deployments
|
||||
- Installations where tools don't change frequently
|
||||
- Simplest setup (no additional infrastructure)
|
||||
|
||||
**SQLAlchemy Backend**
|
||||
|
||||
Stores tool sources in a separate SQLAlchemy-managed database (typically a
|
||||
SQLite file). Useful for shipping read-only tool source bundles via per-conf
|
||||
``tool_source_stores`` entries (see CVMFS recipe below).
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
galaxy:
|
||||
tool_source_database_connection: sqlite:////shared/galaxy/tool_sources.sqlite
|
||||
tool_source_store: sqlalchemy
|
||||
tool_source_disk_path: /path/to/tool_sources.sqlite # SQLite shortcut
|
||||
|
||||
Any other SQLAlchemy-supported database (e.g. PostgreSQL) works as well.
|
||||
Toolbox Selection
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
galaxy:
|
||||
# Opt in to the LazyToolBox. Off by default; setting this to true is
|
||||
# required to activate per-conf store="..." routing.
|
||||
use_lazy_toolbox: true
|
||||
|
||||
The LazyToolBox is opt-in: leave ``use_lazy_toolbox`` unset (or false) and
|
||||
Galaxy uses the traditional eager ToolBox even when the store is populated
|
||||
or when a tool_conf carries a ``store="..."`` attribute. Set
|
||||
``use_lazy_toolbox: true`` to activate lazy loading and per-conf store
|
||||
routing.
|
||||
|
||||
Per-conf Store Routing (CVMFS Recipe)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@@ -61,21 +79,24 @@ read-only SQLite bundle on CVMFS alongside a tool_conf, so worker
|
||||
processes can resolve every tool in that conf with local-cached lookups
|
||||
instead of one network round-trip per JSON file.
|
||||
|
||||
Declare the named stores under the top-level ``tool_source_stores`` key in
|
||||
``galaxy.yml``. Each entry takes a SQLAlchemy ``url`` and optional
|
||||
``read_only`` flag. SQLite is the typical choice for CVMFS bundles (single
|
||||
self-contained file), but any SQLAlchemy-supported database works:
|
||||
Declare the named stores under the new top-level ``tool_source_stores``
|
||||
key in ``galaxy.yml``. The ``sqlalchemy`` backend takes either a SQLAlchemy
|
||||
``url`` or a ``path`` shortcut that builds a SQLite URL. SQLite is the
|
||||
typical choice for CVMFS bundles (single self-contained file), but any
|
||||
SQLAlchemy-supported database works:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
galaxy:
|
||||
tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite
|
||||
tool_source_store: database # the writable default
|
||||
tool_source_stores:
|
||||
cvmfs_main:
|
||||
url: sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true
|
||||
backend: sqlalchemy
|
||||
path: /cvmfs/example.org/tools/sources.sqlite
|
||||
read_only: true
|
||||
site_shared:
|
||||
url: sqlite:///file:/shared/galaxy/tool_sources.sqlite?mode=ro&uri=true
|
||||
backend: sqlalchemy
|
||||
url: postgresql://galaxy_ro@db.example.org/tool_sources
|
||||
read_only: true
|
||||
|
||||
Then point the tool_conf at it via the root element's ``store`` attribute
|
||||
@@ -110,9 +131,20 @@ it, ``populate_store.py`` populates **every writable store** referenced
|
||||
from a tool_conf in the same run.
|
||||
|
||||
Once the bundle is in place on CVMFS (or any read-only mount), restart
|
||||
Galaxy. The ``read_only: true`` flag prevents Galaxy from writing through that
|
||||
store. For SQLite connection-level read-only, use ``mode=ro&uri=true`` in the
|
||||
SQLite URI as shown above.
|
||||
Galaxy.
|
||||
|
||||
Cache Configuration
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
galaxy:
|
||||
# Maximum Tool objects in the LazyToolBox LRU cache (default: 500)
|
||||
lazy_toolbox_cache_size: 500
|
||||
|
||||
The ``lazy_toolbox_cache_size`` determines how many fully-loaded Tool objects
|
||||
are kept in memory by the LazyToolBox. A typical Galaxy installation has
|
||||
500-2000 tools. If you frequently use many different tools, increase this value.
|
||||
|
||||
Populating the Tool Source Store
|
||||
--------------------------------
|
||||
@@ -127,14 +159,6 @@ Basic Usage
|
||||
|
||||
$ python scripts/tool_source/populate_store.py --config /path/to/galaxy.yml
|
||||
|
||||
Deployments that install Galaxy from packages get the same command as the
|
||||
``galaxy-populate-tool-source-store`` console script (shipped with the
|
||||
``galaxy-app`` package), so no Galaxy source checkout is needed:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ galaxy-populate-tool-source-store --config /path/to/galaxy.yml
|
||||
|
||||
This will:
|
||||
|
||||
1. Discover tools from your tool configs (uses the same logic as Galaxy startup)
|
||||
@@ -164,6 +188,7 @@ Command Line Options
|
||||
--verbose, -v Verbose output
|
||||
--watch, -w Watch tool directories and send reload notifications
|
||||
--watch-polling Use polling observer (for NFS/CVMFS/network FS)
|
||||
--debounce SECS Debounce time for watch mode (default: 2.0)
|
||||
|
||||
Examples
|
||||
^^^^^^^^
|
||||
@@ -200,10 +225,10 @@ script on a schedule:
|
||||
Watch Mode (Live Updates)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
As an alternative to cron, you can run the population script in watch mode to
|
||||
keep the store continuously up to date. This uses ``watchdog`` to monitor tool
|
||||
directories for changes and automatically updates the store, then sends a
|
||||
notification via Kombu to trigger cache reloads in all Galaxy processes.
|
||||
For development environments or installations where tools change frequently, you can run
|
||||
the population script in watch mode. This uses ``watchdog`` to monitor tool directories
|
||||
for changes and automatically updates the store, then sends a notification via Kombu
|
||||
to trigger cache reloads in all Galaxy processes.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
@@ -213,12 +238,13 @@ Watch mode options:
|
||||
|
||||
- ``--watch, -w`` - Enable watch mode
|
||||
- ``--watch-polling`` - Use polling observer (required for network filesystems like NFS/CVMFS)
|
||||
- ``--debounce SECS`` - Debounce time for file changes (default: 2.0 seconds)
|
||||
|
||||
Example with polling for network filesystem:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ python scripts/tool_source/populate_store.py -c galaxy.yml --watch --watch-polling
|
||||
$ python scripts/tool_source/populate_store.py -c galaxy.yml --watch --watch-polling --debounce 5.0
|
||||
|
||||
**Requirements:**
|
||||
|
||||
@@ -235,9 +261,9 @@ When a tool XML file changes, watch mode will:
|
||||
|
||||
This is useful for:
|
||||
|
||||
- Installations using shared storage where tools may be updated externally
|
||||
- CI/CD pipelines that deploy tool updates
|
||||
- Development environments where tools are being actively edited
|
||||
- CI/CD pipelines that deploy tool updates
|
||||
- Installations using shared storage where tools may be updated externally
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
@@ -253,17 +279,24 @@ Tools not appearing in the index
|
||||
|
||||
2. Check for parsing errors in the Galaxy log
|
||||
|
||||
Populating an existing installation
|
||||
-----------------------------------
|
||||
High memory usage
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
To set up tool source storage on an existing Galaxy installation:
|
||||
1. Reduce ``lazy_toolbox_cache_size`` to cache fewer Tool objects
|
||||
2. Ensure ``use_lazy_toolbox: true`` is set in ``galaxy.yml``
|
||||
|
||||
Migration from Traditional Toolbox
|
||||
----------------------------------
|
||||
|
||||
To migrate an existing Galaxy installation to use tool source storage:
|
||||
|
||||
1. Add the configuration to ``galaxy.yml``:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
galaxy:
|
||||
tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite
|
||||
tool_source_store: database
|
||||
use_lazy_toolbox: true
|
||||
|
||||
2. Run the population script:
|
||||
|
||||
@@ -272,3 +305,6 @@ To set up tool source storage on an existing Galaxy installation:
|
||||
$ python scripts/tool_source/populate_store.py -c /path/to/galaxy.yml
|
||||
|
||||
3. Restart Galaxy
|
||||
|
||||
The traditional toolbox will continue to work as a fallback if the tool source
|
||||
store is not populated or if a specific tool is not found in the store.
|
||||
|
||||
@@ -1,12 +1,9 @@
|
||||
Tool Source Storage Architecture
|
||||
================================
|
||||
|
||||
This document describes the architecture of the tool source storage subsystem:
|
||||
the store backends, the populator, and the index they build. For operator-facing
|
||||
setup and configuration, see :doc:`/admin/tool_source_storage`.
|
||||
|
||||
A toolbox that consumes this store to load tools on demand is planned as
|
||||
follow-up work; the pieces documented here are the storage layer it will build on.
|
||||
This document describes the architecture of the tool source storage subsystem
|
||||
and the LazyToolBox. For operator-facing setup and configuration, see
|
||||
:doc:`/admin/tool_source_storage`.
|
||||
|
||||
Goals
|
||||
-----
|
||||
@@ -15,38 +12,38 @@ The traditional ``ToolBox`` parses every tool XML at startup, builds full
|
||||
``Tool`` objects, and keeps them all in memory. With thousands of tools that
|
||||
scales poorly: slow boot, large per-process RSS, and expensive worker reloads.
|
||||
|
||||
The tool source storage subsystem moves that parsing work out of the request
|
||||
path:
|
||||
The tool source storage subsystem moves that work out of the request path:
|
||||
|
||||
- A separate process (``populate_store.py``) parses tools once and persists
|
||||
the canonical, macro-expanded source plus a lightweight metadata index.
|
||||
- The store and index are laid out so a consumer can load only the index at
|
||||
startup and materialize ``Tool`` objects on demand, instead of parsing the
|
||||
full tree in-process. That consumer is the planned follow-up toolbox.
|
||||
- Galaxy processes load only the index at startup and materialize ``Tool``
|
||||
objects on demand, with LRU eviction.
|
||||
- Batch endpoints (``/api/tools``, ``/api/tools/tests_summary``,
|
||||
``/api/tool_panels`` …) answer from the index instead of iterating the
|
||||
full toolbox.
|
||||
|
||||
Module Layout
|
||||
-------------
|
||||
|
||||
::
|
||||
|
||||
lib/galaxy/tools/source_store/
|
||||
__init__.py Public re-exports
|
||||
interface.py ToolSourceStore ABC and StoredToolSource
|
||||
factory.py Store construction from Galaxy configuration
|
||||
sqlalchemy.py SqlAlchemyToolSourceStore (any SQLAlchemy URL)
|
||||
lib/galaxy/tool_source_store/
|
||||
__init__.py ToolSourceStore ABC, StoredToolSource, build_tool_source_store()
|
||||
database.py DatabaseToolSourceStore (uses tool_source + tool_index tables)
|
||||
sqlalchemy.py SqlAlchemyToolSourceStore (any SA URL; sqlite shortcut)
|
||||
composite.py CompositeToolSourceStore (per-conf routing, merged index)
|
||||
index.py ToolIndex, ToolIndexEntry (the lightweight metadata)
|
||||
search.py ToolWhooshIndex (Whoosh search index built from a ToolIndex)
|
||||
discover.py discover_tools() — conf walk without booting a ToolBox
|
||||
populator.py Population + watch logic (parse, store, index, broadcast)
|
||||
freshness.py Optional external freshness probes
|
||||
watcher.py Filesystem watch support
|
||||
models.py Pydantic response models for the API layer
|
||||
|
||||
scripts/tool_source/populate_store.py Thin CLI wrapper over populator.main
|
||||
lib/galaxy/tools/lazy_toolbox.py LazyToolBox (subclass of ToolBox)
|
||||
lib/galaxy/tool_util/toolbox/
|
||||
base.py (small hook to support lazy mode)
|
||||
lib/galaxy/tool_util/id_util.py Cheap tool-ID extraction (regex, no XML parser)
|
||||
|
||||
The same ``populator.main`` is registered as the
|
||||
``galaxy-populate-tool-source-store`` console script in the ``galaxy-app``
|
||||
package metadata (``packages/app/pyproject.toml``).
|
||||
lib/galaxy/webapps/galaxy/services/tools.py Batch endpoints (index-aware)
|
||||
|
||||
scripts/tool_source/populate_store.py Population & watch script
|
||||
scripts/tool_source/_discover.py Tool-file discovery (used by populate_store)
|
||||
|
||||
Data Model
|
||||
----------
|
||||
@@ -55,37 +52,35 @@ Two persistence concepts:
|
||||
|
||||
**StoredToolSource** — the canonical macro-expanded XML/YAML for a tool,
|
||||
keyed by SHA-256 of the expanded content. Multiple versions of the same
|
||||
``tool_id`` coexist as separate hashes. The store keeps its own schema in a
|
||||
standalone database (a SQLite file by default, any SQLAlchemy URL for shared
|
||||
deployments) — deliberately outside Galaxy's database: the store is a
|
||||
rebuildable cache and does not participate in Galaxy's migrations or session
|
||||
lifecycle.
|
||||
``tool_id`` coexist as separate hashes. The DB backend persists these in the
|
||||
``tool_source`` table; the Redis and disk backends use their own layout.
|
||||
|
||||
**ToolIndex** — a Pydantic model containing one default ``ToolIndexEntry`` per tool
|
||||
plus its versioned and panel-placement projections,
|
||||
holding everything a store consumer needs (id, name, description, panel section,
|
||||
**ToolIndex** — a single dataclass containing one ``ToolIndexEntry`` per tool,
|
||||
holding everything the batch APIs need (id, name, description, panel section,
|
||||
labels, EDAM, requirements, container info, test counts, hidden/disabled,
|
||||
shed metadata). The index is serialized and gzip-compressed as a blob.
|
||||
|
||||
The schema is auto-created on first open; ``tool_index`` holds a single
|
||||
row per index version.
|
||||
The DB backend gets a new ``tool_index`` table (migration
|
||||
``f5a73c8b9d12_add_tool_index_table``) with a single row per index version.
|
||||
Redis stores the blob under a known key; disk stores it as a file.
|
||||
|
||||
Backend Abstraction
|
||||
-------------------
|
||||
|
||||
``ToolSourceStore`` (in ``tools/source_store/interface.py``) is an ABC defining:
|
||||
``ToolSourceStore`` (in ``tool_source_store/__init__.py``) is an ABC defining:
|
||||
|
||||
- ``store/get/exists/delete/list_all/get_by_tool_id/count`` — per-tool source
|
||||
operations, all keyed by content hash.
|
||||
- ``store_index/load_index/update_index_entry`` — index operations.
|
||||
- ``get_stats()`` — backend-specific stats (count, size, backend name).
|
||||
|
||||
``build_tool_source_store(config)`` is the only entry point used
|
||||
by Galaxy. It builds the default store from
|
||||
``config.tool_source_database_connection`` and uses the same SQLAlchemy-backed
|
||||
store implementation for all configured URIs. ``ConfigurationError`` is raised
|
||||
for missing required settings and is allowed to propagate up so
|
||||
misconfiguration fails fast at startup.
|
||||
``build_tool_source_store(app)`` is the only entry point used by Galaxy. It
|
||||
inspects ``config.tool_source_store`` and lazily imports the chosen backend
|
||||
so deployments only pay for the dependencies they use (no SQLAlchemy import
|
||||
on a Redis-only deploy, no ``redis`` client on a DB-only deploy).
|
||||
``ConfigurationError`` is raised for unknown backends or missing required
|
||||
settings; it is allowed to propagate up so misconfiguration fails fast at
|
||||
startup.
|
||||
|
||||
The ABC defines a ``read_only: bool`` class attribute (default ``False``).
|
||||
``ReadOnlyStoreError`` is raised by mutating methods of stores that opted
|
||||
@@ -100,8 +95,9 @@ or ``store: ...`` key (YAML), ``build_tool_source_store`` instantiates
|
||||
the referenced named stores from ``config.tool_source_stores`` and wraps
|
||||
them with the writable default in a :class:`CompositeToolSourceStore`.
|
||||
|
||||
The composite implements the same ``ToolSourceStore`` interface, so store
|
||||
consumers stay completely unaware of the multi-store layout:
|
||||
The composite implements the same ``ToolSourceStore`` interface, so the
|
||||
LazyToolBox, services, and queue worker stay completely unaware of the
|
||||
multi-store layout:
|
||||
|
||||
- **Reads** iterate ``[per-conf members..., default]`` in order; first
|
||||
hit wins. ``count`` and ``list_all`` dedupe across members.
|
||||
@@ -123,12 +119,11 @@ case.
|
||||
The ``sqlalchemy`` backend (``sqlalchemy.py``) was added to make this
|
||||
useful for CVMFS: a single self-contained ``.sqlite`` file, opened with
|
||||
its own SQLAlchemy ``MetaData`` (independent of ``galaxy.model``) so the
|
||||
file is portable, and openable with a SQLite URI such as
|
||||
``sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true``
|
||||
for read-only mounts. Despite the name, the backend is not sqlite-specific -
|
||||
pass any SQLAlchemy URL (Postgres, MySQL, ...). Auto schema creation runs on
|
||||
first open; on remote backends operators may prefer to manage migrations
|
||||
explicitly.
|
||||
file is portable, and openable with ``mode=ro&uri=true`` for read-only
|
||||
mounts. Despite the name, the backend is not sqlite-specific — pass any
|
||||
SQLAlchemy ``url`` (Postgres, MySQL, …) instead of ``path``. Auto schema
|
||||
creation runs on first open; on remote backends operators may prefer to
|
||||
manage migrations explicitly.
|
||||
|
||||
Per-conf populator routing
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@@ -142,34 +137,65 @@ run; ``--target NAME`` restricts to a single store and raises
|
||||
read-only in default mode are silently skipped (the bundle is treated as
|
||||
authoritative for those entries).
|
||||
|
||||
LazyToolBox
|
||||
-----------
|
||||
|
||||
``LazyToolBox`` extends ``ToolBox`` rather than reimplementing it, so the rest
|
||||
of Galaxy can keep using the same ``trans.app.toolbox`` interface. The key
|
||||
override is ``__init__``: instead of calling ``_init_tools_from_configs`` and
|
||||
loading every tool, it calls ``_init_lazy_toolbox`` which:
|
||||
|
||||
1. Sets up the same internal dicts (``_tools_by_id``, ``_tools_by_uuid``,
|
||||
etc.) that ``AbstractToolBox.__init__`` would.
|
||||
2. Walks the tool conf files just enough to build the panel structure
|
||||
(sections, labels) and a ``tool_id → (section_id, section_name)`` map.
|
||||
3. Loads the ``ToolIndex`` from the store. If the store has tool sources but
|
||||
no index, the index is rebuilt from the stored sources.
|
||||
4. Populates ``_tools_by_id`` with stub entries from the index so
|
||||
``has_tool()`` and similar lookups work without touching the store.
|
||||
|
||||
Full ``Tool`` objects are built on demand and kept in an ``LRUCache`` of
|
||||
``lazy_toolbox_cache_size`` entries (default 500). Cache hits and misses are
|
||||
guarded by an ``RLock`` for thread safety.
|
||||
|
||||
The toolbox auto-enables: if the operator hasn't set ``use_lazy_toolbox``,
|
||||
``galaxy.app._use_lazy_toolbox()`` enables the lazy implementation when the
|
||||
configured store contains at least one tool. This means a fresh deploy
|
||||
without the populator run keeps the eager behavior, and only switches once
|
||||
``populate_store.py`` succeeds.
|
||||
|
||||
Tool ID extraction
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``galaxy.tool_util.id_util`` provides ``extract_tool_id_from_xml`` and
|
||||
``extract_tool_id_from_file``: regex-based ID lookup that reads only the
|
||||
first ~2 KB of the XML. This avoids paying for full XML parsing during
|
||||
panel-structure discovery, where we just need the ID to map a file entry
|
||||
back to an index entry.
|
||||
|
||||
Discovery
|
||||
---------
|
||||
^^^^^^^^^
|
||||
|
||||
``galaxy.tools.source_store.discover.discover_tools`` walks tool config files
|
||||
and yields ``DiscoveredTool`` records without booting a full ``ToolBox``. It is
|
||||
used by:
|
||||
``scripts.tool_source._discover.discover_tools`` walks tool config files
|
||||
and yields ``DiscoveredTool`` records. It is used by:
|
||||
|
||||
- the populator to find tools to parse and store.
|
||||
- watch mode to know which directories to monitor.
|
||||
- callers that compare on-disk confs against the indexed tool set.
|
||||
- ``populate_store.py`` to find tools to parse and store.
|
||||
- ``populate_store.py --watch`` to know which directories to monitor.
|
||||
- (Indirectly) the LazyToolBox panel-structure code path.
|
||||
|
||||
Pulling discovery out of ``ToolBox`` was deliberate: the populator must run
|
||||
*without* a full app (or even a running Galaxy), and the watch mode must run in
|
||||
a long-lived loop with no Galaxy process at all.
|
||||
Pulling discovery out of ``ToolBox`` was deliberate: the population script
|
||||
must run *without* a full app (or even a running Galaxy), and the watch
|
||||
mode must run in a long-lived loop with no Galaxy process at all.
|
||||
|
||||
Population Script
|
||||
-----------------
|
||||
|
||||
``scripts/tool_source/populate_store.py`` is a thin CLI wrapper over
|
||||
``galaxy.tools.source_store.populator.main``. It loads only the Galaxy
|
||||
config and calls ``build_tool_source_store(config)``. Converter discovery builds
|
||||
the datatypes registry, but the standalone process does not initialize the Galaxy
|
||||
model. Tools are parsed in a
|
||||
``scripts/tool_source/populate_store.py`` runs out of process. It builds a
|
||||
minimal app context (datatypes registry + SQLAlchemy model + config) and
|
||||
calls ``build_tool_source_store`` with that context. Tools are parsed in a
|
||||
``ThreadPoolExecutor`` (``--parallel``, default 4 workers); each tool is
|
||||
matched to its source path and carried forward when its raw file hash is unchanged
|
||||
(``--incremental``, the default). Once the JSON index is committed the
|
||||
populator rebuilds the Whoosh search index (``search.py``) so ranked tool
|
||||
search stays in sync with the stored sources.
|
||||
hashed and skipped if an entry with the same hash already exists
|
||||
(``--incremental``, the default).
|
||||
|
||||
Watch mode (``--watch``) uses ``watchdog`` to monitor every directory yielded
|
||||
by ``discover_tools``. File events are debounced (default 2 s), the changed
|
||||
@@ -178,23 +204,84 @@ files are re-parsed, the store is updated, and a single
|
||||
exchange. ``--watch-polling`` switches to ``PollingObserver`` for
|
||||
NFS/CVMFS/network filesystems where inotify is unreliable.
|
||||
|
||||
The broadcast is the populator's half of the contract: it publishes
|
||||
``reload_tool_source_cache`` so peer processes can drop their stale index
|
||||
view. The control-task handler that consumes the message lands with the
|
||||
follow-up toolbox.
|
||||
The control task handler lives in ``galaxy.queue_worker.reload_tool_source_cache``
|
||||
and is wired into the ``control_message_to_task`` map. Each Galaxy process
|
||||
that receives the message:
|
||||
|
||||
1. Calls ``LazyToolBox.invalidate_index_cache()`` (drops the in-memory
|
||||
index reference so the next access reloads from the store).
|
||||
2. Calls ``ToolSourceStore.invalidate_index_cache()`` on the store itself.
|
||||
|
||||
Note that the LRU cache of fully constructed ``Tool`` objects is not
|
||||
flushed by reload — only the index is invalidated. Stale ``Tool`` instances
|
||||
are evicted naturally as new ones are loaded.
|
||||
|
||||
Batch Endpoint Integration
|
||||
--------------------------
|
||||
|
||||
``ToolsService`` (``services/tools.py``) checks at runtime whether
|
||||
``trans.app.toolbox`` is a ``LazyToolBox`` (via ``_get_lazy_toolbox``) and
|
||||
routes accordingly:
|
||||
|
||||
- ``list_tools(in_panel=False)`` → ``index.list_all()`` mapped to API dicts.
|
||||
- ``search_tools`` → ``index.search()`` returning IDs.
|
||||
- ``get_tests_summary`` → ``index.get_tests_summary()``.
|
||||
- ``get_all_requirements`` → ``index.get_all_requirements()``.
|
||||
- ``get_panel_views`` → ``index.get_panel_views()`` for views, plus the
|
||||
toolbox's own ``default_panel_view``.
|
||||
|
||||
In every case there is a fallback path that iterates the eager toolbox, so
|
||||
the same code works on deployments that have not opted into lazy mode.
|
||||
|
||||
``ToolAPICache`` (``api_cache.py``) sits in front of the gzip-compressed JSON
|
||||
representations of these batch responses, keyed by the canonical request
|
||||
URL and TTL'd by ``tool_api_cache_ttl`` (default 300 s, ``0`` disables).
|
||||
|
||||
API Layer
|
||||
---------
|
||||
|
||||
``api/tool_sources.py`` exposes three FastAPI ``Router.cbv`` controllers:
|
||||
|
||||
- ``ToolSourcesAPI`` — store inspection (admin-only).
|
||||
- ``ToolIndexAPI`` — index browsing (mostly public, ``/stats`` is admin).
|
||||
- ``ToolCacheAPI`` — LazyToolBox cache stats and clear (admin-only).
|
||||
|
||||
Static suffix routes (``/stats``, ``/search``) deliberately precede the
|
||||
``/{id}`` and ``/{hash}`` parameterized routes to avoid path shadowing —
|
||||
this is enforced by route declaration order, not by FastAPI itself.
|
||||
|
||||
``services/tool_sources.py`` is the thin service layer over the store and
|
||||
the index. It depends on ``trans.app.tool_source_store``; the LazyToolBox
|
||||
is only consulted for cache stats/clear.
|
||||
|
||||
App Wiring
|
||||
----------
|
||||
|
||||
``galaxy.app.UniverseApplication.__init__`` calls
|
||||
``_init_tool_source_store`` early and registers the result as a singleton
|
||||
under ``ToolSourceStore``. The toolbox is then chosen based on
|
||||
``_use_lazy_toolbox()`` (explicit config override, otherwise auto-detect).
|
||||
The store is exposed as ``app.tool_source_store`` and is ``Optional`` only
|
||||
to satisfy type checkers — in practice the build either succeeds or raises
|
||||
``ConfigurationError``.
|
||||
|
||||
Design Notes
|
||||
------------
|
||||
|
||||
**Why a separate index instead of always querying the store?** A consumer
|
||||
needs O(N) access to N entries; doing that against the backing store on every
|
||||
request is a latency hit. Keeping the index in-process and only paying for
|
||||
invalidation on reload is the better tradeoff.
|
||||
**Why a separate index instead of always-querying-the-store?** Batch
|
||||
endpoints need O(N) access to N entries; doing that against Redis or the
|
||||
disk is a latency hit on every request. Keeping the index in-process and
|
||||
only paying for invalidation on reload is the better tradeoff.
|
||||
|
||||
**Why an out-of-process populator?** Parsing tools and computing macro
|
||||
expansions is expensive and shouldn't block worker startup. Keeping the
|
||||
populator separate also lets it run on a single host while many web workers
|
||||
share the resulting store.
|
||||
share the resulting store (especially with the Redis backend).
|
||||
|
||||
**Why subclass ToolBox instead of building a parallel hierarchy?**
|
||||
``trans.app.toolbox`` is referenced from hundreds of call sites that expect
|
||||
the full ToolBox interface. Subclassing keeps the Liskov-substitution
|
||||
property and lets unmodified callers benefit from lazy loading transparently.
|
||||
|
||||
**Why hash-keyed storage?** Content-addressed storage gives us cheap
|
||||
deduplication across versions and shed installations, and idempotent
|
||||
@@ -204,12 +291,12 @@ effectively a no-op.
|
||||
Testing
|
||||
-------
|
||||
|
||||
- Store unit tests: ``test/unit/app/tools/source_store/`` exercises each backend
|
||||
through the ``ToolSourceStore`` interface (``test_stores.py``,
|
||||
``test_sqlite_store.py``, ``test_composite_store.py``,
|
||||
``test_index_versions.py``).
|
||||
- Populator/discovery unit tests: ``test/unit/scripts/tool_source/``
|
||||
(``test_populate_store.py``, ``test_discover.py``,
|
||||
``test_build_index_entry.py``, ``test_whoosh_dir.py``). These use fakes
|
||||
(not mocks) of ``ToolSourceStore`` so behavior is verified against the real
|
||||
interface.
|
||||
- Unit tests: ``test/unit/tool_source_store/test_stores.py`` exercises each
|
||||
backend through the ``ToolSourceStore`` interface.
|
||||
- Integration tests: ``test/integration/test_tool_source_storage.py`` spins
|
||||
up Galaxy with each backend and verifies end-to-end behavior.
|
||||
- Populator tests: ``test/unit/scripts/tool_source/test_populate_store.py``
|
||||
uses fakes (not mocks) of ``ToolSourceStore`` so behavior is verified
|
||||
against the real interface.
|
||||
- API tests: ``lib/galaxy_test/api/test_tool_sources.py``.
|
||||
- Benchmarks: ``python -m galaxy.tool_source_store.benchmarks --iterations 100``.
|
||||
|
||||
Reference in New Issue
Block a user