opensource/galaxy
3
0
Fork 0
mirror of https://github.com/galaxyproject/galaxy.git synced 2026-09-21 05:45:37 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: rebuild tool source storage docs on dev's tailored text

Browse Source 
The rebase replayed this branch's older doc revisions over the versions
tailored on the store-infra PR. Start from dev's text and add only the
branch-specific content: LazyToolBox opt-in and cache configuration,
batch-endpoint integration, app wiring, the reload handler (including
the content-change refresh), and the LazyToolBox architecture section.
The id_util 'Tool ID extraction' section is gone — the module is being
folded into existing helpers.
This commit is contained in:
mvdbeek
2026-07-28 17:27:48 +02:00
parent b389a46a00
commit 2c7abc8f29
2 changed files with 85 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/admin/tool_source_storage.rst
+46 -45
View File
@@ -1,25 +1,26 @@
Tool Source Storage
===================
Galaxy can pre-parse and store tool sources, plus a lightweight index, in a
configurable backend, and load full ``Tool`` objects on demand through the
``LazyToolBox``. This is especially useful for large Galaxy installations
with thousands of tools.
Overview
--------
By default, Galaxy loads all tools into memory at startup. For installations with many tools,
this can:
By default, Galaxy parses every tool at startup and keeps all of them in
memory. For installations with many tools this:
- Slow down Galaxy startup significantly
- Consume large amounts of memory
- Slows down Galaxy startup significantly
- Consumes large amounts of memory in every Galaxy process
The tool source storage system addresses these issues by:
Tool source storage addresses this by doing the parsing work once, ahead of
time:
1. Pre-parsing and storing tool sources in a configurable backend
2. Maintaining a lightweight index in memory for fast API responses
3. Loading full ``Tool`` objects on demand with LRU caching
1. Tool sources are pre-parsed (with macros expanded) and stored in a
configurable database backend
2. A lightweight index over the stored tools supports fast tool listings and
search without touching tool files
The ``LazyToolBox`` consumes this store to load full ``Tool`` objects on
demand with LRU caching. It is opt-in via ``use_lazy_toolbox``; this document
covers the store, the populator, the index, and the toolbox configuration.
Configuration
-------------
@@ -37,19 +38,19 @@ Default Store
The store lives in a standalone database - a SQLite file under
``<data_dir>/tool_sources.sqlite`` by default - separate from Galaxy's main
database. It is a rebuildable cache: deleting it costs one populator run.
This URI is used by tool source storage code paths, including the population
script and lazy toolbox consumers. Runtime use also requires a populated store
and a toolbox consumer configured to read from tool source storage.
database. It is a rebuildable cache: it can be deleted at any time and
recreated by re-running the population script.
Multi-host deployments must point every Galaxy process (web workers *and*
job handlers) at the same store — typically a SQLite file on a shared
filesystem:
.. code-block:: yaml
galaxy:
tool_source_database_connection: postgresql://galaxy@db.example.org/tool_sources
tool_source_database_connection: sqlite:////shared/galaxy/tool_sources.sqlite
Multi-host deployments must point every Galaxy process (web workers *and*
job handlers) at the same store, such as a SQLite file on a shared filesystem
or a shared database URI.
Any other SQLAlchemy-supported database (e.g. PostgreSQL) works as well.
Toolbox Selection
^^^^^^^^^^^^^^^^^
@@ -63,9 +64,21 @@ Toolbox Selection
The LazyToolBox is opt-in: leave ``use_lazy_toolbox`` unset (or false) and
Galaxy uses the traditional eager ToolBox even when the store is populated
or when a tool_conf carries a ``store="..."`` attribute. Set
``use_lazy_toolbox: true`` to activate lazy loading and per-conf store
routing. The store is only initialized when the LazyToolBox is enabled.
or when a tool_conf carries a ``store="..."`` attribute. The store is only
initialized when the LazyToolBox is enabled.
Cache Configuration
^^^^^^^^^^^^^^^^^^^
.. code-block:: yaml
galaxy:
# Maximum Tool objects in the LazyToolBox LRU cache (default: 500)
lazy_toolbox_cache_size: 500
The ``lazy_toolbox_cache_size`` determines how many fully-loaded Tool objects
are kept in memory by the LazyToolBox. If your users frequently work with
many different tools, increase this value.
Per-conf Store Routing (CVMFS Recipe)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -90,7 +103,7 @@ self-contained file), but any SQLAlchemy-supported database works:
url: sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true
read_only: true
site_shared:
url: postgresql://galaxy_ro@db.example.org/tool_sources
url: sqlite:///file:/shared/galaxy/tool_sources.sqlite?mode=ro&uri=true
read_only: true
Then point the tool_conf at it via the root element's ``store`` attribute
@@ -129,19 +142,6 @@ Galaxy. The ``read_only: true`` flag prevents Galaxy from writing through that
store. For SQLite connection-level read-only, use ``mode=ro&uri=true`` in the
SQLite URI as shown above.
Cache Configuration
^^^^^^^^^^^^^^^^^^^
.. code-block:: yaml
galaxy:
# Maximum Tool objects in the LazyToolBox LRU cache (default: 500)
lazy_toolbox_cache_size: 500
The ``lazy_toolbox_cache_size`` determines how many fully-loaded Tool objects
are kept in memory by the LazyToolBox. A typical Galaxy installation has
500-2000 tools. If you frequently use many different tools, increase this value.
Populating the Tool Source Store
--------------------------------
@@ -228,10 +228,10 @@ script on a schedule:
Watch Mode (Live Updates)
^^^^^^^^^^^^^^^^^^^^^^^^^
For development environments or installations where tools change frequently, you can run
the population script in watch mode. This uses ``watchdog`` to monitor tool directories
for changes and automatically updates the store, then sends a notification via Kombu
to trigger cache reloads in all Galaxy processes.
As an alternative to cron, you can run the population script in watch mode to
keep the store continuously up to date. This uses ``watchdog`` to monitor tool
directories for changes and automatically updates the store, then sends a
notification via Kombu to trigger cache reloads in all Galaxy processes.
.. code-block:: console
@@ -263,9 +263,9 @@ When a tool XML file changes, watch mode will:
This is useful for:
- Development environments where tools are being actively edited
- CI/CD pipelines that deploy tool updates
- Installations using shared storage where tools may be updated externally
- CI/CD pipelines that deploy tool updates
- Development environments where tools are being actively edited
Troubleshooting
---------------
@@ -298,6 +298,7 @@ To set up tool source storage on an existing Galaxy installation:
galaxy:
use_lazy_toolbox: true
tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite
2. Run the population script:
@@ -307,6 +308,6 @@ To set up tool source storage on an existing Galaxy installation:
3. Restart Galaxy
If the store is not populated, or a specific tool is not found in it, the
If the store is not populated, or a specific tool is missing from it, the
LazyToolBox self-heals by populating the missing entries in-process, so the
traditional toolbox behavior is preserved as a fallback.
doc/source/dev/tool_source_storage.rst
+39 -42
View File
@@ -1,9 +1,12 @@
Tool Source Storage Architecture
================================
This document describes the architecture of the tool source storage subsystem
and the LazyToolBox. For operator-facing setup and configuration, see
:doc:`/admin/tool_source_storage`.
This document describes the architecture of the tool source storage subsystem:
the store backends, the populator, and the index they build. For operator-facing
setup and configuration, see :doc:`/admin/tool_source_storage`.
The ``LazyToolBox`` consumes this store to load tools on demand; it is
documented below alongside the storage layer it builds on.
Goals
-----
@@ -12,12 +15,14 @@ The traditional ``ToolBox`` parses every tool XML at startup, builds full
``Tool`` objects, and keeps them all in memory. With thousands of tools that
scales poorly: slow boot, large per-process RSS, and expensive worker reloads.
The tool source storage subsystem moves that work out of the request path:
The tool source storage subsystem moves that parsing work out of the request
path:
- A separate process (``populate_store.py``) parses tools once and persists
the canonical, macro-expanded source plus a lightweight metadata index.
- Galaxy processes load only the index at startup and materialize ``Tool``
objects on demand, with LRU eviction.
- The store and index are laid out so a consumer can load only the index at
startup and materialize ``Tool`` objects on demand, instead of parsing the
full tree in-process. That consumer is the ``LazyToolBox``.
- Batch endpoints (``/api/tools``, ``/api/tools/tests_summary``,
``/api/tool_panels`` …) answer from the index instead of iterating the
full toolbox.
@@ -28,24 +33,23 @@ Module Layout
::
lib/galaxy/tools/source_store/
__init__.py Facade re-exporting the interface + factory
interface.py ToolSourceStore ABC, StoredToolSource, exceptions
factory.py build_tool_source_store() / build_named_store()
__init__.py Public re-exports
interface.py ToolSourceStore ABC and StoredToolSource
factory.py Store construction from Galaxy configuration
sqlalchemy.py SqlAlchemyToolSourceStore (any SQLAlchemy URL)
composite.py CompositeToolSourceStore (per-conf routing, merged index)
index.py ToolIndex, ToolIndexEntry (the lightweight metadata)
search.py ToolWhooshIndex (Whoosh index built from a ToolIndex)
search.py ToolWhooshIndex (Whoosh search index built from a ToolIndex)
discover.py discover_tools() — conf walk without booting a ToolBox
populator.py Population + watch logic (parse, store, index, broadcast)
freshness.py Optional external freshness probes
watcher.py Filesystem watch support
benchmarks.py Store/index micro-benchmarks
lib/galaxy/tools/lazy_toolbox.py LazyToolBox (subclass of ToolBox), LazyTool
lib/galaxy/tools/search/__init__.py LazyToolboxSearch (queries every store's index)
lib/galaxy/tool_util/id_util.py Cheap tool-ID extraction (regex, no XML parser)
lib/galaxy/webapps/galaxy/services/tools.py Batch endpoints (lazy-aware)
scripts/tool_source/populate_store.py CLI entry point for the populator
scripts/tool_source/populate_store.py Thin CLI wrapper over populator.main
The same ``populator.main`` is registered as the
``galaxy-populate-tool-source-store`` console script in the ``galaxy-app``
@@ -66,10 +70,9 @@ lifecycle.
**ToolIndex** — a Pydantic model containing one default ``ToolIndexEntry`` per tool
plus its versioned and panel-placement projections,
holding everything the batch APIs and the lazy panel render need (id, name,
description, panel section, labels, EDAM, xrefs, icon, requirements, container
info, test counts, hidden/disabled, shed metadata, ``data_manager_id``). The
index is serialized and gzip-compressed as a blob.
holding everything a store consumer needs (id, name, description, panel section,
labels, EDAM, requirements, container info, test counts, hidden/disabled,
shed metadata). The index is serialized and gzip-compressed as a blob.
The schema is auto-created on first open; ``tool_index`` holds a single
row per index version.
@@ -105,9 +108,8 @@ or ``store: ...`` key (YAML), ``build_tool_source_store`` instantiates
the referenced named stores from ``config.tool_source_stores`` and wraps
them with the writable default in a :class:`CompositeToolSourceStore`.
The composite implements the same ``ToolSourceStore`` interface, so the
LazyToolBox, services, and queue worker stay completely unaware of the
multi-store layout:
The composite implements the same ``ToolSourceStore`` interface, so store
consumers stay completely unaware of the multi-store layout:
- **Reads** iterate ``[per-conf members..., default]`` in order; first
hit wins. ``count`` and ``list_all`` dedupe across members.
@@ -174,15 +176,6 @@ Opting in is explicit: only ``use_lazy_toolbox: true`` activates the lazy
toolbox. A populated store on its own (e.g. brought in by a per-conf
``store="..."`` attribute) does not flip a default deployment to lazy mode.
Tool ID extraction
^^^^^^^^^^^^^^^^^^
``galaxy.tool_util.id_util`` provides ``extract_tool_id_from_xml`` and
``extract_tool_id_from_file``: regex-based ID lookup that reads only the
first ~2 KB of the XML. This avoids paying for full XML parsing during
panel-structure discovery, where we just need the ID to map a file entry
back to an index entry.
Discovery
---------
@@ -193,7 +186,6 @@ used by:
- the populator to find tools to parse and store.
- watch mode to know which directories to monitor.
- callers that compare on-disk confs against the indexed tool set.
- (indirectly) the LazyToolBox panel-structure code path.
It also walks ``data_manager_conf``/``shed_data_manager_conf`` and the
datatype converters so data-manager and converter tools — loaded post-boot
@@ -208,9 +200,9 @@ Population Script
``scripts/tool_source/populate_store.py`` is a thin CLI wrapper over
``galaxy.tools.source_store.populator.main``. It loads only the Galaxy
config and calls ``build_tool_source_store(config)`` — the standalone store
builds the datatypes registry for converter discovery but does not initialize
the Galaxy model. Tools are parsed in a
config and calls ``build_tool_source_store(config)``. Converter discovery builds
the datatypes registry, but the standalone process does not initialize the Galaxy
model. Tools are parsed in a
``ThreadPoolExecutor`` (``--parallel``, default 4 workers); each tool is
matched to its source path and carried forward when its raw file hash is unchanged
(``--incremental``, the default). Once the JSON index is committed the
@@ -224,17 +216,21 @@ files are re-parsed, the store is updated, and a single
exchange. ``--watch-polling`` switches to ``PollingObserver`` for
NFS/CVMFS/network filesystems where inotify is unreliable.
The control task handler lives in ``galaxy.queue_worker.reload_tool_source_cache``
and is wired into the ``control_message_to_task`` map. Each Galaxy process
that receives the message:
The broadcast is the populator's half of the contract: it publishes
``reload_tool_source_cache`` so peer processes can drop their stale index
view. The control task handler lives in
``galaxy.queue_worker.reload_tool_source_cache`` and is wired into the
``control_message_to_task`` map. Each Galaxy process that receives the
message:
1. Calls ``LazyToolBox.invalidate_index_cache()`` (drops the in-memory
index reference so the next access reloads from the store).
2. Calls ``ToolSourceStore.invalidate_index_cache()`` on the store itself.
Note that the LRU cache of fully constructed ``Tool`` objects is not
flushed by reload — only the index is invalidated. Stale ``Tool`` instances
are evicted naturally as new ones are loaded.
Reload also refreshes already-materialised tools: entries whose source hash
changed have their LRU entries, stubs, and registered ``Tool`` objects
purged, so the next access re-materialises from the new source. Unchanged
entries keep their cached ``Tool`` objects.
Batch Endpoint Integration
--------------------------
@@ -259,8 +255,9 @@ single materialise chokepoint) must not move across any of these endpoints.
``LazyToolboxSearch`` (``tools/search/__init__.py``) queries the whoosh index
of *every* configured store — the default plus each named per-conf store —
via ``ToolWhooshIndex.search_scored``, then merges the per-store hit lists by
BM25 score. A tool served from a named store is therefore reachable through
``/api/tools?q=`` even though its source lives outside the default store.
BM25 score and post-filters them to the requested panel view. A tool served
from a named store is therefore reachable through ``/api/tools?q=`` even
though its source lives outside the default store.
App Wiring
----------