diff --git a/doc/source/admin/tool_source_storage.rst b/doc/source/admin/tool_source_storage.rst index 81de3effefd..b283e55d15f 100644 --- a/doc/source/admin/tool_source_storage.rst +++ b/doc/source/admin/tool_source_storage.rst @@ -1,25 +1,26 @@ Tool Source Storage =================== -Galaxy can pre-parse and store tool sources, plus a lightweight index, in a -configurable backend, and load full ``Tool`` objects on demand through the -``LazyToolBox``. This is especially useful for large Galaxy installations -with thousands of tools. - Overview -------- -By default, Galaxy loads all tools into memory at startup. For installations with many tools, -this can: +By default, Galaxy parses every tool at startup and keeps all of them in +memory. For installations with many tools this: -- Slow down Galaxy startup significantly -- Consume large amounts of memory +- Slows down Galaxy startup significantly +- Consumes large amounts of memory in every Galaxy process -The tool source storage system addresses these issues by: +Tool source storage addresses this by doing the parsing work once, ahead of +time: -1. Pre-parsing and storing tool sources in a configurable backend -2. Maintaining a lightweight index in memory for fast API responses -3. Loading full ``Tool`` objects on demand with LRU caching +1. Tool sources are pre-parsed (with macros expanded) and stored in a + configurable database backend +2. A lightweight index over the stored tools supports fast tool listings and + search without touching tool files + +The ``LazyToolBox`` consumes this store to load full ``Tool`` objects on +demand with LRU caching. It is opt-in via ``use_lazy_toolbox``; this document +covers the store, the populator, the index, and the toolbox configuration. Configuration ------------- @@ -37,19 +38,19 @@ Default Store The store lives in a standalone database - a SQLite file under ``/tool_sources.sqlite`` by default - separate from Galaxy's main -database. It is a rebuildable cache: deleting it costs one populator run. -This URI is used by tool source storage code paths, including the population -script and lazy toolbox consumers. Runtime use also requires a populated store -and a toolbox consumer configured to read from tool source storage. +database. It is a rebuildable cache: it can be deleted at any time and +recreated by re-running the population script. + +Multi-host deployments must point every Galaxy process (web workers *and* +job handlers) at the same store — typically a SQLite file on a shared +filesystem: .. code-block:: yaml galaxy: - tool_source_database_connection: postgresql://galaxy@db.example.org/tool_sources + tool_source_database_connection: sqlite:////shared/galaxy/tool_sources.sqlite -Multi-host deployments must point every Galaxy process (web workers *and* -job handlers) at the same store, such as a SQLite file on a shared filesystem -or a shared database URI. +Any other SQLAlchemy-supported database (e.g. PostgreSQL) works as well. Toolbox Selection ^^^^^^^^^^^^^^^^^ @@ -63,9 +64,21 @@ Toolbox Selection The LazyToolBox is opt-in: leave ``use_lazy_toolbox`` unset (or false) and Galaxy uses the traditional eager ToolBox even when the store is populated -or when a tool_conf carries a ``store="..."`` attribute. Set -``use_lazy_toolbox: true`` to activate lazy loading and per-conf store -routing. The store is only initialized when the LazyToolBox is enabled. +or when a tool_conf carries a ``store="..."`` attribute. The store is only +initialized when the LazyToolBox is enabled. + +Cache Configuration +^^^^^^^^^^^^^^^^^^^ + +.. code-block:: yaml + + galaxy: + # Maximum Tool objects in the LazyToolBox LRU cache (default: 500) + lazy_toolbox_cache_size: 500 + +The ``lazy_toolbox_cache_size`` determines how many fully-loaded Tool objects +are kept in memory by the LazyToolBox. If your users frequently work with +many different tools, increase this value. Per-conf Store Routing (CVMFS Recipe) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -90,7 +103,7 @@ self-contained file), but any SQLAlchemy-supported database works: url: sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true read_only: true site_shared: - url: postgresql://galaxy_ro@db.example.org/tool_sources + url: sqlite:///file:/shared/galaxy/tool_sources.sqlite?mode=ro&uri=true read_only: true Then point the tool_conf at it via the root element's ``store`` attribute @@ -129,19 +142,6 @@ Galaxy. The ``read_only: true`` flag prevents Galaxy from writing through that store. For SQLite connection-level read-only, use ``mode=ro&uri=true`` in the SQLite URI as shown above. -Cache Configuration -^^^^^^^^^^^^^^^^^^^ - -.. code-block:: yaml - - galaxy: - # Maximum Tool objects in the LazyToolBox LRU cache (default: 500) - lazy_toolbox_cache_size: 500 - -The ``lazy_toolbox_cache_size`` determines how many fully-loaded Tool objects -are kept in memory by the LazyToolBox. A typical Galaxy installation has -500-2000 tools. If you frequently use many different tools, increase this value. - Populating the Tool Source Store -------------------------------- @@ -228,10 +228,10 @@ script on a schedule: Watch Mode (Live Updates) ^^^^^^^^^^^^^^^^^^^^^^^^^ -For development environments or installations where tools change frequently, you can run -the population script in watch mode. This uses ``watchdog`` to monitor tool directories -for changes and automatically updates the store, then sends a notification via Kombu -to trigger cache reloads in all Galaxy processes. +As an alternative to cron, you can run the population script in watch mode to +keep the store continuously up to date. This uses ``watchdog`` to monitor tool +directories for changes and automatically updates the store, then sends a +notification via Kombu to trigger cache reloads in all Galaxy processes. .. code-block:: console @@ -263,9 +263,9 @@ When a tool XML file changes, watch mode will: This is useful for: -- Development environments where tools are being actively edited -- CI/CD pipelines that deploy tool updates - Installations using shared storage where tools may be updated externally +- CI/CD pipelines that deploy tool updates +- Development environments where tools are being actively edited Troubleshooting --------------- @@ -298,6 +298,7 @@ To set up tool source storage on an existing Galaxy installation: galaxy: use_lazy_toolbox: true + tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite 2. Run the population script: @@ -307,6 +308,6 @@ To set up tool source storage on an existing Galaxy installation: 3. Restart Galaxy -If the store is not populated, or a specific tool is not found in it, the +If the store is not populated, or a specific tool is missing from it, the LazyToolBox self-heals by populating the missing entries in-process, so the traditional toolbox behavior is preserved as a fallback. diff --git a/doc/source/dev/tool_source_storage.rst b/doc/source/dev/tool_source_storage.rst index 585288f0d98..6dc7c53a8ff 100644 --- a/doc/source/dev/tool_source_storage.rst +++ b/doc/source/dev/tool_source_storage.rst @@ -1,9 +1,12 @@ Tool Source Storage Architecture ================================ -This document describes the architecture of the tool source storage subsystem -and the LazyToolBox. For operator-facing setup and configuration, see -:doc:`/admin/tool_source_storage`. +This document describes the architecture of the tool source storage subsystem: +the store backends, the populator, and the index they build. For operator-facing +setup and configuration, see :doc:`/admin/tool_source_storage`. + +The ``LazyToolBox`` consumes this store to load tools on demand; it is +documented below alongside the storage layer it builds on. Goals ----- @@ -12,12 +15,14 @@ The traditional ``ToolBox`` parses every tool XML at startup, builds full ``Tool`` objects, and keeps them all in memory. With thousands of tools that scales poorly: slow boot, large per-process RSS, and expensive worker reloads. -The tool source storage subsystem moves that work out of the request path: +The tool source storage subsystem moves that parsing work out of the request +path: - A separate process (``populate_store.py``) parses tools once and persists the canonical, macro-expanded source plus a lightweight metadata index. -- Galaxy processes load only the index at startup and materialize ``Tool`` - objects on demand, with LRU eviction. +- The store and index are laid out so a consumer can load only the index at + startup and materialize ``Tool`` objects on demand, instead of parsing the + full tree in-process. That consumer is the ``LazyToolBox``. - Batch endpoints (``/api/tools``, ``/api/tools/tests_summary``, ``/api/tool_panels`` …) answer from the index instead of iterating the full toolbox. @@ -28,24 +33,23 @@ Module Layout :: lib/galaxy/tools/source_store/ - __init__.py Facade re-exporting the interface + factory - interface.py ToolSourceStore ABC, StoredToolSource, exceptions - factory.py build_tool_source_store() / build_named_store() + __init__.py Public re-exports + interface.py ToolSourceStore ABC and StoredToolSource + factory.py Store construction from Galaxy configuration sqlalchemy.py SqlAlchemyToolSourceStore (any SQLAlchemy URL) composite.py CompositeToolSourceStore (per-conf routing, merged index) index.py ToolIndex, ToolIndexEntry (the lightweight metadata) - search.py ToolWhooshIndex (Whoosh index built from a ToolIndex) + search.py ToolWhooshIndex (Whoosh search index built from a ToolIndex) discover.py discover_tools() — conf walk without booting a ToolBox populator.py Population + watch logic (parse, store, index, broadcast) + freshness.py Optional external freshness probes + watcher.py Filesystem watch support benchmarks.py Store/index micro-benchmarks lib/galaxy/tools/lazy_toolbox.py LazyToolBox (subclass of ToolBox), LazyTool lib/galaxy/tools/search/__init__.py LazyToolboxSearch (queries every store's index) - lib/galaxy/tool_util/id_util.py Cheap tool-ID extraction (regex, no XML parser) - lib/galaxy/webapps/galaxy/services/tools.py Batch endpoints (lazy-aware) - - scripts/tool_source/populate_store.py CLI entry point for the populator + scripts/tool_source/populate_store.py Thin CLI wrapper over populator.main The same ``populator.main`` is registered as the ``galaxy-populate-tool-source-store`` console script in the ``galaxy-app`` @@ -66,10 +70,9 @@ lifecycle. **ToolIndex** — a Pydantic model containing one default ``ToolIndexEntry`` per tool plus its versioned and panel-placement projections, -holding everything the batch APIs and the lazy panel render need (id, name, -description, panel section, labels, EDAM, xrefs, icon, requirements, container -info, test counts, hidden/disabled, shed metadata, ``data_manager_id``). The -index is serialized and gzip-compressed as a blob. +holding everything a store consumer needs (id, name, description, panel section, +labels, EDAM, requirements, container info, test counts, hidden/disabled, +shed metadata). The index is serialized and gzip-compressed as a blob. The schema is auto-created on first open; ``tool_index`` holds a single row per index version. @@ -105,9 +108,8 @@ or ``store: ...`` key (YAML), ``build_tool_source_store`` instantiates the referenced named stores from ``config.tool_source_stores`` and wraps them with the writable default in a :class:`CompositeToolSourceStore`. -The composite implements the same ``ToolSourceStore`` interface, so the -LazyToolBox, services, and queue worker stay completely unaware of the -multi-store layout: +The composite implements the same ``ToolSourceStore`` interface, so store +consumers stay completely unaware of the multi-store layout: - **Reads** iterate ``[per-conf members..., default]`` in order; first hit wins. ``count`` and ``list_all`` dedupe across members. @@ -174,15 +176,6 @@ Opting in is explicit: only ``use_lazy_toolbox: true`` activates the lazy toolbox. A populated store on its own (e.g. brought in by a per-conf ``store="..."`` attribute) does not flip a default deployment to lazy mode. -Tool ID extraction -^^^^^^^^^^^^^^^^^^ - -``galaxy.tool_util.id_util`` provides ``extract_tool_id_from_xml`` and -``extract_tool_id_from_file``: regex-based ID lookup that reads only the -first ~2 KB of the XML. This avoids paying for full XML parsing during -panel-structure discovery, where we just need the ID to map a file entry -back to an index entry. - Discovery --------- @@ -193,7 +186,6 @@ used by: - the populator to find tools to parse and store. - watch mode to know which directories to monitor. - callers that compare on-disk confs against the indexed tool set. -- (indirectly) the LazyToolBox panel-structure code path. It also walks ``data_manager_conf``/``shed_data_manager_conf`` and the datatype converters so data-manager and converter tools — loaded post-boot @@ -208,9 +200,9 @@ Population Script ``scripts/tool_source/populate_store.py`` is a thin CLI wrapper over ``galaxy.tools.source_store.populator.main``. It loads only the Galaxy -config and calls ``build_tool_source_store(config)`` — the standalone store -builds the datatypes registry for converter discovery but does not initialize -the Galaxy model. Tools are parsed in a +config and calls ``build_tool_source_store(config)``. Converter discovery builds +the datatypes registry, but the standalone process does not initialize the Galaxy +model. Tools are parsed in a ``ThreadPoolExecutor`` (``--parallel``, default 4 workers); each tool is matched to its source path and carried forward when its raw file hash is unchanged (``--incremental``, the default). Once the JSON index is committed the @@ -224,17 +216,21 @@ files are re-parsed, the store is updated, and a single exchange. ``--watch-polling`` switches to ``PollingObserver`` for NFS/CVMFS/network filesystems where inotify is unreliable. -The control task handler lives in ``galaxy.queue_worker.reload_tool_source_cache`` -and is wired into the ``control_message_to_task`` map. Each Galaxy process -that receives the message: +The broadcast is the populator's half of the contract: it publishes +``reload_tool_source_cache`` so peer processes can drop their stale index +view. The control task handler lives in +``galaxy.queue_worker.reload_tool_source_cache`` and is wired into the +``control_message_to_task`` map. Each Galaxy process that receives the +message: 1. Calls ``LazyToolBox.invalidate_index_cache()`` (drops the in-memory index reference so the next access reloads from the store). 2. Calls ``ToolSourceStore.invalidate_index_cache()`` on the store itself. -Note that the LRU cache of fully constructed ``Tool`` objects is not -flushed by reload — only the index is invalidated. Stale ``Tool`` instances -are evicted naturally as new ones are loaded. +Reload also refreshes already-materialised tools: entries whose source hash +changed have their LRU entries, stubs, and registered ``Tool`` objects +purged, so the next access re-materialises from the new source. Unchanged +entries keep their cached ``Tool`` objects. Batch Endpoint Integration -------------------------- @@ -259,8 +255,9 @@ single materialise chokepoint) must not move across any of these endpoints. ``LazyToolboxSearch`` (``tools/search/__init__.py``) queries the whoosh index of *every* configured store — the default plus each named per-conf store — via ``ToolWhooshIndex.search_scored``, then merges the per-store hit lists by -BM25 score. A tool served from a named store is therefore reachable through -``/api/tools?q=`` even though its source lives outside the default store. +BM25 score and post-filters them to the requested panel view. A tool served +from a named store is therefore reachable through ``/api/tools?q=`` even +though its source lives outside the default store. App Wiring ----------