galaxy/doc/source/dev/tool_source_storage.rst

Tool Source Storage Architecture
================================

This document describes the architecture of the tool source storage subsystem:
the store backends, the populator, and the index they build. For operator-facing
setup and configuration, see :doc:`/admin/tool_source_storage`.

A toolbox that consumes this store to load tools on demand is planned as
follow-up work; the pieces documented here are the storage layer it will build on.

Goals
-----

The traditional ``ToolBox`` parses every tool XML at startup, builds full
``Tool`` objects, and keeps them all in memory. With thousands of tools that
scales poorly: slow boot, large per-process RSS, and expensive worker reloads.

The tool source storage subsystem moves that parsing work out of the request
path:

- A separate process (``populate_store.py``) parses tools once and persists
  the canonical, macro-expanded source plus a lightweight metadata index.
- The store and index are laid out so a consumer can load only the index at
  startup and materialize ``Tool`` objects on demand, instead of parsing the
  full tree in-process. That consumer is the planned follow-up toolbox.

Module Layout
-------------

::

    lib/galaxy/tools/source_store/
      __init__.py        Public re-exports
      interface.py       ToolSourceStore ABC and StoredToolSource
      factory.py         Store construction from Galaxy configuration
      sqlalchemy.py      SqlAlchemyToolSourceStore (any SQLAlchemy URL)
      composite.py       CompositeToolSourceStore (per-conf routing, merged index)
      index.py           ToolIndex, ToolIndexEntry (the lightweight metadata)
      search.py          ToolWhooshIndex (Whoosh search index built from a ToolIndex)
      discover.py        discover_tools() — conf walk without booting a ToolBox
      populator.py       Population + watch logic (parse, store, index, broadcast)
      freshness.py       Optional external freshness probes
      watcher.py         Filesystem watch support

    scripts/tool_source/populate_store.py    Thin CLI wrapper over populator.main

The same ``populator.main`` is registered as the
``galaxy-populate-tool-source-store`` console script in the ``galaxy-app``
package metadata (``packages/app/pyproject.toml``).

Data Model
----------

Two persistence concepts:

**StoredToolSource** — the canonical macro-expanded XML/YAML for a tool,
keyed by SHA-256 of the expanded content. Multiple versions of the same
``tool_id`` coexist as separate hashes. The store keeps its own schema in a
standalone database (a SQLite file by default, any SQLAlchemy URL for shared
deployments) — deliberately outside Galaxy's database: the store is a
rebuildable cache and does not participate in Galaxy's migrations or session
lifecycle.

**ToolIndex** — a Pydantic model containing one default ``ToolIndexEntry`` per tool
plus its versioned and panel-placement projections,
holding everything a store consumer needs (id, name, description, panel section,
labels, EDAM, requirements, container info, test counts, hidden/disabled,
shed metadata). The index is serialized and gzip-compressed as a blob.

The schema is auto-created on first open; ``tool_index`` holds a single
row per index version.

Backend Abstraction
-------------------

``ToolSourceStore`` (in ``tools/source_store/interface.py``) is an ABC defining:

- ``store/get/exists/delete/list_all/get_by_tool_id/count`` — per-tool source
  operations, all keyed by content hash.
- ``store_index/load_index/update_index_entry`` — index operations.
- ``get_stats()`` — backend-specific stats (count, size, backend name).

``build_tool_source_store(config)`` is the only entry point used
by Galaxy. It builds the default store from
``config.tool_source_database_connection`` and uses the same SQLAlchemy-backed
store implementation for all configured URIs. ``ConfigurationError`` is raised
for missing required settings and is allowed to propagate up so
misconfiguration fails fast at startup.

The ABC defines a ``read_only: bool`` class attribute (default ``False``).
``ReadOnlyStoreError`` is raised by mutating methods of stores that opted
in. The populator, watch reload, and composite all consult this flag to
route around read-only members rather than crashing.

Per-conf composition
^^^^^^^^^^^^^^^^^^^^

If any tool_conf carries a top-level ``store="..."`` attribute (XML root)
or ``store: ...`` key (YAML), ``build_tool_source_store`` instantiates
the referenced named stores from ``config.tool_source_stores`` and wraps
them with the writable default in a :class:`CompositeToolSourceStore`.

The composite implements the same ``ToolSourceStore`` interface, so store
consumers stay completely unaware of the multi-store layout:

- **Reads** iterate ``[per-conf members..., default]`` in order; first
  hit wins. ``count`` and ``list_all`` dedupe across members.
- **Writes** always land on the designated default. The default may not
  itself be ``read_only``; that's enforced at construction.
- ``load_index()`` calls each member's ``load_index()`` and folds the
  entries into a single :class:`ToolIndex`. Earlier members shadow later
  ones on tool-id collisions; ``by_section`` is unioned; ``built_at``
  takes the most recent value.
- ``invalidate_index_cache()`` fans out so a single Kombu reload hits
  every member.
- ``store_to(name, ...)`` lets the populator address a specific member by
  name without going through composite write routing.

When no tool_conf opts in, ``build_tool_source_store`` returns the
default store directly — the composite path is zero-cost for the common
case.

The ``sqlalchemy`` backend (``sqlalchemy.py``) was added to make this
useful for CVMFS: a single self-contained ``.sqlite`` file, opened with
its own SQLAlchemy ``MetaData`` (independent of ``galaxy.model``) so the
file is portable, and openable with a SQLite URI such as
``sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true``
for read-only mounts. Despite the name, the backend is not sqlite-specific -
pass any SQLAlchemy URL (Postgres, MySQL, ...). Auto schema creation runs on
first open; on remote backends operators may prefer to manage migrations
explicitly.

Per-conf populator routing
^^^^^^^^^^^^^^^^^^^^^^^^^^

``scripts/tool_source/populate_store.py`` is per-conf aware. It reads
``parse_store_name()`` for each tool_conf, builds every named store plus
the default, and routes each ``DiscoveredTool.path`` to the store its
conf points at. By default it populates *every* writable store in one
run; ``--target NAME`` restricts to a single store and raises
``ReadOnlyStoreError`` if that store is read-only. Tools whose target is
read-only in default mode are silently skipped (the bundle is treated as
authoritative for those entries).

Discovery
---------

``galaxy.tools.source_store.discover.discover_tools`` walks tool config files
and yields ``DiscoveredTool`` records without booting a full ``ToolBox``. It is
used by:

- the populator to find tools to parse and store.
- watch mode to know which directories to monitor.
- callers that compare on-disk confs against the indexed tool set.

Pulling discovery out of ``ToolBox`` was deliberate: the populator must run
*without* a full app (or even a running Galaxy), and the watch mode must run in
a long-lived loop with no Galaxy process at all.

Population Script
-----------------

``scripts/tool_source/populate_store.py`` is a thin CLI wrapper over
``galaxy.tools.source_store.populator.main``. It loads only the Galaxy
config and calls ``build_tool_source_store(config)``. Converter discovery builds
the datatypes registry, but the standalone process does not initialize the Galaxy
model. Tools are parsed in a
``ThreadPoolExecutor`` (``--parallel``, default 4 workers); each tool is
matched to its source path and carried forward when its raw file hash is unchanged
(``--incremental``, the default). Once the JSON index is committed the
populator rebuilds the Whoosh search index (``search.py``) so ranked tool
search stays in sync with the stored sources.

Watch mode (``--watch``) uses ``watchdog`` to monitor every directory yielded
by ``discover_tools``. File events are debounced (default 2 s), the changed
files are re-parsed, the store is updated, and a single
``reload_tool_source_cache`` Kombu control task is published on the Galaxy
exchange. ``--watch-polling`` switches to ``PollingObserver`` for
NFS/CVMFS/network filesystems where inotify is unreliable.

The broadcast is the populator's half of the contract: it publishes
``reload_tool_source_cache`` so peer processes can drop their stale index
view. The control-task handler that consumes the message lands with the
follow-up toolbox.

Design Notes
------------

**Why a separate index instead of always querying the store?** A consumer
needs O(N) access to N entries; doing that against the backing store on every
request is a latency hit. Keeping the index in-process and only paying for
invalidation on reload is the better tradeoff.

**Why an out-of-process populator?** Parsing tools and computing macro
expansions is expensive and shouldn't block worker startup. Keeping the
populator separate also lets it run on a single host while many web workers
share the resulting store.

**Why hash-keyed storage?** Content-addressed storage gives us cheap
deduplication across versions and shed installations, and idempotent
incremental updates: re-running the populator over an unchanged tree is
effectively a no-op.

Testing
-------

- Store unit tests: ``test/unit/app/tools/source_store/`` exercises each backend
  through the ``ToolSourceStore`` interface (``test_stores.py``,
  ``test_sqlite_store.py``, ``test_composite_store.py``,
  ``test_index_versions.py``).
- Populator/discovery unit tests: ``test/unit/scripts/tool_source/``
  (``test_populate_store.py``, ``test_discover.py``,
  ``test_build_index_entry.py``, ``test_whoosh_dir.py``). These use fakes
  (not mocks) of ``ToolSourceStore`` so behavior is verified against the real
  interface.