Add pluggable tool source store infrastructure

Introduces galaxy.tool_source_store: a queryable store of parsed,
macro-expanded tool sources plus a pre-computed ToolIndex of the metadata
batch consumers need (panel section, labels, EDAM, requirements, test
counts, shed provenance).

- Backends: database (store-owned tool_source_record + tool_index tables,
  migration f5a73c8b9d12), sqlalchemy (read-only SQLite bundles, e.g.
  CVMFS-shipped), and a composite that layers named stores over the
  default with read fall-through.
- scripts/tool_source/populate_store.py walks tool_conf files and writes
  sources + index + whoosh search index; --watch updates the store on
  file changes and broadcasts reload_tool_source_cache. The populator is
  the single writer; consumers are read-only.
- Per-conf opt-in via store="name" on a toolbox conf root, resolved
  against the tool_source_stores catalog in galaxy.yml.
- The tool_source_record table is deliberately separate from tool_source,
  whose rows belong to the job-request path and carry a raw-string
  payload contract.

Nothing reads from the store yet — an on-demand-loading toolbox consuming
it is follow-up work (#22633). Extracted from that PR.
This commit is contained in:
mvdbeek
2026-07-13 10:26:57 +02:00
parent 023021ba46
commit 8162853207
36 changed files with 5819 additions and 1 deletions
+1
View File
@@ -0,0 +1 @@
# Tool source management scripts
+22
View File
@@ -0,0 +1,22 @@
#!/usr/bin/env python
"""Standalone CLI entry for the tool source store populator.
Thin shim: extends ``sys.path`` so ``galaxy.*`` resolves when the script is
invoked directly (no venv-installed galaxy package required), then delegates
to :func:`galaxy.tool_source_store.populator.main`. All flags, including
``--watch``, are documented on that function's argparse setup.
In-process callers should import the populator module directly rather than
shelling out to this script.
"""
import sys
from pathlib import Path
galaxy_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(galaxy_root / "lib"))
from galaxy.tool_source_store.populator import main
if __name__ == "__main__":
main()