Adopt #23067's store infrastructure wholesale where it is canonical, keeping
only genuine lazy-toolbox additions as deltas on top:
- Store package: adopt #23067's factory.py/interface.py split and facade
__init__.py, and its URL-only SqlAlchemyToolSourceStore. Re-apply lazy-only
deltas — ToolIndexEntry panel-contract fields (icon/xrefs/model_class/
form_style/is_workflow_compatible/source_path) and data_manager_id;
composite per-version index merge; scored multi-store whoosh search
(search_scored + tool_tags field); populator panel-contract derivation via
expand_ontology_data + biotools and data_manager_id stamping; data-manager /
converter discovery in discover.py; benchmarks.py.
- Config: replace tool_source_store + tool_source_disk_path with the single
SQLAlchemy URI tool_source_database_connection (defaulted in config/__init__.py,
validated via try_parsing, schema attr added), adopt #23067's tool_source_stores
wording, and keep the branch-only use_lazy_toolbox / lazy_toolbox_cache_size
options. Regenerated galaxy.yml.sample, galaxy_options.rst, and the schema-type
stub. galaxy_mock uses tool_source_database_connection.
- Docs: adopt #23067's tool_source_storage.rst (admin + dev) as the base and
re-add the lazy sections (LazyToolBox, batch-endpoint integration,
materialisation-count guard, LazyToolboxSearch multi-store search, benchmarks).
- Tests: adopt #23067's store + scripts unit tests; re-add the ours-only
composite entries_by_version merge test, data-manager discovery /
build-index tests, and multi-store search test, all on the URI config.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The store is a rebuildable, content-addressed cache — it does not need
Galaxy's database, and living there was the source of every subtle
lifecycle mechanism in the subsystem: store.commit() committed whatever
request or job transaction was in flight, reads needed a private-Session
isolation hack, the LazyToolBox constructor carried a post-init commit to
avoid queue-worker idle-in-transaction locks, shutdown needed a haltable
ordered before engine disposal, and a sa_session parameter threaded
through the factory, populator, watcher and CLI.
Standalone stores are now the only kind: the default is a SQLite file
under tool_source_disk_path (any SQLAlchemy URL via tool_source_stores
for shared multi-host deployments), committing per write on its own
engine. The tool_source_record/tool_index tables, their migration, the
model classes and the whole session plumbing are gone, and the store is
only initialized when use_lazy_toolbox is enabled — default deployments
no longer touch any of this at boot. The ToolSourceStore ABC loses
commit() (every backend persists per write). The dev/watch-mode watcher
now populates per file without broadcasting and keeps its single
debounced reload notification for the batch.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The LazyTool docstring claimed strict-by-default with a nonexistent
LAZY_TOOL_PERMISSIVE toggle (the code is permissive by default, toggled
by LAZY_TOOL_STRICT=1). The dev doc described a Redis backend, an API
layer (api/tool_sources.py, ToolAPICache), a _init_lazy_toolbox
mechanism, and an auto-enable behavior that don't exist, plus pre-move
module paths. Also drop change-narration comments referencing removed
intermediate code, an unused SqliteToolSourceStore alias, and hoist the
populator's local os/biotools imports now that the module lives inside
galaxy.tools.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The job-request path creates and reads tool_source rows whose source
column carries the raw source string; the store was writing dict-shaped
payloads into the same table, and once identity hashes aligned the job
path could resolve a store row and fail pydantic validation (500 on
every request-style tool execution under lazy mode).
Give the store its own content-addressed table with real columns —
get_by_tool_id / get_by_source_path become indexed queries instead of
full-table JSON scans, populator pruning can never delete rows the job
path references, and the identity_hash coupling (and the legacy
__tool_index__ sentinel-row fallback) disappear.
- doc/source/admin/tool_source_storage.rst: operator-facing guide
covering backend choice, populate_store usage, watch mode, and the
per-conf composite story.
- doc/source/dev/tool_source_storage.rst: developer-facing
architecture overview of the store, LazyToolBox, and the batch
endpoint integration.
The store is a rebuildable, content-addressed cache — it does not need
Galaxy's database, and living there was the source of every subtle
lifecycle mechanism in the subsystem: store.commit() committed whatever
request or job transaction was in flight, reads needed a private-Session
isolation hack, and a sa_session parameter threaded through the factory,
populator, watcher and CLI.
Standalone stores are now the only kind: the default is a SQLite file
under tool_source_disk_path (any SQLAlchemy URL via tool_source_stores
for shared multi-host deployments), committing per write on its own
engine. The tool_source_record/tool_index tables, their migration, the
model classes and the whole session plumbing are gone. The ToolSourceStore
ABC loses commit() (every backend persists per write). The dev/watch-mode
watcher now populates per file without broadcasting and keeps its single
debounced reload notification for the batch.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink. Also drop an unused back-compat alias and
fix stale doc references.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Introduces galaxy.tool_source_store: a queryable store of parsed,
macro-expanded tool sources plus a pre-computed ToolIndex of the metadata
batch consumers need (panel section, labels, EDAM, requirements, test
counts, shed provenance).
- Backends: database (store-owned tool_source_record + tool_index tables,
migration f5a73c8b9d12), sqlalchemy (read-only SQLite bundles, e.g.
CVMFS-shipped), and a composite that layers named stores over the
default with read fall-through.
- scripts/tool_source/populate_store.py walks tool_conf files and writes
sources + index + whoosh search index; --watch updates the store on
file changes and broadcasts reload_tool_source_cache. The populator is
the single writer; consumers are read-only.
- Per-conf opt-in via store="name" on a toolbox conf root, resolved
against the tool_source_stores catalog in galaxy.yml.
- The tool_source_record table is deliberately separate from tool_source,
whose rows belong to the job-request path and carry a raw-string
payload contract.
Nothing reads from the store yet — an on-demand-loading toolbox consuming
it is follow-up work (#22633). Extracted from that PR.
mvdbeek