_writable_store_index_needs_population compared the set of stored row
hashes against the set of index-referenced hashes symmetrically. The
store is append-only (only reconcile_index prunes), so rows orphaned by
a content change, an ad-hoc self-heal superseded by conf context, or an
uninstall accumulate legitimately - and one orphan made the comparison
unequal forever, turning every boot and every post-install reload into a
full inline repopulate and permanently defeating the freshness-token
scan skip (observed: 5 inline populates across one shed-install
integration test, 3 orphaned rows as the stable delta).
Check the direction that is actually an invariant: every source_hash the
index references must resolve to a stored row, or materialising that
tool would fail. Orphaned rows are logged at debug as prune candidates.
Measured on the same test: 5 inline populates -> 2 (cold boot + the
legitimate freshness-token change from the install itself).
The from-index fast panel init registered stubs into the live panel only,
keyed per collapsed entry id. Three fidelity gaps against the eager conf
walk broke the first lazy CI runs carrying it:
1. The integrated tool panel stayed empty on fresh boots. It is what
_load_tool_panel_views renders EDAM/static views from (and what
_save_integrated_tool_panel persists), so every panel view rendered
empty: the EDAM toolbox tests, all panel_views_dir integration tests,
and /api/users favorite EDAM operations 404d.
2. ToolIndex.entries collapses same-id tools to one winner, so it cannot
represent the panel: a top-level sectionless tool (upload1 in the
functional conf - test_index), a tool id referenced both top-level and
inside a section (multiple_versions), or expanded-content twins each
lost whichever placement their winning entry didn't carry.
3. Runtime shed installs lost their panel position and repository
metadata across the reload that follows: a full repopulate reset
placement order to conf order (eager inserts at the section head via
update_or_append), and the unchanged-file carry kept serving the
ad-hoc self-heal entry written during metadata generation, which has
no shed conf context - so the rendered panel entry lacked
tool_shed_repository.
Fixes, index side: ToolIndexEntry.in_panel marks conf tool items
(converter and data-manager discoveries are indexed for lookups but never
placed); ToolIndex.panel_items records one placement per conf reference
in discovery order, appended by add_entry (keyed on tool id + section,
composite-merged across stores); discovery yields every conf reference of
a path instead of dedup-ing (the store layer is content-addressed and
absorbs repeats); the unchanged-file carry reconciles conf-level context
(section, shed repository coordinates) from the current discovery; a full
rebuild merges the previous index's placement order (the placements list
is the panel-order memory, the role integrated_tool_panel.xml plays for
eager) and partial updates insert new placements at the head of their
section, matching the eager runtime insert.
Toolbox side: _init_tools_from_index registers all stubs unplaced and
replays placements through a shared _place_stub that mirrors every
placement into the integrated panel (hidden included, like the eager
__add_tool always did) and slots visible ones into the live panel.
Verified: lazy integration test_panel_views + test_edam_toolbox 18/18
(previously 16 of the lazy CI dispatch failures), lazy api test_index +
test_favorite_edam_operations, eager panel views 16/16, 153 unit, tox
lint/format/mypy. The index schema hash changes, so existing persisted
indexes rebuild once on upgrade.
The DRY refactor that unified tool-path resolution onto the toolbox's
resolve_tool_path (which expands ${tool_conf_dir} but leaves a bare
relative path relative, no abspath) changed what _collect_new_tool_paths
yields for a relative tool_path. discover routes through the same
function, so the two still agree and the partial-populate filter matches;
only the test's hard-coded absolute expectation was stale. Derive the
expected paths from resolve_tool_path so the test pins the consistency
contract instead of an implementation detail.
_index_versions_for prefix-scanned every index entry per lineage lookup,
and the EDAM panel views resolve a lineage per shed tool — ~11.5k scans
over ~9k entries. Profiled as the single largest toolbox boot cost:
18.0s cumulative, and _configure_toolbox drops from 25.8s to 11.7s
(under cProfile) with the scan replaced by one precomputed
versionless-guid -> (entry_id, version) map.
The map is cached against the ToolIndex object identity plus entry
count — reloads swap the object, registrations/removals mutate
membership in place — and in-place removals also reset it explicitly
(a pop-then-add could keep the count stable while changing membership).
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Walking a CVMFS shed conf, the eager _load_tool_tag_set paid two per-tool
costs the lazy toolbox doesn't need: an os.path.exists stat (network
round trip on CVMFS — minutes over ~10k tools) and, since conf-provided
repositories have no install-DB row by design, one 'Attempted to load
tool shed tool, but the repository ... was not found in database'
warning per tool per boot.
Two overridable hooks in AbstractToolBox, both eager-behavior-preserving:
_tool_file_on_disk (the existence gate) and _missing_repository_log_level
(the warning's severity). LazyToolBox answers both from an identity-cached
set of indexed source_paths: the populator established existence when it
stored the source and materialisation reads the raw source from the
store, so the stat proves nothing; the missing install-DB row is expected,
so it logs at debug. Unindexed paths keep exact eager behavior.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Comparing a stamped CVMFS revision at boot is unsound for the canonical
publishing workflow: the populator runs inside the publish transaction
at revision N, the publish commits as N+1, so every client reads
current != stamped forever. The revision can't record the bump it is
part of.
It's also unnecessary. A read-only store ships in the same transaction
as the tools it indexes, so a schema-valid index (load_index already
gates on INDEX_SCHEMA_HASH) is authoritative by construction — trust
it. index_is_fresh on a read-only store now returns whether the index
loads; False means no loadable index at all, which the composite warns
about and continues (only the publisher can fix it).
freshness: cvmfs is thereby watcher-only: the watcher compares current
revision against the last value seen on this client — transition
detection, immune to the off-by-one. The writable default store keeps
the tool_confs stamp-and-compare, where populate and boot share a host
and the comparison is meaningful.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The populator now stamps a freshness token into each writable store's
persisted ToolIndex, captured before the tree walk. At boot,
_index_needs_population re-probes and a match skips the conf walk and
its per-file existence stats entirely — the store provably covers the
current tree. A mismatch (or no probe) falls back to the batched
coverage scan as before, so a wrong token can only cost work, never
correctness.
Two probe kinds (galaxy.tools.source_store.freshness):
- tool_confs: md5 over tool/data-manager conf contents plus recursive
directory mtimes of tool_dir entries. Detects the same
addition/removal drift the coverage scan does, at ~a-dozen-file cost.
Wired to the default store automatically.
- cvmfs: the CernVM-FS repository revision via the user.revision
extended attribute on the mount point — one syscall covers the whole
repository. Opt-in per named store (freshness: cvmfs, optional
freshness_path), since a store published on CVMFS in the same
transaction as its tools makes a matching revision a hard consistency
proof.
Composite aggregation: a stale writable member triggers the populator;
a stale read-only member only warns (the publisher owns repopulation);
a probe-less member downgrades to the coverage scan.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The only local import left is lazy_toolbox's send_control_task, which
is a genuine cycle: galaxy.queue_worker imports LazyToolBox at module
level. Everything else - app boot, queue_worker, tool_panel_manager,
services/tools, search, populator, benchmarks, uses_shed and the test
modules - now imports at the top; the claimed app/tools circularity
never existed (galaxy.app already imports galaxy.tools).
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
walk_tool_directories() in tool_util.toolbox.base is now the single
implementation of the hidden/private-skipping directory walk;
AbstractToolBox.__watch_directory consumes it (keeping per-directory
watcher registration) and discover drops its parallel copy.
Two deliberate behaviour touch-ups on the eager side: entries are now
walked in sorted order (deterministic panel order for tool_dir tags),
and a directory's watcher now registers when any tool in it loaded
rather than only when the last candidate did.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
GALAXY_CONFIG_OVERRIDE_USE_LAZY_TOOLBOX=true (the lazy workflow_dispatch
env) trumps per-class config kwds, so TestEagerBootSkipsStore cannot get
an eager boot on those runs and the store it finds is legitimate.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Adopt #23067's store infrastructure wholesale where it is canonical, keeping
only genuine lazy-toolbox additions as deltas on top:
- Store package: adopt #23067's factory.py/interface.py split and facade
__init__.py, and its URL-only SqlAlchemyToolSourceStore. Re-apply lazy-only
deltas — ToolIndexEntry panel-contract fields (icon/xrefs/model_class/
form_style/is_workflow_compatible/source_path) and data_manager_id;
composite per-version index merge; scored multi-store whoosh search
(search_scored + tool_tags field); populator panel-contract derivation via
expand_ontology_data + biotools and data_manager_id stamping; data-manager /
converter discovery in discover.py; benchmarks.py.
- Config: replace tool_source_store + tool_source_disk_path with the single
SQLAlchemy URI tool_source_database_connection (defaulted in config/__init__.py,
validated via try_parsing, schema attr added), adopt #23067's tool_source_stores
wording, and keep the branch-only use_lazy_toolbox / lazy_toolbox_cache_size
options. Regenerated galaxy.yml.sample, galaxy_options.rst, and the schema-type
stub. galaxy_mock uses tool_source_database_connection.
- Docs: adopt #23067's tool_source_storage.rst (admin + dev) as the base and
re-add the lazy sections (LazyToolBox, batch-endpoint integration,
materialisation-count guard, LazyToolboxSearch multi-store search, benchmarks).
- Tests: adopt #23067's store + scripts unit tests; re-add the ours-only
composite entries_by_version merge test, data-manager discovery /
build-index tests, and multi-store search test, all on the URI config.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The store is a rebuildable, content-addressed cache — it does not need
Galaxy's database, and living there was the source of every subtle
lifecycle mechanism in the subsystem: store.commit() committed whatever
request or job transaction was in flight, reads needed a private-Session
isolation hack, the LazyToolBox constructor carried a post-init commit to
avoid queue-worker idle-in-transaction locks, shutdown needed a haltable
ordered before engine disposal, and a sa_session parameter threaded
through the factory, populator, watcher and CLI.
Standalone stores are now the only kind: the default is a SQLite file
under tool_source_disk_path (any SQLAlchemy URL via tool_source_stores
for shared multi-host deployments), committing per write on its own
engine. The tool_source_record/tool_index tables, their migration, the
model classes and the whole session plumbing are gone, and the store is
only initialized when use_lazy_toolbox is enabled — default deployments
no longer touch any of this at boot. The ToolSourceStore ABC loses
commit() (every backend persists per write). The dev/watch-mode watcher
now populates per file without broadcasting and keeps its single
debounced reload notification for the batch.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The populator writes one whoosh index per store, but search only opened
the default store's directory — tools served from a named store
(per-conf store="...") listed and ran fine yet never appeared in
/api/tools?q=. Search each configured store's index and merge hits by
BM25 score (ToolWhooshIndex.search_scored); scores from different
indexes aren't strictly comparable, but interleaving beats invisible
tools, and resolve_search_hit drops ids foreign to this toolbox.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The merge loop copied only entries and by_section, leaving the merged
index's per-version map empty — ToolIndex.get(tool_id, tool_version)
resolves exact versions through entries_by_version, so every non-newest
version of a member-store tool was unreachable (wrong default served,
get_all_versions empty). Merge per (id, version) with the same
earlier-member-wins rule.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Two gaps found by review: (1) invalidate_index_cache was add/update
only — it registered new index entries as stubs and refreshed existing
ones, but never dropped registrations whose ids vanished from the
reloaded index, so a peer-process uninstall left this process serving
the stale stub via the eager get_tool fall-through; (2) remove_tool_by_id
persisted the index removal locally but never broadcast, so peers had no
reason to reload until an unrelated populate ran.
Extract the in-memory removal bookkeeping into _remove_tool_in_memory,
diff the previous index against the reloaded one in
invalidate_index_cache and pop vanished ids (diff-based, so internal
and dynamic tools that never enter the index are untouched), and send
reload_tool_source_cache (noop_self) after persisting a removal.
_register_lazy_entry now registers the old-id bucket even when
old_id == tool_id, matching the eager __add_tool — super's removal
unconditionally pops that bucket.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Add a materialise counter to LazyToolBox (_lazy_materialize_count, bumped
at the single _create_tool_from_stored_source chokepoint) and an
integration test that hits every batch reader the client uses — flat
listing, panel walk, panel views, tests-summary, all_requirements, and
search — asserting a zero materialise delta across them.
This catches accidental whole-toolbox materialisation that LAZY_TOOL_STRICT
can't: a tool-filter or _MATERIALIZE_OK attribute read inside a full-index
sweep parses tools without ever hitting the strict __getattr__ path. The
test was written against the search-materialisation regression fixed in
the previous commit and fails without it.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
ToolsService.search_tools resolved every whoosh hit through
_get_tool -> toolbox.get_tool, which in lazy mode loads the tool from the
store on demand. A single /api/tools?q= query therefore parsed one Tool
per hit. The populator-owned whoosh index also carries ids that were
never loaded into this toolbox (tool_conf.xml.sample alone lists ~150
legacy tools whose files aren't present), so lazy search both parsed and
returned tools the eager path skips: eager whoosh only holds loaded
tools, and eager _get_tool returns None for un-loaded ids.
Add LazyToolBox.resolve_search_hit, which looks the hit up in the
registered stubs (_tools_by_id, plus the shed short-id map) and returns
None for anything not part of this toolbox — no parse. search_tools uses
it in lazy mode and keeps the eager _get_tool path unchanged. Result:
search stays O(hits) index reads and matches eager scoping.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
AgentTools.get_tool_categories iterates toolbox.tools() and reads
tool.get_panel_section()[1] on every visible tool. get_panel_section was
not on the LazyTool stub surface, so under the permissive default that
sweep materialised the entire toolbox to build the category list (and
under LAZY_TOOL_STRICT=1 it would now raise).
The populator already stamps panel_section_id / panel_section_name onto
ToolIndexEntry, so forward the call off the entry — matching the
(section_id, section_name) / (None, None) contract of
Tool.get_panel_section — and keep the walk lazy.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The partial populate filters discovered tools on exact path strings.
_collect_new_tool_paths joined the shed conf dict's raw tool_path —
relative for typical shed_tool_conf files — while discovery resolves a
relative tool_path against the conf file's directory and returns an
absolute path. The strings never matched, so the enriched conf-driven
discovery was filtered out and the ad-hoc synthesis populated a
metadata-poor entry instead: the panel served fastp without
tool_shed_repository (test_only_latest_version_in_panel_fastp), and the
store grew a duplicate row under the unresolved path string.
Route both through discover.resolve_tool_path (now public) and thread
each tool's guid so even a genuinely-unreachable path yields a
guid-keyed entry.
The data_manager_id stamp landed in index.entries only —
update_index_entry never touched entries_by_version, and the two maps
serialize independently. Job-time materialise passes a tool_version, so
it resolves through the per-version map and got the unstamped entry
back after the next index reload (populate broadcasts one constantly):
exec_after_process still failed with 'Invalid data manager (<guid>)'.
Both backends now route updates through ToolIndex.add_entry, which
keeps the maps in step (and stops an older-version update from
clobbering the newer default entry). create_tool additionally stamps
the in-memory per-version twin, which is a distinct object after a
from_dict reload.
_register_new_index_entries_as_stubs skipped every tool already in
_tools_by_id, so a stub minted from an install-time ad-hoc entry kept
serving that metadata-poor entry after the conf-driven populate
enriched the index — the fastp panel payload was missing
tool_shed_repository (test_only_latest_version_in_panel_fastp).
Swap the entry behind the existing stub instead: the panel and
registries hold the stub object, and _overrides survive.
Shed-installed data managers load their tool at install time, before
shed_data_manager_conf.xml exists — the self-healed entry carries
data_manager_id=None, materialise never restores the conf id, and
DataManagerTool.exec_after_process fails with 'Invalid data manager
(<guid>)' at job finish (all six data-manager integration tests).
DataManager._load_tool already passes data_manager_id through
load_hidden_tool; create_tool now stamps it onto the resolved entry and
persists it, so job handlers materialising from the shared index
resolve the registry correctly.
remove_tool_by_id popped the entry from every in-memory map, but the
persisted singleton index still carried it. Every populate broadcasts a
reload_tool_source_cache invalidation, so the very next reload handed
the entry back and the uninstalled tool resolved again
(test_repository_uninstall expected err_msg, got the full tool).
ToolSourceStore grows remove_index_entry — the uninstall counterpart of
update_index_entry — and remove_tool_by_id writes the removal through.
Discovery stamped hidden=True on data manager tools, but eager's
load_hidden_tool only means 'not in the panel' — Tool.hidden stays
falsy, and the flat /api/tools?in_panel=false listing filters hidden
tools, so the lazy listing dropped them
(test_data_manager_async_submission_with_mismatched_conf_id).
Also gate LazyTool.allow_user_access on tool_type 'manage_data' — the
value DataManagerTool actually carries; 'data_manager' never occurs, so
the admin gate on stubs never fired and lazy was more permissive than
eager.
Shed installs never survived lazy mode: metadata generation
(installed_repository_metadata_manager.get_repository_tools_tups) loads
each freshly cloned tool before add_to_tool_panel persists the conf and
populates the index, so create_tool raised on every install and the
repository landed broken (data managers, workflow test tools, fastp
panel tests).
A miss for a file that exists on disk now populates that single path —
still through the single-writer populator; populate_store_inline
synthesizes a DiscoveredTool for requested paths outside every conf and
threads the caller's guid so the ad-hoc entry is keyed like its eventual
conf-driven replacement — then reloads the index and retries. Misses
for files that don't exist still raise.
Six test/functional tools (upload.xml, export_remote.xml,
ucsc_tablebrowser.xml, parse_values_from_file.xml, catWrapper.xml,
for_tours/filtering.xml) expand to byte-identical content as their
root tools/ counterparts. With hash unique, whichever file populates
first owns the row; the twin's populate is skipped and
get_by_source_path(second path) returns nothing, so LazyToolBox.create_tool
raises and the toolbox silently drops the tool — upload breaks for any
instance whose config discovers both trees (CI integration shards do).
hash becomes a non-unique index (migration amended in place — unreleased),
store() upserts by source_path, the populator skips only when the same
path already holds the same content, and get/exists/delete tolerate
multiple rows per hash.
ToolIndexEntry gains icon, xrefs, model_class, form_style and
is_workflow_compatible, derived at populate time the same way
Tool.__init__/to_dict derive them (tool_type class registry, root
workflow_compatible attribute, page count). EDAM fields now go through
expand_ontology_data, picking up the curated mapping overrides and
legacy bio.tools xrefs the direct parse_edam_* calls missed.
LazyTool.to_panel_entry emits the full client payload -- including
tool_shed_repository (the field the panel-view integration tests
assert), versions from the registered lineage, uuid, and the
admin-gated config_file from source_path.
The populator writes through its own store instances, so after the
inline backfill the toolbox's store still served its cached pre-populate
index. Single-instance that cache is merely stale-but-identical; on a
shared database (the CI integration shards share one postgres DB, and
any multi-server deployment shares one in production) the cached index
is another instance's view and the eager walk drops every tool missing
from it — the post-reload 'no index entry for upload.xml' cascade.
Reload integration tests drive galaxy.queue_worker.reload_toolbox
directly and reproduce the failure with a foreign index row; the earlier
send_control_task-based attempts passed vacuously because the control
message never executes in this harness.
curated_tool_tags sweeps every tool for .tool_tags, which force-
materialised the whole toolbox on first /api/tags/tool_tags request and
500d whenever one indexed tool can't build (filter_data_table's
column="value" dynamic options). The tag mapping is keyed purely by
tool id, so the stub now answers without materialising.
Data manager tools are loaded post-boot via DataManagers ->
load_hidden_tool, which the discover walk missed — create_tool's index
miss aborted DataManager loading and the tools 404d under lazy mode.
discover_tools now walks data_manager_conf/shed_data_manager_conf
(path + guid resolution mirroring DataManager._load_from_element), and
the seam's source-path lookups normalize to absolute paths since data
manager tool files are joined from a possibly relative tool_path.
- AbstractToolBox gains a no-op invalidate_index_cache (LazyToolBox
overrides it); shed installs no longer depend on the subclass type.
- Annotate LazyToolboxSearch.build_index, ToolFileWatcher.observer,
populator session cast, UUID key coercion in _register_lazy_entry,
and drop stale type: ignore comments.
The lazy ``to_dict`` fast path was short-circuiting *both* the
panel-view walk (intended) *and* the ``/api/tools/{id}`` show endpoint
(unintended) — they're indistinguishable at the ``to_dict(io_details=
False, link_details=False)`` callsite. ``test_legacy_biotools_xref_
injection`` caught it: the show endpoint returned the entry-shape
dict and no longer carried ``xrefs``.
Split the two call sites at the API rather than the parameter:
* ``Tool.to_panel_entry(trans)`` — new method, returns the cheap
entry-shape dict (id/name/version/description/labels/edam/panel
section/link). ``LazyTool`` overrides it to skip materialise.
* ``AbstractToolBox.get_tool_to_dict`` calls ``to_panel_entry`` when
``tool_help=False`` (the panel-view default); ``tool_help=True``
still falls through to ``to_dict`` so the help payload renders.
* ``LazyTool.to_dict`` always materialises now — it's the
``/api/tools/{id}`` contract and must ship ``xrefs`` / ``versions``
/ ``is_workflow_compatible`` / ``tool_shed_repository`` / etc. The
materialise-failure fallback now reuses ``to_panel_entry``.
Net effect: ``/api/tool_panels/{view}`` and ``/api/tools?in_panel=
true`` stay materialise-free for the lazy toolbox; ``/api/tools/{id}``
serves the full show-endpoint dict in both modes.
With ``link_details`` no longer gating any field, the toolbox-internal
panel / listing callers stopped passing it (``f64858bf55e``).
``LazyTool.to_dict`` can now return the entry-shape dict for the
default-call case — every panel-view tool serves from the index, zero
materialise.
Materialise still fires when the caller asks for the parsed-Tool
payload: ``io_details=True`` (the ``/api/tools/{id}`` show endpoint
contract) or ``tool_help=True`` (rendered help). Failure-to-materialise
still falls back to the entry-shape dict so the show endpoint never
returns 500 when a specific tool's XML chokes the parameter factory.
The ``link`` field is derived from ``self.id`` matching what
``Tool.to_dict`` now emits unconditionally — ``/tool_runner?tool_id={id}``
for the standard tool runner.
Tests updated:
- ``test_to_dict_entry_fast_path_does_not_materialise`` asserts the
default call serves the entry dict and never invokes the callback.
- ``test_to_dict_io_details_materialises`` covers the show contract.
- ``test_to_dict_falls_back_to_entry_when_materialise_fails`` covers the
parameter-factory-failure case.
23/23 unit tests pass.
After five rounds of CI surfacing missing stub attrs and the latest
round still hitting indirect failures (workflow steps with ``module``
unset because some inner attribute access raised under strict), the
explicit ``_MATERIALIZE_OK`` set has stopped converging. The strict
guard caught the obvious surface and now slows ship velocity without
adding signal.
Flip the default: unknown attr reads on ``LazyTool`` now log a WARNING
and materialise. The strict raise is still available for debugging via
``LAZY_TOOL_STRICT=1`` — the path that surfaces ``add to the stub
surface or _MATERIALIZE_OK`` for new audit work.
``test_strict_getattr_raises_with_clear_message`` now monkey-patches
the module flag to force strict mode for the assertion. Other tests are
unchanged.
The entry-only fast path was missing fields the ``/api/tools/<id>``
show endpoint serves (``xrefs``, ``inputs`` when ``io_details=False``,
…). The flat ``/api/tools?in_panel=False`` listing — the only caller
that benefits from the fast path — already bypasses this method:
``services/tools.py:list_tools`` walks ``tool_index.list_all()`` and
calls ``entry.to_api_dict()`` directly.
Always materialise here; the entry-only dict survives as the fallback
when materialise itself raises (a tool whose XML the parameter
factory chokes on — ``upload_dataset``, ``column="value"``, …) so the
caller still gets *something* renderable rather than a 500.
Tests updated:
- ``test_to_dict_fast_path_does_not_materialise`` → renamed to
``test_to_dict_falls_back_to_entry_dict_when_materialise_fails``.
- ``test_to_dict_link_details_materialises_exactly_once`` → renamed
for accuracy; behaviour unchanged.
``tool_action`` graduated to ``_MATERIALIZE_OK`` in the previous commit
(it's read on the execute path). The strict-raise assertion needs an
attribute the stub neither forwards nor materialises;
``totally_not_a_tool_attr`` is guaranteed unaccounted-for.
``galaxy.tools.special_tools`` already enumerates Galaxy-internal tool
files loaded outside the conf walk (``load_hidden_lib_tool`` → the four
``imp_exp`` / ``data_fetch`` entries, plus ``set_metadata_tool.xml``
which the datatypes registry loads separately). The lazy refactor was
missing index entries for those, so the previous step relaxed
``LazyToolBox.create_tool`` to fall through to the eager parent on miss
to keep boot working.
Add ``hidden_lib_tool_paths()`` in ``special_tools.py``: returns the
absolute paths of every hidden-lib tool, including
``set_metadata_tool.xml``. ``SPECIAL_TOOLS`` (consumed by
``load_lib_tools``) stays unchanged so ``set_metadata_tool`` isn't
double-loaded — the new ``_EXTRA_HIDDEN_LIB_TOOLS`` dict only feeds
the populator-facing helper.
``galaxy.tool_source_store.discover.discover_tools`` walks those paths
after the conf + bundled walks, yielding ``DiscoveredTool`` entries
with ``tool_conf="<hidden-lib>"``. The populator's cold-start scan
indexes them like any other tool, so the post-boot
``load_hidden_lib_tool`` calls now hit the index branch.
With coverage complete, ``LazyToolBox.create_tool`` reverts to the
strict raise the plan called for: a miss is now a contract failure
(operator forgot to repopulate, or a new ad-hoc tool load didn't get
added to ``hidden_lib_tool_paths``). The unit test asserts the raise
again.
15/16 integration tests still pass; the remaining failure is the
pre-existing flake ``test_run_specific_version_executes_that_version``.
Three review-driven cleanups on top of commit 7ae5bf442e:
1. **``ToolSource.to_string()``** is the canonical serialiser on the
parser interface (``lib/galaxy/tool_util/parser/interface.py:450``)
and is implemented by every concrete source (XML, YAML, CWL). Use
it instead of branching on ``xml_tree`` / falling back to raw file
read — fewer special cases, correct for every source class.
2. **Drop ``populate_single_path``.** It was a heavyweight recovery
for ``load_hidden_lib_tool`` paths that aren't in any tool_conf.
The cleaner answer: on index miss, ``LazyToolBox.create_tool``
delegates to ``super().create_tool(...)``. The eager parent parses
the file and registers a real ``Tool`` (not a ``LazyTool``) — lib
tools like ``set_metadata_tool.xml`` live in memory, never enter
the populator-owned store, and re-load fresh on every boot. The
"raise on miss" contract is downgraded to "fall through to eager"
— the populator still owns every tool that *should* be in the
index, and the test seam (``test_create_tool_falls_through_to_eager_on_index_miss``)
asserts the new behaviour.
3. **Hoist inline imports** in both files. Module-level
``StoredToolSource`` / ``ToolSourceStore`` / ``ReadOnlyStoreError`` /
``DiscoveredTool`` / ``discover_tools`` / ``ToolIndex`` /
``ToolIndexEntry`` / ``ToolSection`` / ``populate_store_inline`` /
``get_toolbox_parser`` / ``get_tool_source`` so the structural deps
are visible at the top of each module. The heavy CLI-only imports
(``galaxy.config``, ``galaxy.model``, ``galaxy.datatypes.registry``,
the whoosh ``ToolSearchTuning`` / ``ToolWhooshIndex`` pair) stay
inline — they're optional, used only on populator entry points,
and would slow ``--help`` if pulled in.
14/16 integration tests pass; the two remaining failures predate this
stack (``test_run_specific_version_executes_that_version`` is the
noted pre-existing flake, ``test_default_panel_view_section_tools_use_id_list``
needs separate investigation).
The populator (cold-start auto-populate at boot, ``populate_for_paths``
on shed install) is now the single writer of the index + whoosh. The
toolbox stops parsing tool XML in the boot path.
Deleted:
- ``_persist_tool_source`` (~76 lines) — the in-toolbox fallthrough that
parsed + stored + broadcasted on index miss.
- ``_build_index_entry_from_stored`` and ``_make_index_entry`` (~120
lines combined) — only called from ``_persist_tool_source``.
- The ``hashlib`` and ``datetime`` imports they pulled in.
``create_tool`` now raises ``RuntimeError`` on index miss with a message
pointing at the populator (CLI or in-process API). Cold-start
auto-populate (commit 9) guarantees coverage at boot, so a raise here
means an out-of-band conf change without a populator run — operator
action, not silent recovery.
``lazy_toolbox.py`` shrinks from 1314 to 1188 lines.
Added a unit test asserting the raise; the existing seam tests (LazyTool
return on guid + source-path hit, ``load_tool_from_cache``, etc.) still
pass. 22/22 unit tests green.
With ``_tool_panel`` filled with ``LazyTool`` stubs by the seam-driven
walk, the parent ``AbstractToolBox.to_panel_view`` renders the default
view from ``_tool_panel_view_rendered`` (cached at boot by
``_load_tool_panel_views``). For each tool encountered it calls
``ToolSection.to_dict(only_ids=True)`` → ``tool.to_dict(link_details=False)``,
which lands in ``LazyTool.to_dict``'s entry-only fast path. No
materialisation; the lazy-custom rendering is redundant.
``_index_entry_to_api_dict`` goes away with it — the same shape now
comes from ``LazyTool.to_dict(link_details=False)``.
Also drops the leftover ``_populate_tool_registry_from_index`` block
(missed in commit 4f82c319fcc — it referenced ``_tool_section_map``
which no longer exists, and was orphaned the moment ``__init__``
stopped calling it). With its removal, ``ToolSection`` /
``panel_item_types`` are still referenced by the panel post-walk
stamping helpers, but ``EdamToolPanelView`` is now fully unused —
import dropped.
``test_default_panel_view_section_tools_use_id_list`` was asserting
that section ``tools`` lists held only id strings — that was specific
to the lazy fast path. The eager renderer emits
``ToolSectionLabel`` dicts interleaved with id strings (that's what
``ToolSection.to_dict(only_ids=True)`` produces for a section that
contains labels). The test's true intent is to prevent regression
where full *Tool* dicts appear in ``tools``; loosen the assertion to
accept id strings + ``ToolSectionLabel`` dicts but no other shape.
10/11 integration suite still passing; unit and tox clean.
Introduces the seam the conceptual-simplification plan
(``.claude/plans/witty-drifting-clock.md``) hangs the LazyToolBox refactor on:
* ``LazyTool`` — strict-by-default proxy over ``ToolIndexEntry`` that
forwards the narrow attribute surface the eager pipeline actually reads
(audited at plan time: ~28 attrs + 3 methods, zero ``isinstance(_, Tool)``
checks anywhere). Mutations (``tool.hidden = True``, shed metadata,
``tool.labels = ...``) land in ``_overrides`` and survive any later
materialise. ``to_dict(link_details=False)`` stays on the entry — only
``link_details=True`` (and the ``_MATERIALIZE_OK`` allow-list) parse XML.
Unknown attributes raise ``NotImplementedError`` with a clear message;
``LAZY_TOOL_PERMISSIVE=1`` downgrades to warn + materialise for
debugging.
* ``LazyToolBox.create_tool`` — returns a ``LazyTool`` when the index
already has the source; otherwise falls through to the eager
``ToolBox.create_tool`` and persists the freshly-parsed tool via
``_persist_tool_source`` so the next boot is a hit. Paired with
``load_tool_from_cache``/``add_tool_to_cache`` no-ops so the disk
``ToolCache`` doesn't shadow the seam.
* ``_resolve_index_entry`` — guid → ``store.get_by_source_path`` →
short-id fallback. ``_materialize_for_lazy_tool`` routes the
stub→Tool promotion through the existing
``_create_tool_from_stored_source`` + ``_register_loaded_tool`` so
``_tools_by_id`` / ``_tool_versions_by_id`` / ``_tools_by_old_id`` /
lineage all agree with the eager bookkeeping.
The seam is **dormant** in this commit: ``LazyToolBox.__init__`` still
runs its hand-written ``_init_lazy_toolbox`` mirror, so ``create_tool``
is never invoked on the boot path. A follow-up commit will flip
``__init__`` to ``super().__init__()`` and delete the duplicate
panel-parsing complex (~1500 lines per Phase 2 of the plan); this
commit is purely additive so the activation can be reviewed in
isolation.
21 unit tests cover the forwarded surface, mutator overrides
round-tripping through materialise, the ``to_dict`` fast/slow paths,
strict / permissive ``__getattr__`` behavior, and the
``create_tool`` index-hit paths (guid + source_path).
Round out the test suite that exercises LazyToolBox end-to-end against
a real boot:
- Extend ``BaseToolSourceStorageIntegrationTestCase`` helpers to hit
``/api/tools?in_panel=False`` (the index-backed listing) and to
reach the lazy ``tools_by_id`` map directly when the resolver path
isn't what's being asserted.
- New unit tests under ``test/unit/tool_source_store/`` pin the index's
version handling: default-version selection, sibling-version
discovery, and ``packaging.version`` ordering.
Tests that bind specific behaviours to specific LazyToolBox commits
land alongside the code commits that introduced them; this commit
collects the cross-cutting helpers and pure-unit coverage.
Support tool shed installs end-to-end on top of the persistent tool
source store:
- Section-install paths funnel through ``_lazy_register_tool_item`` so
each shed install is indexed alongside the panel section it lives in.
- Shed entries are keyed by full toolshed guid (not the short id) so
workflow-side ``has_tool(guid)`` lookups resolve, and short-id
references resolve back to the latest matching guid.
- ``tool_shed_repository``/``guid`` metadata is stamped on stored
entries and restored on materialisation; cross-version lineage
follows the guid so the same lineage covers every installed revision.
- ``_prune_orphaned_shed_entries`` drops persisted entries whose
backing shed_tool_conf no longer references them, so a
``reset_shed_tools`` test doesn't leak installs into the next test.
- Section re-materialisation rebuilds ``section.elems`` (shed tools
inserted at section head, ahead of local tools) instead of clearing
the existing entries; ``remove_tool_by_id`` clears
``_tool_versions_by_id`` and lineage state under the toolbox lock.
- ``reload_tool_source_cache`` broadcasts with ``noop_self=True`` so
the local process keeps its just-stored cache instead of racing the
queue-worker session.
- Configured shed/migrated tool confs are registered as
on-demand placeholders when the file is absent at boot, matching
the eager toolbox's behaviour.
Round out the index discovery so the lazy toolbox boots against the full
shipping tool set:
- ``scripts/tool_source/_discover.py`` now handles ``<tool_dir>`` panel
entries and reads YAML tool ids alongside XML, so multi-version tool
directories and YAML tools land in the index.
- ``LazyToolBox`` extends the bootstrap path to register every variant
discovered for a single tool id (was previously dropping siblings) and
uses the shared YAML id reader when chasing direct ``<tool file=...>``
references from the panel config.
- ``DatabaseToolSourceStore.store_index`` accepts the richer per-id
payload from the discovery script.
Test coverage is extended in ``test_tool_source_storage`` to assert that
the bootstrapped index sees both directory-based and YAML-defined tools.
End-to-end test that boots Galaxy through the composite store +
LazyToolBox path and verifies tools are discoverable via /api/tools,
covering the wiring exercised by all of the preceding commits.
mvdbeek