The LazyTool docstring claimed strict-by-default with a nonexistent
LAZY_TOOL_PERMISSIVE toggle (the code is permissive by default, toggled
by LAZY_TOOL_STRICT=1). The dev doc described a Redis backend, an API
layer (api/tool_sources.py, ToolAPICache), a _init_lazy_toolbox
mechanism, and an auto-enable behavior that don't exist, plus pre-move
module paths. Also drop change-narration comments referencing removed
intermediate code, an unused SqliteToolSourceStore alias, and hoist the
populator's local os/biotools imports now that the module lives inside
galaxy.tools.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The LAZY_TOOL_STRICT=1 lazy Integration run surfaced three execution-path
attributes that were only materialising via the permissive fallback:
- build_dependency_shell_commands (on __SET_METADATA__, which follows
most jobs) — the runner builds the dependency/env shell commands for
the tool it's about to run
- params_with_missing_data_table_entry / params_with_missing_index_file
(data-manager / index-file tools) — param validation that walks the
parsed parameter tree at job time
All three genuinely need a parsed Tool, so add them to _MATERIALIZE_OK.
Under permissive mode this was a silent WARN + materialise; under strict
it raised NotImplementedError and failed every job-running integration
test. This is exactly the audit strict is meant to force — the fix makes
the materialisation explicit rather than accidental.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Add a materialise counter to LazyToolBox (_lazy_materialize_count, bumped
at the single _create_tool_from_stored_source chokepoint) and an
integration test that hits every batch reader the client uses — flat
listing, panel walk, panel views, tests-summary, all_requirements, and
search — asserting a zero materialise delta across them.
This catches accidental whole-toolbox materialisation that LAZY_TOOL_STRICT
can't: a tool-filter or _MATERIALIZE_OK attribute read inside a full-index
sweep parses tools without ever hitting the strict __getattr__ path. The
test was written against the search-materialisation regression fixed in
the previous commit and fails without it.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
ToolsService.search_tools resolved every whoosh hit through
_get_tool -> toolbox.get_tool, which in lazy mode loads the tool from the
store on demand. A single /api/tools?q= query therefore parsed one Tool
per hit. The populator-owned whoosh index also carries ids that were
never loaded into this toolbox (tool_conf.xml.sample alone lists ~150
legacy tools whose files aren't present), so lazy search both parsed and
returned tools the eager path skips: eager whoosh only holds loaded
tools, and eager _get_tool returns None for un-loaded ids.
Add LazyToolBox.resolve_search_hit, which looks the hit up in the
registered stubs (_tools_by_id, plus the shed short-id map) and returns
None for anything not part of this toolbox — no parse. search_tools uses
it in lazy mode and keeps the eager _get_tool path unchanged. Result:
search stays O(hits) index reads and matches eager scoping.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
AgentTools.get_tool_categories iterates toolbox.tools() and reads
tool.get_panel_section()[1] on every visible tool. get_panel_section was
not on the LazyTool stub surface, so under the permissive default that
sweep materialised the entire toolbox to build the category list (and
under LAZY_TOOL_STRICT=1 it would now raise).
The populator already stamps panel_section_id / panel_section_name onto
ToolIndexEntry, so forward the call off the entry — matching the
(section_id, section_name) / (None, None) contract of
Tool.get_panel_section — and keep the walk lazy.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
test_repository_uninstall kept resurrecting the tool after all
index-level fixes. Local repro + probes showed removal completed on the
right toolbox but two in-memory paths still served the tool:
- The materialise LRU is keyed by whatever id the caller resolved with,
so the same tool sits under both its guid and its short id; the
guid-prefix purge left the short-id entry behind. Purge by cached
tool identity instead.
- _tools_by_old_id is keyed by old_id — the SHORT id for a shed guid —
and super().remove_tool_by_id deletes from it by object identity
only, which misses when the bucket holds an earlier registration
(stub or materialised instance) than _tools_by_id. The leftover
resurfaced through the eager get_tool fall-through. Scrub every
object belonging to the removed guid; sibling installs sharing the
short id survive.
Verified locally: the full TestRepositoryInstallIntegrationTestCase
passes 3/3 consecutive lazy runs (install, uninstall, update).
remove_tool_by_id runs under app._toolbox_lock, but the invalidation
path (queue-worker reload_tool_source_cache handler — and every
populate broadcasts one) reloaded the index and re-registered stubs
without it. Interleaved with an uninstall, that reloads the pre-removal
index and re-registers the just-removed tool. Take the same RLock.
When the target <section> already exists in the on-disk shed_tool_conf,
add_to_shed_tool_config merges the incoming children into it via lxml
append — which reparents them, leaving the section elements in
elem_list empty. The lazy branch persists the conf BEFORE loading (the
populate needs it on disk), so both the partial populate (no paths →
no enriched entries) and load_item (empty section → tool never enters
the in-memory panel; only the async conf-watcher reload rescued it) ran
against gutted elements. First install into a fresh section was immune,
which is why only the second fastp changeset lost tool_shed_repository
(test_only_latest_version_in_panel_fastp — now reproduced and verified
locally in both lazy and eager modes).
Deep-copy the elements before persisting and drive the populate +
load_item from the snapshot. The eager branch loads before persisting
and stays untouched.
The partial populate filters discovered tools on exact path strings.
_collect_new_tool_paths joined the shed conf dict's raw tool_path —
relative for typical shed_tool_conf files — while discovery resolves a
relative tool_path against the conf file's directory and returns an
absolute path. The strings never matched, so the enriched conf-driven
discovery was filtered out and the ad-hoc synthesis populated a
metadata-poor entry instead: the panel served fastp without
tool_shed_repository (test_only_latest_version_in_panel_fastp), and the
store grew a duplicate row under the unresolved path string.
Route both through discover.resolve_tool_path (now public) and thread
each tool's guid so even a genuinely-unreachable path yields a
guid-keyed entry.
The data_manager_id stamp landed in index.entries only —
update_index_entry never touched entries_by_version, and the two maps
serialize independently. Job-time materialise passes a tool_version, so
it resolves through the per-version map and got the unstamped entry
back after the next index reload (populate broadcasts one constantly):
exec_after_process still failed with 'Invalid data manager (<guid>)'.
Both backends now route updates through ToolIndex.add_entry, which
keeps the maps in step (and stops an older-version update from
clobbering the newer default entry). create_tool additionally stamps
the in-memory per-version twin, which is a distinct object after a
from_dict reload.
_register_new_index_entries_as_stubs skipped every tool already in
_tools_by_id, so a stub minted from an install-time ad-hoc entry kept
serving that metadata-poor entry after the conf-driven populate
enriched the index — the fastp panel payload was missing
tool_shed_repository (test_only_latest_version_in_panel_fastp).
Swap the entry behind the existing stub instead: the panel and
registries hold the stub object, and _overrides survive.
Shed-installed data managers load their tool at install time, before
shed_data_manager_conf.xml exists — the self-healed entry carries
data_manager_id=None, materialise never restores the conf id, and
DataManagerTool.exec_after_process fails with 'Invalid data manager
(<guid>)' at job finish (all six data-manager integration tests).
DataManager._load_tool already passes data_manager_id through
load_hidden_tool; create_tool now stamps it onto the resolved entry and
persists it, so job handlers materialising from the shared index
resolve the registry correctly.
remove_tool_by_id popped the entry from every in-memory map, but the
persisted singleton index still carried it. Every populate broadcasts a
reload_tool_source_cache invalidation, so the very next reload handed
the entry back and the uninstalled tool resolved again
(test_repository_uninstall expected err_msg, got the full tool).
ToolSourceStore grows remove_index_entry — the uninstall counterpart of
update_index_entry — and remove_tool_by_id writes the removal through.
DataManagerTool resolves the registry by the <data_manager id> conf id,
which may differ from the tool XML id
(test_data_manager_async_submission_with_mismatched_conf_id). Eager
threads it through load_hidden_tool(data_manager_id=...); lazy
materialisation built the tool from the stored source alone, so
DataManagerTool.__init__ fell back to the tool id and
exec_after_process failed with 'Invalid data manager requested'.
Discovery now records the conf id, the index entry carries it, and
_create_tool_from_stored_source restores it at materialise time. Also
stamp is_local from the discovered guid — it defaulted True, which kept
the ToolConfRepository branch of the shed materialise path dead even
with repository metadata present.
Discovery stamped hidden=True on data manager tools, but eager's
load_hidden_tool only means 'not in the panel' — Tool.hidden stays
falsy, and the flat /api/tools?in_panel=false listing filters hidden
tools, so the lazy listing dropped them
(test_data_manager_async_submission_with_mismatched_conf_id).
Also gate LazyTool.allow_user_access on tool_type 'manage_data' — the
value DataManagerTool actually carries; 'data_manager' never occurs, so
the admin gate on stubs never fired and lazy was more permissive than
eager.
The lazy path capped whoosh results at tool_search_limit (default 20)
while eager ToolPanelViewSearch passes limit=None. Uniform-score matches
truncate in doc-insertion order, so a tag query fanning out to 23 tools
silently lost the last three (test_search_curated_tool_tags).
Shed installs never survived lazy mode: metadata generation
(installed_repository_metadata_manager.get_repository_tools_tups) loads
each freshly cloned tool before add_to_tool_panel persists the conf and
populates the index, so create_tool raised on every install and the
repository landed broken (data managers, workflow test tools, fastp
panel tests).
A miss for a file that exists on disk now populates that single path —
still through the single-writer populator; populate_store_inline
synthesizes a DiscoveredTool for requested paths outside every conf and
threads the caller's guid so the ad-hoc entry is keyed like its eventual
conf-driven replacement — then reloads the index and retries. Misses
for files that don't exist still raise.
Six test/functional tools (upload.xml, export_remote.xml,
ucsc_tablebrowser.xml, parse_values_from_file.xml, catWrapper.xml,
for_tours/filtering.xml) expand to byte-identical content as their
root tools/ counterparts. With hash unique, whichever file populates
first owns the row; the twin's populate is skipped and
get_by_source_path(second path) returns nothing, so LazyToolBox.create_tool
raises and the toolbox silently drops the tool — upload breaks for any
instance whose config discovers both trees (CI integration shards do).
hash becomes a non-unique index (migration amended in place — unreleased),
store() upserts by source_path, the populator skips only when the same
path already holds the same content, and get/exists/delete tolerate
multiple rows per hash.
ToolIndexEntry gains icon, xrefs, model_class, form_style and
is_workflow_compatible, derived at populate time the same way
Tool.__init__/to_dict derive them (tool_type class registry, root
workflow_compatible attribute, page count). EDAM fields now go through
expand_ontology_data, picking up the curated mapping overrides and
legacy bio.tools xrefs the direct parse_edam_* calls missed.
LazyTool.to_panel_entry emits the full client payload -- including
tool_shed_repository (the field the panel-view integration tests
assert), versions from the registered lineage, uuid, and the
admin-gated config_file from source_path.
The populator writes through its own store instances, so after the
inline backfill the toolbox's store still served its cached pre-populate
index. Single-instance that cache is merely stale-but-identical; on a
shared database (the CI integration shards share one postgres DB, and
any multi-server deployment shares one in production) the cached index
is another instance's view and the eager walk drops every tool missing
from it — the post-reload 'no index entry for upload.xml' cascade.
Reload integration tests drive galaxy.queue_worker.reload_toolbox
directly and reproduce the failure with a foreign index row; the earlier
send_control_task-based attempts passed vacuously because the control
message never executes in this harness.
- The lazy whoosh schema lacked the tool_tags field the eager schema
indexes, so a fielded phrase query (tool_tags:"send data") fanned out
to positionless NGRAMWORDS fields and raised QueryError -> 500. Index
tool_tags per entry (curated mapping over the entry ids) and search it.
- Data manager tool paths resolve with the same relative-to-conf
fallback DataManagers.load_from_xml applies for planemo layouts.
- LazyTool consults _MATERIALIZE_OK before the underscore guard and adds
_view, so admin dependency-management endpoints materialise instead of
AttributeError.
The packages/app tree links each galaxy subpackage explicitly and
tool_source_store was never added, so the built wheel lacked the package
and per-package mypy resolved its imports as Any (surfacing as
'Returning Any' in LazyToolboxSearch.search).
curated_tool_tags sweeps every tool for .tool_tags, which force-
materialised the whole toolbox on first /api/tags/tool_tags request and
500d whenever one indexed tool can't build (filter_data_table's
column="value" dynamic options). The tag mapping is keyed purely by
tool id, so the stub now answers without materialising.
Data manager tools are loaded post-boot via DataManagers ->
load_hidden_tool, which the discover walk missed — create_tool's index
miss aborted DataManager loading and the tools 404d under lazy mode.
discover_tools now walks data_manager_conf/shed_data_manager_conf
(path + guid resolution mirroring DataManager._load_from_element), and
the seam's source-path lookups normalize to absolute paths since data
manager tool files are joined from a possibly relative tool_path.
GalaxyAppConfiguration materializes every schema option, so the
hasattr/getattr guards around tool_configs, shed_tool_config_file,
migrated_tools_config, root, tool_search_index_dir and use_lazy_toolbox
were dead (StandaloneInstallationTarget's Config stub gains the
use_lazy_toolbox attribute the real config carries). The
tool_source_url getattr had no matching schema option — the default
sqlalchemy store is now disk-path only; full URLs remain available via
named tool_source_stores entries.
getattr(entry, "source_path") was hiding a real gap: ToolIndexEntry
never carried the field, so LazyTool.config_file always materialised.
The populator now stamps source_path onto every entry.
discover.py's duplicated _looks_like_a_tool is replaced by the real
galaxy.tool_util.loader_directory.looks_like_a_tool (threading
enable_beta_tool_formats), and the works-without-galaxy fallback for
MODEL_TOOLS_PATH is gone — the populator always runs with galaxy
importable.
The job-request path creates and reads tool_source rows whose source
column carries the raw source string; the store was writing dict-shaped
payloads into the same table, and once identity hashes aligned the job
path could resolve a store row and fail pydantic validation (500 on
every request-style tool execution under lazy mode).
Give the store its own content-addressed table with real columns —
get_by_tool_id / get_by_source_path become indexed queries instead of
full-table JSON scans, populator pruning can never delete rows the job
path references, and the identity_hash coupling (and the legacy
__tool_index__ sentinel-row fallback) disappear.
- AbstractToolBox gains a no-op invalidate_index_cache (LazyToolBox
overrides it); shed installs no longer depend on the subclass type.
- Annotate LazyToolboxSearch.build_index, ToolFileWatcher.observer,
populator session cast, UUID key coercion in _register_lazy_entry,
and drop stale type: ignore comments.
The lazy path persists shed_tool_conf before load_item because the
populator's conf walk needs it on disk; that same reorder broke eager
version-replacement in the panel (test_add_twice). Split the paths:
eager is now byte-identical to upstream, the reorder applies only under
use_lazy_toolbox.
The lazy ``to_dict`` fast path was short-circuiting *both* the
panel-view walk (intended) *and* the ``/api/tools/{id}`` show endpoint
(unintended) — they're indistinguishable at the ``to_dict(io_details=
False, link_details=False)`` callsite. ``test_legacy_biotools_xref_
injection`` caught it: the show endpoint returned the entry-shape
dict and no longer carried ``xrefs``.
Split the two call sites at the API rather than the parameter:
* ``Tool.to_panel_entry(trans)`` — new method, returns the cheap
entry-shape dict (id/name/version/description/labels/edam/panel
section/link). ``LazyTool`` overrides it to skip materialise.
* ``AbstractToolBox.get_tool_to_dict`` calls ``to_panel_entry`` when
``tool_help=False`` (the panel-view default); ``tool_help=True``
still falls through to ``to_dict`` so the help payload renders.
* ``LazyTool.to_dict`` always materialises now — it's the
``/api/tools/{id}`` contract and must ship ``xrefs`` / ``versions``
/ ``is_workflow_compatible`` / ``tool_shed_repository`` / etc. The
materialise-failure fallback now reuses ``to_panel_entry``.
Net effect: ``/api/tool_panels/{view}`` and ``/api/tools?in_panel=
true`` stay materialise-free for the lazy toolbox; ``/api/tools/{id}``
serves the full show-endpoint dict in both modes.
`LazyTool` materialise during workflow ``inject_all`` calls
``store.get(hash)``; the store ran the SELECT on Galaxy's shared
scoped session and then called ``session.rollback()`` to release the
implicit read transaction. That rollback expired *every*
request-scoped instance attached to the shared session — including the
in-flight ``workflow.steps`` collection. The next access in the same
``populate_module_and_state`` re-fetched a fresh ``WorkflowStep`` list
from the DB, with ``.module`` unset on the new instances, so
``compute_runtime_state``'s ``assert step.module`` raised
``AttributeError: 'WorkflowStep' object has no attribute 'module'``.
Every workflow-invocation test under ``use_lazy_toolbox=true`` 500'd.
Fix: read methods on ``DatabaseToolSourceStore`` (``get``, ``exists``,
``get_by_tool_id``, ``get_by_source_path``, ``count``, ``list_all``,
``load_index``) now open a private ``Session`` bound to the same
engine and close it on exit. The shared scoped session — and its
caller's transaction state — is untouched. The original lock-release
intent (cold-start populator, queue-worker init) still holds because
each private session's read transaction ends when the session closes.
Writes (``store``, ``store_index``, ``delete``, ``update_index_entry``)
still use ``_get_session()`` so they participate in the caller's
transaction and commit in the right context (populator, queue worker,
``LazyToolBox.__init__``).
With ``link_details`` no longer gating any field, the toolbox-internal
panel / listing callers stopped passing it (``f64858bf55e``).
``LazyTool.to_dict`` can now return the entry-shape dict for the
default-call case — every panel-view tool serves from the index, zero
materialise.
Materialise still fires when the caller asks for the parsed-Tool
payload: ``io_details=True`` (the ``/api/tools/{id}`` show endpoint
contract) or ``tool_help=True`` (rendered help). Failure-to-materialise
still falls back to the entry-shape dict so the show endpoint never
returns 500 when a specific tool's XML chokes the parameter factory.
The ``link`` field is derived from ``self.id`` matching what
``Tool.to_dict`` now emits unconditionally — ``/tool_runner?tool_id={id}``
for the standard tool runner.
Tests updated:
- ``test_to_dict_entry_fast_path_does_not_materialise`` asserts the
default call serves the entry dict and never invokes the callback.
- ``test_to_dict_io_details_materialises`` covers the show contract.
- ``test_to_dict_falls_back_to_entry_when_materialise_fails`` covers the
parameter-factory-failure case.
23/23 unit tests pass.
These were emitted on every entry of /api/tools but no client component
ever read them. The Vue panel renderer derives the link target from
``model_class == "DataSourceTool"`` via ``getTargetById`` and never reads
``min_width`` at all; the Tool interface declared the fields purely to
match the wire shape. ``Tool.target`` / ``Tool.uihints`` likewise had
``to_dict`` as their sole reader in lib/.
Removed:
- ``self.target`` / ``self.uihints`` attributes and the corresponding
``<inputs target>`` and ``<uihints minwidth>`` reads in ``__parse_legacy_features``
- ``DataSourceTool.parse_inputs`` self.target = "_top" override (dead)
- ``min_width`` / ``target`` keys from both the eager ``to_dict`` payload
and the ``LazyTool.to_dict`` stub
The XSD declarations for ``<uihints minwidth>`` and ``<inputs target>``
stay so existing data-source tool XML continues to validate cleanly.
After five rounds of CI surfacing missing stub attrs and the latest
round still hitting indirect failures (workflow steps with ``module``
unset because some inner attribute access raised under strict), the
explicit ``_MATERIALIZE_OK`` set has stopped converging. The strict
guard caught the obvious surface and now slows ship velocity without
adding signal.
Flip the default: unknown attr reads on ``LazyTool`` now log a WARNING
and materialise. The strict raise is still available for debugging via
``LAZY_TOOL_STRICT=1`` — the path that surfaces ``add to the stub
surface or _MATERIALIZE_OK`` for new audit work.
``test_strict_getattr_raises_with_clear_message`` now monkey-patches
the module flag to force strict mode for the assertion. Other tests are
unchanged.
The entry-only fast path was missing fields the ``/api/tools/<id>``
show endpoint serves (``xrefs``, ``inputs`` when ``io_details=False``,
…). The flat ``/api/tools?in_panel=False`` listing — the only caller
that benefits from the fast path — already bypasses this method:
``services/tools.py:list_tools`` walks ``tool_index.list_all()`` and
calls ``entry.to_api_dict()`` directly.
Always materialise here; the entry-only dict survives as the fallback
when materialise itself raises (a tool whose XML the parameter
factory chokes on — ``upload_dataset``, ``column="value"``, …) so the
caller still gets *something* renderable rather than a 500.
Tests updated:
- ``test_to_dict_fast_path_does_not_materialise`` → renamed to
``test_to_dict_falls_back_to_entry_dict_when_materialise_fails``.
- ``test_to_dict_link_details_materialises_exactly_once`` → renamed
for accuracy; behaviour unchanged.
``/api/tools/<id>`` show endpoint passes ``io_details=True`` to surface
``inputs`` / ``outputs`` / parameter detail; the previous condition
only materialised on ``link_details=True``, so the listing-shape dict
came back without inputs and ``test_show_repeat`` / ``test_show_multi_data``
asserted on a missing key.
Add ``io_details`` to the materialise predicate. Forward both flags into
``Tool.to_dict`` so the parent picks the right payload variant. The
entry-only fast path now only applies to the truly cheap case (the
``/api/tools?in_panel=False`` listing).
Materialise failure still falls back to the entry-only dict — listing
remains 200 even if a specific tool can't materialise.
Round-2 CI surfaced five more attributes the tool-execution path
hits before the parameter machinery has finished setting up:
- ``check_and_update_param_values`` — parameter validation called from
``handle_input``.
- ``wants_params_cleaned`` — parameter scrub before execution.
- ``tool_source`` — raw ``ToolSource`` (some callers walk the XML
directly).
- ``dynamic_tool`` — dynamic-tool linkage on execution.
- ``produces_entry_points`` — interactive-tool entry-point lookup.
All five materialise correctly when read; adding to ``_MATERIALIZE_OK``
makes the stub's strict ``__getattr__`` route through the cached
``_real`` Tool on first access instead of raising.
CI surfaced six attributes that the tool-execution path
(``/api/tools/{id}`` POST → ``handle_input``) and the job runner read
on the resulting Tool, all of which fundamentally need a parsed
parameter tree:
- ``handle_input`` — entry point for tool execution.
- ``inputs``, ``parameters``, ``new_state`` — the parameter machinery.
- ``input_translator`` — runtime parameter translation.
- ``requires_galaxy_python_environment`` — job-runner environment hint.
Each was raising ``NotImplementedError`` for every tool-execution
request, taking out ~hundreds of API tests at once. Add them to
``_MATERIALIZE_OK`` so the stub materialises on first read and the
real Tool answers the rest of the call.
Materialise on tool execution is exactly the cost model the lazy
toolbox is designed for — pay parse cost only when the tool is
actually used. The stub still surfaces unaccounted-for attribute reads
loudly, which is the contract the user picked.
``Registry.load_datatype_converters`` (``lib/galaxy/datatypes/registry.py:674``)
walks the converter list from ``datatypes_conf.xml`` after boot and
calls ``toolbox.load_tool(config_path)`` per entry. Under strict
``LazyToolBox.create_tool`` every converter raised ``RuntimeError``
(caught + logged by the registry as ``Error loading converter (…)``)
and the ``datatype_converters`` dict stayed silently empty — so
format-mismatch conversions via
``Dataset.find_conversion_destination`` were no-ops.
Fix: in ``discover_tools()``, after the hidden-lib block, query
``galaxy.model._get_datatypes_registry()`` and yield a
``DiscoveredTool`` for each converter the registry has parsed. Same
source of truth ``load_datatype_converters`` iterates, so we can't
drift — what the registry will try to load is exactly what the
populator indexes.
The registry is populated by ``set_datatypes_registry()`` at app boot
(``app/__init__.py:853``) before the toolbox initialises and inside
``populate_store(config_file=…)`` before ``populate_store_inline`` runs
— both paths produce a live registry by the time the populator walks.
On the (atypical) case where the registry isn't set, the try/except
falls through and the converters remain unindexed; the eager
``load_datatype_converters`` then continues to catch + log the
``Error loading converter`` messages, matching today's pre-refactor
behaviour.
After this commit, ``Persisted ToolIndex for store __default__``
reports 580 entries (up from 564 = +16 converters that
``sample_tool_conf.xml`` doesn't list) and zero ``Error loading
converter`` messages appear in the integration log. 16/16 integration
tests + 113/113 unit tests still pass.
``_create_tool_from_stored_source`` was hitting the install database on
every shed-tool materialise to fetch ``ToolShedRepository`` just so the
Tool ctor's ``populate_tool_shed_info`` could stamp four scalar fields.
The eager pipeline already has a namedtuple stub for exactly this case
(``ToolConfRepository`` in ``lib/galaxy/tool_util/toolbox/base.py:87``,
used for shed installs whose install-DB row hasn't appeared yet) —
build it directly from ``ToolIndexEntry``.
Eliminates:
- ``_lookup_tool_shed_repository`` method (~30 lines).
- ``galaxy.tool_shed.util.repository_util.get_installed_repository``
module-level import.
- A DB round-trip per shed-tool materialise.
``installed_tool_dependencies`` readers see
``tool_dependencies_installed_or_in_error=[]`` for materialised lazy
tools — same shape the eager pipeline produces when the install-DB
row is missing, so downstream callers already tolerate this. If a job
runner ever needs real DB-backed dependency info, it can query
``app.install_model`` directly using the scalar shed metadata already
on ``Tool``.
16/16 integration tests still pass.
Every inline import in these two files was checked against:
1. Does it create an actual import cycle? (No for any of them.)
2. Is it a soft / optional dependency? (kombu + watchdog are in
``pyproject.toml`` and ``pinned-requirements.txt``; the
try/except-ImportError defenses were dead code.)
The "lazy import for startup speed" justifications on the CLI entry
points (``galaxy.config``, ``galaxy.model``, ``galaxy.datatypes.registry``,
``galaxy.tool_source_store.search``, etc.) were also bogus — the
populator module is imported at Galaxy boot via the LazyToolBox
cold-start hook, so those heavy modules get pulled in anyway. The
inline form only delayed the cost by milliseconds while making the
dep graph invisible at the top of the file.
Hoisted:
- ``lazy_toolbox.py``: ``galaxy.exceptions``, ``galaxy.tool_shed.util.repository_util``,
``galaxy.util.tool_version``.
- ``populator.py``: ``kombu``, ``watchdog``, ``galaxy.config``,
``galaxy.datatypes.registry``, ``galaxy.model``, ``galaxy.model.mapping``,
``galaxy.queues``, ``galaxy.tool_source_store.search`` (Tuning/Whoosh),
``galaxy.util.properties``. The watchdog ``ImportError`` defense and
the kombu inline block both drop with them.
Galaxy boot still works; 113/113 unit tests + 16/16 integration tests
pass; CLI ``populate_store.py --help`` still resolves. The
``TYPE_CHECKING`` block in ``lazy_toolbox.py`` (the only remaining
function-scope ``from``-import area) keeps its lookup-only imports
where they belong.
mvdbeek