remove_tool_by_id popped the entry from every in-memory map, but the
persisted singleton index still carried it. Every populate broadcasts a
reload_tool_source_cache invalidation, so the very next reload handed
the entry back and the uninstalled tool resolved again
(test_repository_uninstall expected err_msg, got the full tool).
ToolSourceStore grows remove_index_entry — the uninstall counterpart of
update_index_entry — and remove_tool_by_id writes the removal through.
DataManagerTool resolves the registry by the <data_manager id> conf id,
which may differ from the tool XML id
(test_data_manager_async_submission_with_mismatched_conf_id). Eager
threads it through load_hidden_tool(data_manager_id=...); lazy
materialisation built the tool from the stored source alone, so
DataManagerTool.__init__ fell back to the tool id and
exec_after_process failed with 'Invalid data manager requested'.
Discovery now records the conf id, the index entry carries it, and
_create_tool_from_stored_source restores it at materialise time. Also
stamp is_local from the discovered guid — it defaulted True, which kept
the ToolConfRepository branch of the shed materialise path dead even
with repository metadata present.
Discovery stamped hidden=True on data manager tools, but eager's
load_hidden_tool only means 'not in the panel' — Tool.hidden stays
falsy, and the flat /api/tools?in_panel=false listing filters hidden
tools, so the lazy listing dropped them
(test_data_manager_async_submission_with_mismatched_conf_id).
Also gate LazyTool.allow_user_access on tool_type 'manage_data' — the
value DataManagerTool actually carries; 'data_manager' never occurs, so
the admin gate on stubs never fired and lazy was more permissive than
eager.
The lazy path capped whoosh results at tool_search_limit (default 20)
while eager ToolPanelViewSearch passes limit=None. Uniform-score matches
truncate in doc-insertion order, so a tag query fanning out to 23 tools
silently lost the last three (test_search_curated_tool_tags).
Shed installs never survived lazy mode: metadata generation
(installed_repository_metadata_manager.get_repository_tools_tups) loads
each freshly cloned tool before add_to_tool_panel persists the conf and
populates the index, so create_tool raised on every install and the
repository landed broken (data managers, workflow test tools, fastp
panel tests).
A miss for a file that exists on disk now populates that single path —
still through the single-writer populator; populate_store_inline
synthesizes a DiscoveredTool for requested paths outside every conf and
threads the caller's guid so the ad-hoc entry is keyed like its eventual
conf-driven replacement — then reloads the index and retries. Misses
for files that don't exist still raise.
Six test/functional tools (upload.xml, export_remote.xml,
ucsc_tablebrowser.xml, parse_values_from_file.xml, catWrapper.xml,
for_tours/filtering.xml) expand to byte-identical content as their
root tools/ counterparts. With hash unique, whichever file populates
first owns the row; the twin's populate is skipped and
get_by_source_path(second path) returns nothing, so LazyToolBox.create_tool
raises and the toolbox silently drops the tool — upload breaks for any
instance whose config discovers both trees (CI integration shards do).
hash becomes a non-unique index (migration amended in place — unreleased),
store() upserts by source_path, the populator skips only when the same
path already holds the same content, and get/exists/delete tolerate
multiple rows per hash.
ToolIndexEntry gains icon, xrefs, model_class, form_style and
is_workflow_compatible, derived at populate time the same way
Tool.__init__/to_dict derive them (tool_type class registry, root
workflow_compatible attribute, page count). EDAM fields now go through
expand_ontology_data, picking up the curated mapping overrides and
legacy bio.tools xrefs the direct parse_edam_* calls missed.
LazyTool.to_panel_entry emits the full client payload -- including
tool_shed_repository (the field the panel-view integration tests
assert), versions from the registered lineage, uuid, and the
admin-gated config_file from source_path.
The populator writes through its own store instances, so after the
inline backfill the toolbox's store still served its cached pre-populate
index. Single-instance that cache is merely stale-but-identical; on a
shared database (the CI integration shards share one postgres DB, and
any multi-server deployment shares one in production) the cached index
is another instance's view and the eager walk drops every tool missing
from it — the post-reload 'no index entry for upload.xml' cascade.
Reload integration tests drive galaxy.queue_worker.reload_toolbox
directly and reproduce the failure with a foreign index row; the earlier
send_control_task-based attempts passed vacuously because the control
message never executes in this harness.
- The lazy whoosh schema lacked the tool_tags field the eager schema
indexes, so a fielded phrase query (tool_tags:"send data") fanned out
to positionless NGRAMWORDS fields and raised QueryError -> 500. Index
tool_tags per entry (curated mapping over the entry ids) and search it.
- Data manager tool paths resolve with the same relative-to-conf
fallback DataManagers.load_from_xml applies for planemo layouts.
- LazyTool consults _MATERIALIZE_OK before the underscore guard and adds
_view, so admin dependency-management endpoints materialise instead of
AttributeError.
The packages/app tree links each galaxy subpackage explicitly and
tool_source_store was never added, so the built wheel lacked the package
and per-package mypy resolved its imports as Any (surfacing as
'Returning Any' in LazyToolboxSearch.search).
curated_tool_tags sweeps every tool for .tool_tags, which force-
materialised the whole toolbox on first /api/tags/tool_tags request and
500d whenever one indexed tool can't build (filter_data_table's
column="value" dynamic options). The tag mapping is keyed purely by
tool id, so the stub now answers without materialising.
Data manager tools are loaded post-boot via DataManagers ->
load_hidden_tool, which the discover walk missed — create_tool's index
miss aborted DataManager loading and the tools 404d under lazy mode.
discover_tools now walks data_manager_conf/shed_data_manager_conf
(path + guid resolution mirroring DataManager._load_from_element), and
the seam's source-path lookups normalize to absolute paths since data
manager tool files are joined from a possibly relative tool_path.
GalaxyAppConfiguration materializes every schema option, so the
hasattr/getattr guards around tool_configs, shed_tool_config_file,
migrated_tools_config, root, tool_search_index_dir and use_lazy_toolbox
were dead (StandaloneInstallationTarget's Config stub gains the
use_lazy_toolbox attribute the real config carries). The
tool_source_url getattr had no matching schema option — the default
sqlalchemy store is now disk-path only; full URLs remain available via
named tool_source_stores entries.
getattr(entry, "source_path") was hiding a real gap: ToolIndexEntry
never carried the field, so LazyTool.config_file always materialised.
The populator now stamps source_path onto every entry.
discover.py's duplicated _looks_like_a_tool is replaced by the real
galaxy.tool_util.loader_directory.looks_like_a_tool (threading
enable_beta_tool_formats), and the works-without-galaxy fallback for
MODEL_TOOLS_PATH is gone — the populator always runs with galaxy
importable.
The job-request path creates and reads tool_source rows whose source
column carries the raw source string; the store was writing dict-shaped
payloads into the same table, and once identity hashes aligned the job
path could resolve a store row and fail pydantic validation (500 on
every request-style tool execution under lazy mode).
Give the store its own content-addressed table with real columns —
get_by_tool_id / get_by_source_path become indexed queries instead of
full-table JSON scans, populator pruning can never delete rows the job
path references, and the identity_hash coupling (and the legacy
__tool_index__ sentinel-row fallback) disappear.
- AbstractToolBox gains a no-op invalidate_index_cache (LazyToolBox
overrides it); shed installs no longer depend on the subclass type.
- Annotate LazyToolboxSearch.build_index, ToolFileWatcher.observer,
populator session cast, UUID key coercion in _register_lazy_entry,
and drop stale type: ignore comments.
The lazy path persists shed_tool_conf before load_item because the
populator's conf walk needs it on disk; that same reorder broke eager
version-replacement in the panel (test_add_twice). Split the paths:
eager is now byte-identical to upstream, the reorder applies only under
use_lazy_toolbox.
The lazy ``to_dict`` fast path was short-circuiting *both* the
panel-view walk (intended) *and* the ``/api/tools/{id}`` show endpoint
(unintended) — they're indistinguishable at the ``to_dict(io_details=
False, link_details=False)`` callsite. ``test_legacy_biotools_xref_
injection`` caught it: the show endpoint returned the entry-shape
dict and no longer carried ``xrefs``.
Split the two call sites at the API rather than the parameter:
* ``Tool.to_panel_entry(trans)`` — new method, returns the cheap
entry-shape dict (id/name/version/description/labels/edam/panel
section/link). ``LazyTool`` overrides it to skip materialise.
* ``AbstractToolBox.get_tool_to_dict`` calls ``to_panel_entry`` when
``tool_help=False`` (the panel-view default); ``tool_help=True``
still falls through to ``to_dict`` so the help payload renders.
* ``LazyTool.to_dict`` always materialises now — it's the
``/api/tools/{id}`` contract and must ship ``xrefs`` / ``versions``
/ ``is_workflow_compatible`` / ``tool_shed_repository`` / etc. The
materialise-failure fallback now reuses ``to_panel_entry``.
Net effect: ``/api/tool_panels/{view}`` and ``/api/tools?in_panel=
true`` stay materialise-free for the lazy toolbox; ``/api/tools/{id}``
serves the full show-endpoint dict in both modes.
`LazyTool` materialise during workflow ``inject_all`` calls
``store.get(hash)``; the store ran the SELECT on Galaxy's shared
scoped session and then called ``session.rollback()`` to release the
implicit read transaction. That rollback expired *every*
request-scoped instance attached to the shared session — including the
in-flight ``workflow.steps`` collection. The next access in the same
``populate_module_and_state`` re-fetched a fresh ``WorkflowStep`` list
from the DB, with ``.module`` unset on the new instances, so
``compute_runtime_state``'s ``assert step.module`` raised
``AttributeError: 'WorkflowStep' object has no attribute 'module'``.
Every workflow-invocation test under ``use_lazy_toolbox=true`` 500'd.
Fix: read methods on ``DatabaseToolSourceStore`` (``get``, ``exists``,
``get_by_tool_id``, ``get_by_source_path``, ``count``, ``list_all``,
``load_index``) now open a private ``Session`` bound to the same
engine and close it on exit. The shared scoped session — and its
caller's transaction state — is untouched. The original lock-release
intent (cold-start populator, queue-worker init) still holds because
each private session's read transaction ends when the session closes.
Writes (``store``, ``store_index``, ``delete``, ``update_index_entry``)
still use ``_get_session()`` so they participate in the caller's
transaction and commit in the right context (populator, queue worker,
``LazyToolBox.__init__``).
With ``link_details`` no longer gating any field, the toolbox-internal
panel / listing callers stopped passing it (``f64858bf55e``).
``LazyTool.to_dict`` can now return the entry-shape dict for the
default-call case — every panel-view tool serves from the index, zero
materialise.
Materialise still fires when the caller asks for the parsed-Tool
payload: ``io_details=True`` (the ``/api/tools/{id}`` show endpoint
contract) or ``tool_help=True`` (rendered help). Failure-to-materialise
still falls back to the entry-shape dict so the show endpoint never
returns 500 when a specific tool's XML chokes the parameter factory.
The ``link`` field is derived from ``self.id`` matching what
``Tool.to_dict`` now emits unconditionally — ``/tool_runner?tool_id={id}``
for the standard tool runner.
Tests updated:
- ``test_to_dict_entry_fast_path_does_not_materialise`` asserts the
default call serves the entry dict and never invokes the callback.
- ``test_to_dict_io_details_materialises`` covers the show contract.
- ``test_to_dict_falls_back_to_entry_when_materialise_fails`` covers the
parameter-factory-failure case.
23/23 unit tests pass.
These were emitted on every entry of /api/tools but no client component
ever read them. The Vue panel renderer derives the link target from
``model_class == "DataSourceTool"`` via ``getTargetById`` and never reads
``min_width`` at all; the Tool interface declared the fields purely to
match the wire shape. ``Tool.target`` / ``Tool.uihints`` likewise had
``to_dict`` as their sole reader in lib/.
Removed:
- ``self.target`` / ``self.uihints`` attributes and the corresponding
``<inputs target>`` and ``<uihints minwidth>`` reads in ``__parse_legacy_features``
- ``DataSourceTool.parse_inputs`` self.target = "_top" override (dead)
- ``min_width`` / ``target`` keys from both the eager ``to_dict`` payload
and the ``LazyTool.to_dict`` stub
The XSD declarations for ``<uihints minwidth>`` and ``<inputs target>``
stay so existing data-source tool XML continues to validate cleanly.
After five rounds of CI surfacing missing stub attrs and the latest
round still hitting indirect failures (workflow steps with ``module``
unset because some inner attribute access raised under strict), the
explicit ``_MATERIALIZE_OK`` set has stopped converging. The strict
guard caught the obvious surface and now slows ship velocity without
adding signal.
Flip the default: unknown attr reads on ``LazyTool`` now log a WARNING
and materialise. The strict raise is still available for debugging via
``LAZY_TOOL_STRICT=1`` — the path that surfaces ``add to the stub
surface or _MATERIALIZE_OK`` for new audit work.
``test_strict_getattr_raises_with_clear_message`` now monkey-patches
the module flag to force strict mode for the assertion. Other tests are
unchanged.
The entry-only fast path was missing fields the ``/api/tools/<id>``
show endpoint serves (``xrefs``, ``inputs`` when ``io_details=False``,
…). The flat ``/api/tools?in_panel=False`` listing — the only caller
that benefits from the fast path — already bypasses this method:
``services/tools.py:list_tools`` walks ``tool_index.list_all()`` and
calls ``entry.to_api_dict()`` directly.
Always materialise here; the entry-only dict survives as the fallback
when materialise itself raises (a tool whose XML the parameter
factory chokes on — ``upload_dataset``, ``column="value"``, …) so the
caller still gets *something* renderable rather than a 500.
Tests updated:
- ``test_to_dict_fast_path_does_not_materialise`` → renamed to
``test_to_dict_falls_back_to_entry_dict_when_materialise_fails``.
- ``test_to_dict_link_details_materialises_exactly_once`` → renamed
for accuracy; behaviour unchanged.
``/api/tools/<id>`` show endpoint passes ``io_details=True`` to surface
``inputs`` / ``outputs`` / parameter detail; the previous condition
only materialised on ``link_details=True``, so the listing-shape dict
came back without inputs and ``test_show_repeat`` / ``test_show_multi_data``
asserted on a missing key.
Add ``io_details`` to the materialise predicate. Forward both flags into
``Tool.to_dict`` so the parent picks the right payload variant. The
entry-only fast path now only applies to the truly cheap case (the
``/api/tools?in_panel=False`` listing).
Materialise failure still falls back to the entry-only dict — listing
remains 200 even if a specific tool can't materialise.
Round-2 CI surfaced five more attributes the tool-execution path
hits before the parameter machinery has finished setting up:
- ``check_and_update_param_values`` — parameter validation called from
``handle_input``.
- ``wants_params_cleaned`` — parameter scrub before execution.
- ``tool_source`` — raw ``ToolSource`` (some callers walk the XML
directly).
- ``dynamic_tool`` — dynamic-tool linkage on execution.
- ``produces_entry_points`` — interactive-tool entry-point lookup.
All five materialise correctly when read; adding to ``_MATERIALIZE_OK``
makes the stub's strict ``__getattr__`` route through the cached
``_real`` Tool on first access instead of raising.
CI surfaced six attributes that the tool-execution path
(``/api/tools/{id}`` POST → ``handle_input``) and the job runner read
on the resulting Tool, all of which fundamentally need a parsed
parameter tree:
- ``handle_input`` — entry point for tool execution.
- ``inputs``, ``parameters``, ``new_state`` — the parameter machinery.
- ``input_translator`` — runtime parameter translation.
- ``requires_galaxy_python_environment`` — job-runner environment hint.
Each was raising ``NotImplementedError`` for every tool-execution
request, taking out ~hundreds of API tests at once. Add them to
``_MATERIALIZE_OK`` so the stub materialises on first read and the
real Tool answers the rest of the call.
Materialise on tool execution is exactly the cost model the lazy
toolbox is designed for — pay parse cost only when the tool is
actually used. The stub still surfaces unaccounted-for attribute reads
loudly, which is the contract the user picked.
``Registry.load_datatype_converters`` (``lib/galaxy/datatypes/registry.py:674``)
walks the converter list from ``datatypes_conf.xml`` after boot and
calls ``toolbox.load_tool(config_path)`` per entry. Under strict
``LazyToolBox.create_tool`` every converter raised ``RuntimeError``
(caught + logged by the registry as ``Error loading converter (…)``)
and the ``datatype_converters`` dict stayed silently empty — so
format-mismatch conversions via
``Dataset.find_conversion_destination`` were no-ops.
Fix: in ``discover_tools()``, after the hidden-lib block, query
``galaxy.model._get_datatypes_registry()`` and yield a
``DiscoveredTool`` for each converter the registry has parsed. Same
source of truth ``load_datatype_converters`` iterates, so we can't
drift — what the registry will try to load is exactly what the
populator indexes.
The registry is populated by ``set_datatypes_registry()`` at app boot
(``app/__init__.py:853``) before the toolbox initialises and inside
``populate_store(config_file=…)`` before ``populate_store_inline`` runs
— both paths produce a live registry by the time the populator walks.
On the (atypical) case where the registry isn't set, the try/except
falls through and the converters remain unindexed; the eager
``load_datatype_converters`` then continues to catch + log the
``Error loading converter`` messages, matching today's pre-refactor
behaviour.
After this commit, ``Persisted ToolIndex for store __default__``
reports 580 entries (up from 564 = +16 converters that
``sample_tool_conf.xml`` doesn't list) and zero ``Error loading
converter`` messages appear in the integration log. 16/16 integration
tests + 113/113 unit tests still pass.
``_create_tool_from_stored_source`` was hitting the install database on
every shed-tool materialise to fetch ``ToolShedRepository`` just so the
Tool ctor's ``populate_tool_shed_info`` could stamp four scalar fields.
The eager pipeline already has a namedtuple stub for exactly this case
(``ToolConfRepository`` in ``lib/galaxy/tool_util/toolbox/base.py:87``,
used for shed installs whose install-DB row hasn't appeared yet) —
build it directly from ``ToolIndexEntry``.
Eliminates:
- ``_lookup_tool_shed_repository`` method (~30 lines).
- ``galaxy.tool_shed.util.repository_util.get_installed_repository``
module-level import.
- A DB round-trip per shed-tool materialise.
``installed_tool_dependencies`` readers see
``tool_dependencies_installed_or_in_error=[]`` for materialised lazy
tools — same shape the eager pipeline produces when the install-DB
row is missing, so downstream callers already tolerate this. If a job
runner ever needs real DB-backed dependency info, it can query
``app.install_model`` directly using the scalar shed metadata already
on ``Tool``.
16/16 integration tests still pass.
Every inline import in these two files was checked against:
1. Does it create an actual import cycle? (No for any of them.)
2. Is it a soft / optional dependency? (kombu + watchdog are in
``pyproject.toml`` and ``pinned-requirements.txt``; the
try/except-ImportError defenses were dead code.)
The "lazy import for startup speed" justifications on the CLI entry
points (``galaxy.config``, ``galaxy.model``, ``galaxy.datatypes.registry``,
``galaxy.tool_source_store.search``, etc.) were also bogus — the
populator module is imported at Galaxy boot via the LazyToolBox
cold-start hook, so those heavy modules get pulled in anyway. The
inline form only delayed the cost by milliseconds while making the
dep graph invisible at the top of the file.
Hoisted:
- ``lazy_toolbox.py``: ``galaxy.exceptions``, ``galaxy.tool_shed.util.repository_util``,
``galaxy.util.tool_version``.
- ``populator.py``: ``kombu``, ``watchdog``, ``galaxy.config``,
``galaxy.datatypes.registry``, ``galaxy.model``, ``galaxy.model.mapping``,
``galaxy.queues``, ``galaxy.tool_source_store.search`` (Tuning/Whoosh),
``galaxy.util.properties``. The watchdog ``ImportError`` defense and
the kombu inline block both drop with them.
Galaxy boot still works; 113/113 unit tests + 16/16 integration tests
pass; CLI ``populate_store.py --help`` still resolves. The
``TYPE_CHECKING`` block in ``lazy_toolbox.py`` (the only remaining
function-scope ``from``-import area) keeps its lookup-only imports
where they belong.
``galaxy.tools.special_tools`` already enumerates Galaxy-internal tool
files loaded outside the conf walk (``load_hidden_lib_tool`` → the four
``imp_exp`` / ``data_fetch`` entries, plus ``set_metadata_tool.xml``
which the datatypes registry loads separately). The lazy refactor was
missing index entries for those, so the previous step relaxed
``LazyToolBox.create_tool`` to fall through to the eager parent on miss
to keep boot working.
Add ``hidden_lib_tool_paths()`` in ``special_tools.py``: returns the
absolute paths of every hidden-lib tool, including
``set_metadata_tool.xml``. ``SPECIAL_TOOLS`` (consumed by
``load_lib_tools``) stays unchanged so ``set_metadata_tool`` isn't
double-loaded — the new ``_EXTRA_HIDDEN_LIB_TOOLS`` dict only feeds
the populator-facing helper.
``galaxy.tool_source_store.discover.discover_tools`` walks those paths
after the conf + bundled walks, yielding ``DiscoveredTool`` entries
with ``tool_conf="<hidden-lib>"``. The populator's cold-start scan
indexes them like any other tool, so the post-boot
``load_hidden_lib_tool`` calls now hit the index branch.
With coverage complete, ``LazyToolBox.create_tool`` reverts to the
strict raise the plan called for: a miss is now a contract failure
(operator forgot to repopulate, or a new ad-hoc tool load didn't get
added to ``hidden_lib_tool_paths``). The unit test asserts the raise
again.
15/16 integration tests still pass; the remaining failure is the
pre-existing flake ``test_run_specific_version_executes_that_version``.
Two boot-time and panel-render fixes that the integration suite
surfaced once cold-start indexing covered every conf-discovered tool:
1. **Version defaulting in ``build_index_entry_from_source``.** Calling
``tool_source.parse_version()`` directly returns ``None`` for tools
without an explicit ``<tool version="...">`` (test fixtures like
``sam_to_unsorted_bam.xml`` and ``implicit_conversion.xml``). The
eager pipeline routes through
``galaxy.tool_util.parser.util.parse_tool_version_with_defaults``
which falls back to ``"1.0.0"`` on profiles < 16.04. Use the same
helper at index-build time so ``LazyTool.version`` is never
``None`` and ``ToolLineage.register_version`` can parse it.
2. **Materialise fallback in ``LazyTool.to_dict(link_details=True)``.**
``to_panel_view`` materialises every panel-visible tool to call
``to_dict(link_details=True)``. Some tools have XML the parameter
factory chokes on at materialise time (``upload1``'s
``upload_dataset`` input_type, ``filter_data_table.xml``'s
``column="value"`` dynamic-options filter). The eager toolbox
catches these in ``_load_tool_tag_set`` at boot and drops the tool
from ``_tools_by_id``; the lazy path postpones the failure to first
``to_dict``, where it now logs a WARNING and falls back to the
entry-only fast-path dict so the panel render still completes.
15/16 ``test_tool_source_storage.py`` integration tests pass; the
remaining failure is the noted pre-existing
``test_run_specific_version_executes_that_version`` flake.
Galaxy vendors a ``parse_version`` (lib/galaxy/tool_util/version.py:45)
that falls back to ``LegacyVersion`` when ``packaging``'s ``Version``
raises ``InvalidVersion`` — the eager toolbox uses it
(lib/galaxy/tools/__init__.py:128). The new lazy paths (LazyTool's
``version_object`` / ``_ver_key``, ``ToolIndex.add_entry``'s
multi-version tiebreaker) were importing ``packaging.version.parse``
directly, bypassing the LegacyVersion fallback.
Swap to the vendored function in three places:
- ``lib/galaxy/tools/lazy_toolbox.py`` module-level import.
- ``ToolIndex.add_entry``'s inline import.
- The matching comment in ``_ver_key``.
No behaviour change for versions that parse as PEP-440; the difference
shows up on Galaxy-specific version strings that need
``LegacyVersion`` semantics.
``AbstractToolBox._newer_tool`` (lib/galaxy/tool_util/toolbox/base.py:1500)
compares ``tool1.version_object > tool2.version_object``; it's reached
from ``_lineage_in_panel`` during ``to_panel_view`` rendering. Without
the property the strict ``__getattr__`` raised ``NotImplementedError``
for ~hundreds of tools and the API returned 500.
Compute it the same way ``Tool.version_object`` does
(lib/galaxy/tools/__init__.py:1213) — parse the entry's ``version``
string, special-casing the ``+galaxy`` suffix per PEP-440. ``parse_version``
is already imported at module top.
Three review-driven cleanups on top of commit 7ae5bf442e:
1. **``ToolSource.to_string()``** is the canonical serialiser on the
parser interface (``lib/galaxy/tool_util/parser/interface.py:450``)
and is implemented by every concrete source (XML, YAML, CWL). Use
it instead of branching on ``xml_tree`` / falling back to raw file
read — fewer special cases, correct for every source class.
2. **Drop ``populate_single_path``.** It was a heavyweight recovery
for ``load_hidden_lib_tool`` paths that aren't in any tool_conf.
The cleaner answer: on index miss, ``LazyToolBox.create_tool``
delegates to ``super().create_tool(...)``. The eager parent parses
the file and registers a real ``Tool`` (not a ``LazyTool``) — lib
tools like ``set_metadata_tool.xml`` live in memory, never enter
the populator-owned store, and re-load fresh on every boot. The
"raise on miss" contract is downgraded to "fall through to eager"
— the populator still owns every tool that *should* be in the
index, and the test seam (``test_create_tool_falls_through_to_eager_on_index_miss``)
asserts the new behaviour.
3. **Hoist inline imports** in both files. Module-level
``StoredToolSource`` / ``ToolSourceStore`` / ``ReadOnlyStoreError`` /
``DiscoveredTool`` / ``discover_tools`` / ``ToolIndex`` /
``ToolIndexEntry`` / ``ToolSection`` / ``populate_store_inline`` /
``get_toolbox_parser`` / ``get_tool_source`` so the structural deps
are visible at the top of each module. The heavy CLI-only imports
(``galaxy.config``, ``galaxy.model``, ``galaxy.datatypes.registry``,
the whoosh ``ToolSearchTuning`` / ``ToolWhooshIndex`` pair) stay
inline — they're optional, used only on populator entry points,
and would slow ``--help`` if pulled in.
14/16 integration tests pass; the two remaining failures predate this
stack (``test_run_specific_version_executes_that_version`` is the
noted pre-existing flake, ``test_default_panel_view_section_tools_use_id_list``
needs separate investigation).
After the populator/single-writer refactor, ``LazyToolBox.create_tool``
raises ``RuntimeError`` on index miss (commit ``56a66da6a9``). That broke
``load_hidden_lib_tool`` paths: ``set_metadata_tool.xml`` (and other
Galaxy-internal tools loaded from ``lib/galaxy/datatypes/`` after boot)
live outside any tool_conf and ``discover_tools`` doesn't yield them, so
the populator never indexed them.
New ``galaxy.tool_source_store.populator.populate_single_path`` parses
one tool file by absolute path, builds a ``StoredToolSource`` +
``ToolIndexEntry`` (with no panel section — the tool isn't in any conf),
writes it to every writable store, and updates the cached index.
Errors are logged and rolled back; the session is left clean.
``LazyToolBox.create_tool`` now retries via ``populate_single_path`` on
miss before raising. The single-writer contract is preserved (only the
populator writes), the raise still fires when the file genuinely doesn't
exist on disk, and the test seam ``test_create_tool_raises_on_index_miss``
still passes because the path ``/tools/unknown.xml`` doesn't exist.
Also adds ``LazyTool.tool_shed_repository`` (forwarded from
``_overrides``, defaults to ``None``) so the eager pipeline's
shed-metadata setattr doesn't trip the strict ``__getattr__``. Lib tools
have no repo; shed tools store theirs via the override.
Per-tool commits in ``populate_store_inline``'s store loop release the
write transaction between tools so SQLite readers (the test client's
``/api/tools`` request, the queue worker) aren't blocked behind a
484-row write transaction. The error-handling path rolls back the
shared session so a flush failure doesn't leak ``PendingRollbackError``
to the next caller.
``load_index`` already does this — the comment there explains why: each
``session.execute(select(...))`` on the shared scoped session opens an
implicit read transaction that nothing later closes, so on SQLite the
read lock blocks subsequent writers (in particular the cold-start
populator), and on Postgres it accumulates idle-in-transaction rows.
Apply the same ``finally: session.rollback()`` pattern to ``get``,
``exists``, ``get_by_tool_id``, and ``get_by_source_path``. The
populator and ``LazyToolBox._index_needs_population`` call these in
tight loops at cold boot (484× ``exists`` and 484× ``get_by_source_path``
on a default checkout), so the leak compounded fast: integration tests
hit ``OperationalError: database is locked`` on the very next
``SELECT FROM tool_index`` from the test client.
``rollback()`` after a pure read is a no-op for data but closes the
transaction.
``app.toolbox_search`` becomes the single search singleton for both
toolbox flavours. In lazy mode it's :class:`LazyToolboxSearch` — a
``ToolBoxSearch`` subclass that opens the populator-owned whoosh dir
(``tool_search_index_dir/_lazy_default``) on every ``search()`` call and
no-ops ``build_index`` (the populator does the writes). In eager mode it
stays :class:`ToolBoxSearch` as before.
Effects:
- ``services/tools.py:search_tools`` drops its ``_get_lazy_toolbox`` branch
and just calls ``self._search(query, view)``. One code path, regardless
of toolbox flavour.
- ``LazyToolBox.search_tools``, ``_get_search_index``, the
``_whoosh_search_index`` slot, and the whoosh clear in
``invalidate_index_cache`` all drop. The unused
``ToolSearchTuning`` / ``ToolWhooshIndex`` imports go with them.
- ``LazyToolboxSearch`` keeps ``index_count`` so the existing
``queue_worker.rebuild_toolbox_search_index`` watermark check is
satisfied without doing work.
``lazy_toolbox.py`` drops a further ~55 lines (1257 → 1202).
69/69 unit tests still pass.
In-process callers (LazyToolBox cold-start, ``populate_for_paths`` on
shed install, ``reconcile_index`` for ``reset_shed_tools``) share
Galaxy's ``app.model.context`` SQLAlchemy session. The
``DatabaseToolSourceStore.store()`` write path goes through that
session, and ``Session`` is not thread-safe — running the
``ThreadPoolExecutor`` with the previous default of 4 workers caused
SQLite "database is locked" errors and a full integration suite that
took 51 minutes with 13 failures.
The CLI is unaffected: ``populate_store(config_file=...)`` constructs
its own ``model.context`` via ``init_models_from_config`` (no concurrent
readers) and explicitly passes ``parallel=args.parallel`` (defaults to 4),
so ``populate_store.py --parallel N`` still fans out.
``ToolFileWatcher._process_tool_file`` used to write only
``StoredToolSource`` on a content change, leaving the ``ToolIndex``
stale until the next full populator run. After this commit, the
watcher delegates to ``populate_for_paths`` once the lightweight hash
check confirms the content actually changed — same single-writer entry
point shed installs use — so the index, whoosh search, and peer-process
reload broadcast all stay in sync with on-disk edits in dev mode.
``ToolFileWatcher`` gains an optional ``sa_session`` arg; ``watch_mode``
passes the freshly-initialised ``model.context``. Callers that pass
``None`` (legacy / test mocks) fall back to the previous store-only
behaviour with a one-line "index left stale" log — the contract is
preserved for the 25 fake-store unit tests that exercise the watcher
without a DB.
69/69 unit tests still pass.
After ``setup_shed_config`` blanks ``shed_tool_conf.xml``, the toolbox
reload that follows would otherwise inherit stale shed-tool
``ToolIndexEntry`` rows: the populator-driven cold-start check only
re-runs when discovery finds a *new* path, not when it finds *fewer*
than the index already has.
Calling ``galaxy.tool_source_store.populator.reconcile_index`` between
the conf rewrite and the ``reload_toolbox`` broadcast prunes those
orphans. The next boot reads a clean index that matches the empty
shed conf; integration tests that depend on ``reset_shed_tools``
leaving zero shed entries (``test_repository_*``) get the contract
they want.
Wrapped in try/except so a missing populator dependency in a stripped-
down test environment doesn't break the rest of the reset path.
Adds a module-level logger.
``add_to_tool_panel`` now writes ``shed_tool_conf.xml`` *before* loading
items into the toolbox, then calls
``galaxy.tool_source_store.populator.populate_for_paths`` on the new
tool file paths. The populator writes ``StoredToolSource`` +
``ToolIndexEntry`` + the whoosh index for each path and broadcasts
``reload_tool_source_cache`` so peer Galaxy processes refresh.
After the populator returns, the in-process toolbox's
``invalidate_index_cache`` is called synchronously so this process sees
the new entries immediately (the install API response can't wait for
AMQP loopback). Commit 11's stub-registration runs inside that call.
``load_item`` is still invoked per elem so the in-memory ``_tool_panel``
and ``_integrated_tool_panel`` get the standard panel bookkeeping —
``create_tool``'s index lookup now succeeds (commit 10's raise stays
quiet) because the populator just wrote the entry.
New helper ``_collect_new_tool_paths`` walks ``elem_list``, flattening
``<section>`` children, and returns absolute paths matching what
``discover_tools`` yields after the conf is on disk.
When a peer process runs the populator (shed install reroute in commit
12, ``reset_shed_tools`` in commit 13, ``--watch`` mode), it broadcasts
``reload_tool_source_cache`` which lands in every Galaxy process's
``invalidate_index_cache``. Until now that only re-read the index — the
new ids were reachable via ``get_tool`` lookups but didn't appear in
``_tools_by_id`` until the next boot, so ``/api/tools`` under-reported.
Two new helpers close the gap:
- ``_register_new_index_entries_as_stubs`` walks the freshly-loaded
``_tool_index.entries`` and registers anything not already in
``_tools_by_id``.
- ``_register_lazy_entry(entry)`` is the inverse of the existing
``_register_loaded_tool``: builds a ``LazyTool`` stub, populates
``_tools_by_id`` / ``_tool_versions_by_id`` / ``_tools_by_old_id`` /
``_tools_by_uuid``, assigns ``_lineage`` via ``LazyLineageMap.get``,
and slots the stub into ``_tool_panel`` under its declared
``panel_section_id`` (creating the ``ToolSection`` if absent).
This is the broadcast-side hook that makes peer-process installs visible
in this process without waiting for a full toolbox reload.
22/22 unit tests still pass.
mvdbeek