The job-request path creates and reads tool_source rows whose source
column carries the raw source string; the store was writing dict-shaped
payloads into the same table, and once identity hashes aligned the job
path could resolve a store row and fail pydantic validation (500 on
every request-style tool execution under lazy mode).
Give the store its own content-addressed table with real columns —
get_by_tool_id / get_by_source_path become indexed queries instead of
full-table JSON scans, populator pruning can never delete rows the job
path references, and the identity_hash coupling (and the legacy
__tool_index__ sentinel-row fallback) disappear.
- AbstractToolBox gains a no-op invalidate_index_cache (LazyToolBox
overrides it); shed installs no longer depend on the subclass type.
- Annotate LazyToolboxSearch.build_index, ToolFileWatcher.observer,
populator session cast, UUID key coercion in _register_lazy_entry,
and drop stale type: ignore comments.
The lazy path persists shed_tool_conf before load_item because the
populator's conf walk needs it on disk; that same reorder broke eager
version-replacement in the panel (test_add_twice). Split the paths:
eager is now byte-identical to upstream, the reorder applies only under
use_lazy_toolbox.
The lazy ``to_dict`` fast path was short-circuiting *both* the
panel-view walk (intended) *and* the ``/api/tools/{id}`` show endpoint
(unintended) — they're indistinguishable at the ``to_dict(io_details=
False, link_details=False)`` callsite. ``test_legacy_biotools_xref_
injection`` caught it: the show endpoint returned the entry-shape
dict and no longer carried ``xrefs``.
Split the two call sites at the API rather than the parameter:
* ``Tool.to_panel_entry(trans)`` — new method, returns the cheap
entry-shape dict (id/name/version/description/labels/edam/panel
section/link). ``LazyTool`` overrides it to skip materialise.
* ``AbstractToolBox.get_tool_to_dict`` calls ``to_panel_entry`` when
``tool_help=False`` (the panel-view default); ``tool_help=True``
still falls through to ``to_dict`` so the help payload renders.
* ``LazyTool.to_dict`` always materialises now — it's the
``/api/tools/{id}`` contract and must ship ``xrefs`` / ``versions``
/ ``is_workflow_compatible`` / ``tool_shed_repository`` / etc. The
materialise-failure fallback now reuses ``to_panel_entry``.
Net effect: ``/api/tool_panels/{view}`` and ``/api/tools?in_panel=
true`` stay materialise-free for the lazy toolbox; ``/api/tools/{id}``
serves the full show-endpoint dict in both modes.
`LazyTool` materialise during workflow ``inject_all`` calls
``store.get(hash)``; the store ran the SELECT on Galaxy's shared
scoped session and then called ``session.rollback()`` to release the
implicit read transaction. That rollback expired *every*
request-scoped instance attached to the shared session — including the
in-flight ``workflow.steps`` collection. The next access in the same
``populate_module_and_state`` re-fetched a fresh ``WorkflowStep`` list
from the DB, with ``.module`` unset on the new instances, so
``compute_runtime_state``'s ``assert step.module`` raised
``AttributeError: 'WorkflowStep' object has no attribute 'module'``.
Every workflow-invocation test under ``use_lazy_toolbox=true`` 500'd.
Fix: read methods on ``DatabaseToolSourceStore`` (``get``, ``exists``,
``get_by_tool_id``, ``get_by_source_path``, ``count``, ``list_all``,
``load_index``) now open a private ``Session`` bound to the same
engine and close it on exit. The shared scoped session — and its
caller's transaction state — is untouched. The original lock-release
intent (cold-start populator, queue-worker init) still holds because
each private session's read transaction ends when the session closes.
Writes (``store``, ``store_index``, ``delete``, ``update_index_entry``)
still use ``_get_session()`` so they participate in the caller's
transaction and commit in the right context (populator, queue worker,
``LazyToolBox.__init__``).
With ``link_details`` no longer gating any field, the toolbox-internal
panel / listing callers stopped passing it (``f64858bf55e``).
``LazyTool.to_dict`` can now return the entry-shape dict for the
default-call case — every panel-view tool serves from the index, zero
materialise.
Materialise still fires when the caller asks for the parsed-Tool
payload: ``io_details=True`` (the ``/api/tools/{id}`` show endpoint
contract) or ``tool_help=True`` (rendered help). Failure-to-materialise
still falls back to the entry-shape dict so the show endpoint never
returns 500 when a specific tool's XML chokes the parameter factory.
The ``link`` field is derived from ``self.id`` matching what
``Tool.to_dict`` now emits unconditionally — ``/tool_runner?tool_id={id}``
for the standard tool runner.
Tests updated:
- ``test_to_dict_entry_fast_path_does_not_materialise`` asserts the
default call serves the entry dict and never invokes the callback.
- ``test_to_dict_io_details_materialises`` covers the show contract.
- ``test_to_dict_falls_back_to_entry_when_materialise_fails`` covers the
parameter-factory-failure case.
23/23 unit tests pass.
These were emitted on every entry of /api/tools but no client component
ever read them. The Vue panel renderer derives the link target from
``model_class == "DataSourceTool"`` via ``getTargetById`` and never reads
``min_width`` at all; the Tool interface declared the fields purely to
match the wire shape. ``Tool.target`` / ``Tool.uihints`` likewise had
``to_dict`` as their sole reader in lib/.
Removed:
- ``self.target`` / ``self.uihints`` attributes and the corresponding
``<inputs target>`` and ``<uihints minwidth>`` reads in ``__parse_legacy_features``
- ``DataSourceTool.parse_inputs`` self.target = "_top" override (dead)
- ``min_width`` / ``target`` keys from both the eager ``to_dict`` payload
and the ``LazyTool.to_dict`` stub
The XSD declarations for ``<uihints minwidth>`` and ``<inputs target>``
stay so existing data-source tool XML continues to validate cleanly.
After five rounds of CI surfacing missing stub attrs and the latest
round still hitting indirect failures (workflow steps with ``module``
unset because some inner attribute access raised under strict), the
explicit ``_MATERIALIZE_OK`` set has stopped converging. The strict
guard caught the obvious surface and now slows ship velocity without
adding signal.
Flip the default: unknown attr reads on ``LazyTool`` now log a WARNING
and materialise. The strict raise is still available for debugging via
``LAZY_TOOL_STRICT=1`` — the path that surfaces ``add to the stub
surface or _MATERIALIZE_OK`` for new audit work.
``test_strict_getattr_raises_with_clear_message`` now monkey-patches
the module flag to force strict mode for the assertion. Other tests are
unchanged.
The entry-only fast path was missing fields the ``/api/tools/<id>``
show endpoint serves (``xrefs``, ``inputs`` when ``io_details=False``,
…). The flat ``/api/tools?in_panel=False`` listing — the only caller
that benefits from the fast path — already bypasses this method:
``services/tools.py:list_tools`` walks ``tool_index.list_all()`` and
calls ``entry.to_api_dict()`` directly.
Always materialise here; the entry-only dict survives as the fallback
when materialise itself raises (a tool whose XML the parameter
factory chokes on — ``upload_dataset``, ``column="value"``, …) so the
caller still gets *something* renderable rather than a 500.
Tests updated:
- ``test_to_dict_fast_path_does_not_materialise`` → renamed to
``test_to_dict_falls_back_to_entry_dict_when_materialise_fails``.
- ``test_to_dict_link_details_materialises_exactly_once`` → renamed
for accuracy; behaviour unchanged.
``/api/tools/<id>`` show endpoint passes ``io_details=True`` to surface
``inputs`` / ``outputs`` / parameter detail; the previous condition
only materialised on ``link_details=True``, so the listing-shape dict
came back without inputs and ``test_show_repeat`` / ``test_show_multi_data``
asserted on a missing key.
Add ``io_details`` to the materialise predicate. Forward both flags into
``Tool.to_dict`` so the parent picks the right payload variant. The
entry-only fast path now only applies to the truly cheap case (the
``/api/tools?in_panel=False`` listing).
Materialise failure still falls back to the entry-only dict — listing
remains 200 even if a specific tool can't materialise.
``tool_action`` graduated to ``_MATERIALIZE_OK`` in the previous commit
(it's read on the execute path). The strict-raise assertion needs an
attribute the stub neither forwards nor materialises;
``totally_not_a_tool_attr`` is guaranteed unaccounted-for.
Round-2 CI surfaced five more attributes the tool-execution path
hits before the parameter machinery has finished setting up:
- ``check_and_update_param_values`` — parameter validation called from
``handle_input``.
- ``wants_params_cleaned`` — parameter scrub before execution.
- ``tool_source`` — raw ``ToolSource`` (some callers walk the XML
directly).
- ``dynamic_tool`` — dynamic-tool linkage on execution.
- ``produces_entry_points`` — interactive-tool entry-point lookup.
All five materialise correctly when read; adding to ``_MATERIALIZE_OK``
makes the stub's strict ``__getattr__`` route through the cached
``_real`` Tool on first access instead of raising.
CI surfaced six attributes that the tool-execution path
(``/api/tools/{id}`` POST → ``handle_input``) and the job runner read
on the resulting Tool, all of which fundamentally need a parsed
parameter tree:
- ``handle_input`` — entry point for tool execution.
- ``inputs``, ``parameters``, ``new_state`` — the parameter machinery.
- ``input_translator`` — runtime parameter translation.
- ``requires_galaxy_python_environment`` — job-runner environment hint.
Each was raising ``NotImplementedError`` for every tool-execution
request, taking out ~hundreds of API tests at once. Add them to
``_MATERIALIZE_OK`` so the stub materialises on first read and the
real Tool answers the rest of the call.
Materialise on tool execution is exactly the cost model the lazy
toolbox is designed for — pay parse cost only when the tool is
actually used. The stub still surfaces unaccounted-for attribute reads
loudly, which is the contract the user picked.
``Registry.load_datatype_converters`` (``lib/galaxy/datatypes/registry.py:674``)
walks the converter list from ``datatypes_conf.xml`` after boot and
calls ``toolbox.load_tool(config_path)`` per entry. Under strict
``LazyToolBox.create_tool`` every converter raised ``RuntimeError``
(caught + logged by the registry as ``Error loading converter (…)``)
and the ``datatype_converters`` dict stayed silently empty — so
format-mismatch conversions via
``Dataset.find_conversion_destination`` were no-ops.
Fix: in ``discover_tools()``, after the hidden-lib block, query
``galaxy.model._get_datatypes_registry()`` and yield a
``DiscoveredTool`` for each converter the registry has parsed. Same
source of truth ``load_datatype_converters`` iterates, so we can't
drift — what the registry will try to load is exactly what the
populator indexes.
The registry is populated by ``set_datatypes_registry()`` at app boot
(``app/__init__.py:853``) before the toolbox initialises and inside
``populate_store(config_file=…)`` before ``populate_store_inline`` runs
— both paths produce a live registry by the time the populator walks.
On the (atypical) case where the registry isn't set, the try/except
falls through and the converters remain unindexed; the eager
``load_datatype_converters`` then continues to catch + log the
``Error loading converter`` messages, matching today's pre-refactor
behaviour.
After this commit, ``Persisted ToolIndex for store __default__``
reports 580 entries (up from 564 = +16 converters that
``sample_tool_conf.xml`` doesn't list) and zero ``Error loading
converter`` messages appear in the integration log. 16/16 integration
tests + 113/113 unit tests still pass.
``_create_tool_from_stored_source`` was hitting the install database on
every shed-tool materialise to fetch ``ToolShedRepository`` just so the
Tool ctor's ``populate_tool_shed_info`` could stamp four scalar fields.
The eager pipeline already has a namedtuple stub for exactly this case
(``ToolConfRepository`` in ``lib/galaxy/tool_util/toolbox/base.py:87``,
used for shed installs whose install-DB row hasn't appeared yet) —
build it directly from ``ToolIndexEntry``.
Eliminates:
- ``_lookup_tool_shed_repository`` method (~30 lines).
- ``galaxy.tool_shed.util.repository_util.get_installed_repository``
module-level import.
- A DB round-trip per shed-tool materialise.
``installed_tool_dependencies`` readers see
``tool_dependencies_installed_or_in_error=[]`` for materialised lazy
tools — same shape the eager pipeline produces when the install-DB
row is missing, so downstream callers already tolerate this. If a job
runner ever needs real DB-backed dependency info, it can query
``app.install_model`` directly using the scalar shed metadata already
on ``Tool``.
16/16 integration tests still pass.
Every inline import in these two files was checked against:
1. Does it create an actual import cycle? (No for any of them.)
2. Is it a soft / optional dependency? (kombu + watchdog are in
``pyproject.toml`` and ``pinned-requirements.txt``; the
try/except-ImportError defenses were dead code.)
The "lazy import for startup speed" justifications on the CLI entry
points (``galaxy.config``, ``galaxy.model``, ``galaxy.datatypes.registry``,
``galaxy.tool_source_store.search``, etc.) were also bogus — the
populator module is imported at Galaxy boot via the LazyToolBox
cold-start hook, so those heavy modules get pulled in anyway. The
inline form only delayed the cost by milliseconds while making the
dep graph invisible at the top of the file.
Hoisted:
- ``lazy_toolbox.py``: ``galaxy.exceptions``, ``galaxy.tool_shed.util.repository_util``,
``galaxy.util.tool_version``.
- ``populator.py``: ``kombu``, ``watchdog``, ``galaxy.config``,
``galaxy.datatypes.registry``, ``galaxy.model``, ``galaxy.model.mapping``,
``galaxy.queues``, ``galaxy.tool_source_store.search`` (Tuning/Whoosh),
``galaxy.util.properties``. The watchdog ``ImportError`` defense and
the kombu inline block both drop with them.
Galaxy boot still works; 113/113 unit tests + 16/16 integration tests
pass; CLI ``populate_store.py --help`` still resolves. The
``TYPE_CHECKING`` block in ``lazy_toolbox.py`` (the only remaining
function-scope ``from``-import area) keeps its lookup-only imports
where they belong.
``galaxy.tools.special_tools`` already enumerates Galaxy-internal tool
files loaded outside the conf walk (``load_hidden_lib_tool`` → the four
``imp_exp`` / ``data_fetch`` entries, plus ``set_metadata_tool.xml``
which the datatypes registry loads separately). The lazy refactor was
missing index entries for those, so the previous step relaxed
``LazyToolBox.create_tool`` to fall through to the eager parent on miss
to keep boot working.
Add ``hidden_lib_tool_paths()`` in ``special_tools.py``: returns the
absolute paths of every hidden-lib tool, including
``set_metadata_tool.xml``. ``SPECIAL_TOOLS`` (consumed by
``load_lib_tools``) stays unchanged so ``set_metadata_tool`` isn't
double-loaded — the new ``_EXTRA_HIDDEN_LIB_TOOLS`` dict only feeds
the populator-facing helper.
``galaxy.tool_source_store.discover.discover_tools`` walks those paths
after the conf + bundled walks, yielding ``DiscoveredTool`` entries
with ``tool_conf="<hidden-lib>"``. The populator's cold-start scan
indexes them like any other tool, so the post-boot
``load_hidden_lib_tool`` calls now hit the index branch.
With coverage complete, ``LazyToolBox.create_tool`` reverts to the
strict raise the plan called for: a miss is now a contract failure
(operator forgot to repopulate, or a new ad-hoc tool load didn't get
added to ``hidden_lib_tool_paths``). The unit test asserts the raise
again.
15/16 integration tests still pass; the remaining failure is the
pre-existing flake ``test_run_specific_version_executes_that_version``.
Two boot-time and panel-render fixes that the integration suite
surfaced once cold-start indexing covered every conf-discovered tool:
1. **Version defaulting in ``build_index_entry_from_source``.** Calling
``tool_source.parse_version()`` directly returns ``None`` for tools
without an explicit ``<tool version="...">`` (test fixtures like
``sam_to_unsorted_bam.xml`` and ``implicit_conversion.xml``). The
eager pipeline routes through
``galaxy.tool_util.parser.util.parse_tool_version_with_defaults``
which falls back to ``"1.0.0"`` on profiles < 16.04. Use the same
helper at index-build time so ``LazyTool.version`` is never
``None`` and ``ToolLineage.register_version`` can parse it.
2. **Materialise fallback in ``LazyTool.to_dict(link_details=True)``.**
``to_panel_view`` materialises every panel-visible tool to call
``to_dict(link_details=True)``. Some tools have XML the parameter
factory chokes on at materialise time (``upload1``'s
``upload_dataset`` input_type, ``filter_data_table.xml``'s
``column="value"`` dynamic-options filter). The eager toolbox
catches these in ``_load_tool_tag_set`` at boot and drops the tool
from ``_tools_by_id``; the lazy path postpones the failure to first
``to_dict``, where it now logs a WARNING and falls back to the
entry-only fast-path dict so the panel render still completes.
15/16 ``test_tool_source_storage.py`` integration tests pass; the
remaining failure is the noted pre-existing
``test_run_specific_version_executes_that_version`` flake.
Galaxy vendors a ``parse_version`` (lib/galaxy/tool_util/version.py:45)
that falls back to ``LegacyVersion`` when ``packaging``'s ``Version``
raises ``InvalidVersion`` — the eager toolbox uses it
(lib/galaxy/tools/__init__.py:128). The new lazy paths (LazyTool's
``version_object`` / ``_ver_key``, ``ToolIndex.add_entry``'s
multi-version tiebreaker) were importing ``packaging.version.parse``
directly, bypassing the LegacyVersion fallback.
Swap to the vendored function in three places:
- ``lib/galaxy/tools/lazy_toolbox.py`` module-level import.
- ``ToolIndex.add_entry``'s inline import.
- The matching comment in ``_ver_key``.
No behaviour change for versions that parse as PEP-440; the difference
shows up on Galaxy-specific version strings that need
``LegacyVersion`` semantics.
``AbstractToolBox._newer_tool`` (lib/galaxy/tool_util/toolbox/base.py:1500)
compares ``tool1.version_object > tool2.version_object``; it's reached
from ``_lineage_in_panel`` during ``to_panel_view`` rendering. Without
the property the strict ``__getattr__`` raised ``NotImplementedError``
for ~hundreds of tools and the API returned 500.
Compute it the same way ``Tool.version_object`` does
(lib/galaxy/tools/__init__.py:1213) — parse the entry's ``version``
string, special-casing the ``+galaxy`` suffix per PEP-440. ``parse_version``
is already imported at module top.
Three review-driven cleanups on top of commit 7ae5bf442e:
1. **``ToolSource.to_string()``** is the canonical serialiser on the
parser interface (``lib/galaxy/tool_util/parser/interface.py:450``)
and is implemented by every concrete source (XML, YAML, CWL). Use
it instead of branching on ``xml_tree`` / falling back to raw file
read — fewer special cases, correct for every source class.
2. **Drop ``populate_single_path``.** It was a heavyweight recovery
for ``load_hidden_lib_tool`` paths that aren't in any tool_conf.
The cleaner answer: on index miss, ``LazyToolBox.create_tool``
delegates to ``super().create_tool(...)``. The eager parent parses
the file and registers a real ``Tool`` (not a ``LazyTool``) — lib
tools like ``set_metadata_tool.xml`` live in memory, never enter
the populator-owned store, and re-load fresh on every boot. The
"raise on miss" contract is downgraded to "fall through to eager"
— the populator still owns every tool that *should* be in the
index, and the test seam (``test_create_tool_falls_through_to_eager_on_index_miss``)
asserts the new behaviour.
3. **Hoist inline imports** in both files. Module-level
``StoredToolSource`` / ``ToolSourceStore`` / ``ReadOnlyStoreError`` /
``DiscoveredTool`` / ``discover_tools`` / ``ToolIndex`` /
``ToolIndexEntry`` / ``ToolSection`` / ``populate_store_inline`` /
``get_toolbox_parser`` / ``get_tool_source`` so the structural deps
are visible at the top of each module. The heavy CLI-only imports
(``galaxy.config``, ``galaxy.model``, ``galaxy.datatypes.registry``,
the whoosh ``ToolSearchTuning`` / ``ToolWhooshIndex`` pair) stay
inline — they're optional, used only on populator entry points,
and would slow ``--help`` if pulled in.
14/16 integration tests pass; the two remaining failures predate this
stack (``test_run_specific_version_executes_that_version`` is the
noted pre-existing flake, ``test_default_panel_view_section_tools_use_id_list``
needs separate investigation).
After the populator/single-writer refactor, ``LazyToolBox.create_tool``
raises ``RuntimeError`` on index miss (commit ``56a66da6a9``). That broke
``load_hidden_lib_tool`` paths: ``set_metadata_tool.xml`` (and other
Galaxy-internal tools loaded from ``lib/galaxy/datatypes/`` after boot)
live outside any tool_conf and ``discover_tools`` doesn't yield them, so
the populator never indexed them.
New ``galaxy.tool_source_store.populator.populate_single_path`` parses
one tool file by absolute path, builds a ``StoredToolSource`` +
``ToolIndexEntry`` (with no panel section — the tool isn't in any conf),
writes it to every writable store, and updates the cached index.
Errors are logged and rolled back; the session is left clean.
``LazyToolBox.create_tool`` now retries via ``populate_single_path`` on
miss before raising. The single-writer contract is preserved (only the
populator writes), the raise still fires when the file genuinely doesn't
exist on disk, and the test seam ``test_create_tool_raises_on_index_miss``
still passes because the path ``/tools/unknown.xml`` doesn't exist.
Also adds ``LazyTool.tool_shed_repository`` (forwarded from
``_overrides``, defaults to ``None``) so the eager pipeline's
shed-metadata setattr doesn't trip the strict ``__getattr__``. Lib tools
have no repo; shed tools store theirs via the override.
Per-tool commits in ``populate_store_inline``'s store loop release the
write transaction between tools so SQLite readers (the test client's
``/api/tools`` request, the queue worker) aren't blocked behind a
484-row write transaction. The error-handling path rolls back the
shared session so a flush failure doesn't leak ``PendingRollbackError``
to the next caller.
``load_index`` already does this — the comment there explains why: each
``session.execute(select(...))`` on the shared scoped session opens an
implicit read transaction that nothing later closes, so on SQLite the
read lock blocks subsequent writers (in particular the cold-start
populator), and on Postgres it accumulates idle-in-transaction rows.
Apply the same ``finally: session.rollback()`` pattern to ``get``,
``exists``, ``get_by_tool_id``, and ``get_by_source_path``. The
populator and ``LazyToolBox._index_needs_population`` call these in
tight loops at cold boot (484× ``exists`` and 484× ``get_by_source_path``
on a default checkout), so the leak compounded fast: integration tests
hit ``OperationalError: database is locked`` on the very next
``SELECT FROM tool_index`` from the test client.
``rollback()`` after a pure read is a no-op for data but closes the
transaction.
``app.toolbox_search`` becomes the single search singleton for both
toolbox flavours. In lazy mode it's :class:`LazyToolboxSearch` — a
``ToolBoxSearch`` subclass that opens the populator-owned whoosh dir
(``tool_search_index_dir/_lazy_default``) on every ``search()`` call and
no-ops ``build_index`` (the populator does the writes). In eager mode it
stays :class:`ToolBoxSearch` as before.
Effects:
- ``services/tools.py:search_tools`` drops its ``_get_lazy_toolbox`` branch
and just calls ``self._search(query, view)``. One code path, regardless
of toolbox flavour.
- ``LazyToolBox.search_tools``, ``_get_search_index``, the
``_whoosh_search_index`` slot, and the whoosh clear in
``invalidate_index_cache`` all drop. The unused
``ToolSearchTuning`` / ``ToolWhooshIndex`` imports go with them.
- ``LazyToolboxSearch`` keeps ``index_count`` so the existing
``queue_worker.rebuild_toolbox_search_index`` watermark check is
satisfied without doing work.
``lazy_toolbox.py`` drops a further ~55 lines (1257 → 1202).
69/69 unit tests still pass.
In-process callers (LazyToolBox cold-start, ``populate_for_paths`` on
shed install, ``reconcile_index`` for ``reset_shed_tools``) share
Galaxy's ``app.model.context`` SQLAlchemy session. The
``DatabaseToolSourceStore.store()`` write path goes through that
session, and ``Session`` is not thread-safe — running the
``ThreadPoolExecutor`` with the previous default of 4 workers caused
SQLite "database is locked" errors and a full integration suite that
took 51 minutes with 13 failures.
The CLI is unaffected: ``populate_store(config_file=...)`` constructs
its own ``model.context`` via ``init_models_from_config`` (no concurrent
readers) and explicitly passes ``parallel=args.parallel`` (defaults to 4),
so ``populate_store.py --parallel N`` still fans out.
``ToolFileWatcher._process_tool_file`` used to write only
``StoredToolSource`` on a content change, leaving the ``ToolIndex``
stale until the next full populator run. After this commit, the
watcher delegates to ``populate_for_paths`` once the lightweight hash
check confirms the content actually changed — same single-writer entry
point shed installs use — so the index, whoosh search, and peer-process
reload broadcast all stay in sync with on-disk edits in dev mode.
``ToolFileWatcher`` gains an optional ``sa_session`` arg; ``watch_mode``
passes the freshly-initialised ``model.context``. Callers that pass
``None`` (legacy / test mocks) fall back to the previous store-only
behaviour with a one-line "index left stale" log — the contract is
preserved for the 25 fake-store unit tests that exercise the watcher
without a DB.
69/69 unit tests still pass.
After ``setup_shed_config`` blanks ``shed_tool_conf.xml``, the toolbox
reload that follows would otherwise inherit stale shed-tool
``ToolIndexEntry`` rows: the populator-driven cold-start check only
re-runs when discovery finds a *new* path, not when it finds *fewer*
than the index already has.
Calling ``galaxy.tool_source_store.populator.reconcile_index`` between
the conf rewrite and the ``reload_toolbox`` broadcast prunes those
orphans. The next boot reads a clean index that matches the empty
shed conf; integration tests that depend on ``reset_shed_tools``
leaving zero shed entries (``test_repository_*``) get the contract
they want.
Wrapped in try/except so a missing populator dependency in a stripped-
down test environment doesn't break the rest of the reset path.
Adds a module-level logger.
``add_to_tool_panel`` now writes ``shed_tool_conf.xml`` *before* loading
items into the toolbox, then calls
``galaxy.tool_source_store.populator.populate_for_paths`` on the new
tool file paths. The populator writes ``StoredToolSource`` +
``ToolIndexEntry`` + the whoosh index for each path and broadcasts
``reload_tool_source_cache`` so peer Galaxy processes refresh.
After the populator returns, the in-process toolbox's
``invalidate_index_cache`` is called synchronously so this process sees
the new entries immediately (the install API response can't wait for
AMQP loopback). Commit 11's stub-registration runs inside that call.
``load_item`` is still invoked per elem so the in-memory ``_tool_panel``
and ``_integrated_tool_panel`` get the standard panel bookkeeping —
``create_tool``'s index lookup now succeeds (commit 10's raise stays
quiet) because the populator just wrote the entry.
New helper ``_collect_new_tool_paths`` walks ``elem_list``, flattening
``<section>`` children, and returns absolute paths matching what
``discover_tools`` yields after the conf is on disk.
When a peer process runs the populator (shed install reroute in commit
12, ``reset_shed_tools`` in commit 13, ``--watch`` mode), it broadcasts
``reload_tool_source_cache`` which lands in every Galaxy process's
``invalidate_index_cache``. Until now that only re-read the index — the
new ids were reachable via ``get_tool`` lookups but didn't appear in
``_tools_by_id`` until the next boot, so ``/api/tools`` under-reported.
Two new helpers close the gap:
- ``_register_new_index_entries_as_stubs`` walks the freshly-loaded
``_tool_index.entries`` and registers anything not already in
``_tools_by_id``.
- ``_register_lazy_entry(entry)`` is the inverse of the existing
``_register_loaded_tool``: builds a ``LazyTool`` stub, populates
``_tools_by_id`` / ``_tool_versions_by_id`` / ``_tools_by_old_id`` /
``_tools_by_uuid``, assigns ``_lineage`` via ``LazyLineageMap.get``,
and slots the stub into ``_tool_panel`` under its declared
``panel_section_id`` (creating the ``ToolSection`` if absent).
This is the broadcast-side hook that makes peer-process installs visible
in this process without waiting for a full toolbox reload.
22/22 unit tests still pass.
The populator (cold-start auto-populate at boot, ``populate_for_paths``
on shed install) is now the single writer of the index + whoosh. The
toolbox stops parsing tool XML in the boot path.
Deleted:
- ``_persist_tool_source`` (~76 lines) — the in-toolbox fallthrough that
parsed + stored + broadcasted on index miss.
- ``_build_index_entry_from_stored`` and ``_make_index_entry`` (~120
lines combined) — only called from ``_persist_tool_source``.
- The ``hashlib`` and ``datetime`` imports they pulled in.
``create_tool`` now raises ``RuntimeError`` on index miss with a message
pointing at the populator (CLI or in-process API). Cold-start
auto-populate (commit 9) guarantees coverage at boot, so a raise here
means an out-of-band conf change without a populator run — operator
action, not silent recovery.
``lazy_toolbox.py`` shrinks from 1314 to 1188 lines.
Added a unit test asserting the raise; the existing seam tests (LazyTool
return on guid + source-path hit, ``load_tool_from_cache``, etc.) still
pass. 22/22 unit tests green.
``_init_tools_from_configs`` now checks ``_index_needs_population()``
before delegating to the eager walk; if any config-discovered tool path
isn't in the store, it calls ``populate_store_inline(app.config,
model.context, rebuild_whoosh=True)`` and reloads the index.
The check short-circuits on the first miss — fresh checkouts (empty
store), new conf entries, and wiped stores all trigger a single inline
populator run. ``get_by_source_path`` is indexed on the SqlAlchemy store
backend, so the cost is one DB lookup per discovered tool.
This is the contract that lets ``create_tool`` raise on index miss
(commit 10) instead of silently parsing in-toolbox: boot guarantees the
index covers every conf-known tool before the eager walk runs. UX
matches today's behaviour — operator doesn't need to invoke the
populator script on first boot.
Failures from the inline populator are logged and swallowed for now;
the eager walk's still-present ``_persist_tool_source`` fallthrough
covers per-tool recovery. That belt-and-braces redundancy disappears in
the next commit.
All 21 unit tests still pass.
The populator (commit 5) now owns the on-disk whoosh index. The toolbox
becomes a read-only consumer:
- ``_get_search_index`` opens the cached ``ToolWhooshIndex`` once per
process; it never builds. The whoosh dir resolves via the shared
``whoosh_dir_for_store`` helper so the toolbox and populator can't
drift on path conventions.
- ``search_tools`` drops the ``ToolIndex.compute_version`` check + in-
process rebuild + the substring fallback that masked whoosh failures.
A missing or unreadable on-disk index now propagates the whoosh error
to the caller — operator action ("run the populator") instead of a
silently-degraded scorer response.
- ``invalidate_index_cache`` clears ``self._whoosh_search_index`` so a
peer-process populator run (via the ``reload_tool_source_cache``
broadcast) is picked up on the next query.
The now-unused ``_whoosh_search_index_version`` slot is dropped.
All 21 ``test_lazy_tool.py`` unit tests still pass.
Deletes the three workarounds that existed only because the toolbox and
the populator both wrote to the index:
- ``_stamp_panel_sections_onto_index`` (~45 lines) — section metadata is
now stamped by ``build_index_entry_from_source`` at populator time
from ``DiscoveredTool.section_id`` / ``section_name``.
- ``_sync_tool_mutations_to_index`` (~32 lines) — conf-level hidden /
labels are now captured by ``DiscoveredTool`` and stamped onto the
entry at populator time (commit 4).
- ``_warn_if_index_misses_panel_tools`` (~44 lines) — the diagnostic is
obsolete now that the populator walks the same confs the toolbox
walks; coverage is structural, not best-effort. Its replacement is
``reconcile_index`` (commit 6), which prunes orphans rather than just
warning about them.
Their call sites in ``__init__`` are gone too, and the now-unused
``ToolSection`` import drops with them. ``lazy_toolbox.py`` shrinks from
1455 to 1314 lines.
All 21 unit tests in ``test_lazy_tool.py`` still pass.
Splits the CLI ``populate_store(config_file=...)`` into a thin config-
loading wrapper plus an in-process worker that takes an already-built
``GalaxyAppConfiguration`` and SQLAlchemy session:
- ``populate_store_inline(config, sa_session, *, paths=None, prune=False,
rebuild_whoosh=True, broadcast=False, ...)`` — the main entry. Full
scan replaces the index per writable store; ``paths=[...]`` switches
to partial-update mode (load existing index, add/replace entries for
the listed paths, leave others alone). ``prune=True`` overrides
partial mode back to full replacement.
- ``populate_for_paths(config, sa_session, paths, *, rebuild_whoosh=True)``
— wrapper for shed install: partial update + broadcast.
- ``reconcile_index(config, sa_session, *, rebuild_whoosh=True)`` —
wrapper for ``reset_shed_tools``: full prune + broadcast.
Boot-time auto-populate (commit 9), shed-install reroute (commit 12),
and ``reset_shed_tools`` (commit 13) consume these three entry points.
``broadcast=True`` calls ``send_reload_notification`` so peer Galaxy
processes drop their cached index after the write.
CLI behaviour is unchanged: ``main()`` → ``populate_store(config_file)``
→ ``populate_store_inline(config, sa_session)``. The placeholder
``--rebuild-index`` flag is still accepted but is now a no-op (the index
is rebuilt on every non-dry-run).
After persisting the ``ToolIndex`` for a writable store, the populator
now rebuilds the on-disk Whoosh search index for the same store via
``ToolWhooshIndex.build()``. The toolbox's search hot path becomes a
read-only consumer in a follow-up commit.
The default store's whoosh dir is ``tool_search_index_dir/_lazy_default``
(matches what ``LazyToolBox._get_search_index`` was already writing in
process); named stores get ``tool_search_index_dir/<store_name>``. The
new ``whoosh_dir_for_store(...)`` helper holds this convention so the
toolbox and the populator can't drift.
Whoosh build failures are logged but don't fail the populator run — the
toolbox surfaces them to users at query time, which is the right layer
for "search is broken, fix your deploy".
After the parallel parse, the populator now folds each (DiscoveredTool,
StoredToolSource, parsed source) triple into a fresh ``ToolIndex`` and
calls ``store.store_index() + commit()`` on every writable store.
The new ``build_index_entry_from_source(discovered, stored, tool_source)``
helper assembles a single ``ToolIndexEntry`` from a populator triple,
threading section_id / section_name / labels off ``discovered`` and
EDAM / tool_type / require_login / hidden off the parsed source. Conf-
level ``hidden="true"`` is OR'd with the parsed-source ``<tool hidden=
"true">`` body attribute so the entry honors either; this matches what
the eager pipeline's ``_load_tool_tag_set`` would have done after
``create_tool`` returned, and is what the about-to-be-deleted
``_sync_tool_mutations_to_index`` was doing in-toolbox.
``process_tool`` now returns ``(status, discovered, store_name, stored,
tool_source, err)`` so the main loop can collect every parsed source
(including incremental skips) for the post-walk index build. Skipped
tools still go into the index — only the on-disk ``StoredToolSource``
write is short-circuited.
The placeholder ``--rebuild-index`` log block is removed; index build
now happens on every non-dry-run.
12 new unit tests cover the helper: section + labels capture,
conf-level vs body-level hidden, EDAM/require_login threading, missing-
id → None, and tool_source_class taken from ``stored``. All 44
tool_source tests pass.
``DiscoveredTool`` now carries ``section_id`` / ``section_name`` and the
conf-level ``labels="a,b"`` list parsed off the ``<tool>`` element. The
populator (next commit) stamps these onto ``ToolIndexEntry`` rows so
``LazyToolBox`` can drop the post-walk syncs (``_stamp_panel_sections_onto_index``
and ``_sync_tool_mutations_to_index``) — section + labels arrive in the
index by construction.
``_iter_tool_items`` now yields ``(item, parent_section)`` pairs so the
caller can read the enclosing section's ``id`` and ``name`` for each
discovered tool. ``tool_dir`` walks inherit the section of their
declaring ``<section>`` block.
Tests cover top-level, single-section, mixed, ``tool_dir`` propagation,
hidden preservation, labels parsing, and the dataclass default-factory
guard for the new ``labels: list`` field.
The populator logic now lives at ``lib/galaxy/tool_source_store/populator.py``
and is importable as a normal module: in-process callers (cold-start
auto-populate, shed-install reroute, ``reset_shed_tools``) can use it without
``sys.path`` injection.
``scripts/tool_source/populate_store.py`` is reduced to a CLI shim that
prepends ``galaxy_root/lib`` to ``sys.path`` and dispatches to
``galaxy.tool_source_store.populator.main()``. Existing CLI surface
(``--config``, ``--full``, ``--target``, ``--watch``, ``--rebuild-index``,
etc.) is preserved.
``test_populate_store.py`` imports the four symbols it exercises
(``compute_hash``, ``iter_tool_sources``, ``send_reload_notification``,
``ToolFileWatcher``) from the new lib path; all 25 tests still pass.
Pure refactor; no behaviour change.
Pure refactor: the conf walker now lives at
``lib/galaxy/tool_source_store/discover.py`` and is importable via
``from galaxy.tool_source_store.discover import discover_tools``. No
sys.path injection needed for in-process callers (cold-start
auto-populate, shed-install reroute, test driver in upcoming commits).
``populate_store.py`` updates its two import sites; behaviour unchanged.
mvdbeek