Commit Graph
93599 Commits
Author SHA1 Message Date
mvdbeek 1b2c155bb4 Prune stale comments and fix dev doc drift
The LazyTool docstring claimed strict-by-default with a nonexistent
LAZY_TOOL_PERMISSIVE toggle (the code is permissive by default, toggled
by LAZY_TOOL_STRICT=1). The dev doc described a Redis backend, an API
layer (api/tool_sources.py, ToolAPICache), a _init_lazy_toolbox
mechanism, and an auto-enable behavior that don't exist, plus pre-move
module paths. Also drop change-narration comments referencing removed
intermediate code, an unused SqliteToolSourceStore alias, and hoist the
populator's local os/biotools imports now that the module lives inside
galaxy.tools.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-28 17:27:26 +02:00
mvdbeek 09c7eab08d Relocate tool_source_store to galaxy.tools.source_store
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-28 17:27:26 +02:00
mvdbeek ef325c6a7a LazyTool: whitelist job-setup/param-validation methods for materialise
The LAZY_TOOL_STRICT=1 lazy Integration run surfaced three execution-path
attributes that were only materialising via the permissive fallback:

  - build_dependency_shell_commands (on __SET_METADATA__, which follows
    most jobs) — the runner builds the dependency/env shell commands for
    the tool it's about to run
  - params_with_missing_data_table_entry / params_with_missing_index_file
    (data-manager / index-file tools) — param validation that walks the
    parsed parameter tree at job time

All three genuinely need a parsed Tool, so add them to _MATERIALIZE_OK.
Under permissive mode this was a silent WARN + materialise; under strict
it raised NotImplementedError and failed every job-running integration
test. This is exactly the audit strict is meant to force — the fix makes
the materialisation explicit rather than accidental.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-28 17:27:26 +02:00
mvdbeek 29f75d1a3b test: assert batch endpoints don't materialise tools (budget guard)
Add a materialise counter to LazyToolBox (_lazy_materialize_count, bumped
at the single _create_tool_from_stored_source chokepoint) and an
integration test that hits every batch reader the client uses — flat
listing, panel walk, panel views, tests-summary, all_requirements, and
search — asserting a zero materialise delta across them.

This catches accidental whole-toolbox materialisation that LAZY_TOOL_STRICT
can't: a tool-filter or _MATERIALIZE_OK attribute read inside a full-index
sweep parses tools without ever hitting the strict __getattr__ path. The
test was written against the search-materialisation regression fixed in
the previous commit and fails without it.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-28 17:27:26 +02:00
mvdbeek 3017f46a45 LazyToolBox: resolve search hits without materialising
ToolsService.search_tools resolved every whoosh hit through
_get_tool -> toolbox.get_tool, which in lazy mode loads the tool from the
store on demand. A single /api/tools?q= query therefore parsed one Tool
per hit. The populator-owned whoosh index also carries ids that were
never loaded into this toolbox (tool_conf.xml.sample alone lists ~150
legacy tools whose files aren't present), so lazy search both parsed and
returned tools the eager path skips: eager whoosh only holds loaded
tools, and eager _get_tool returns None for un-loaded ids.

Add LazyToolBox.resolve_search_hit, which looks the hit up in the
registered stubs (_tools_by_id, plus the shed short-id map) and returns
None for anything not part of this toolbox — no parse. search_tools uses
it in lazy mode and keeps the eager _get_tool path unchanged. Result:
search stays O(hits) index reads and matches eager scoping.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-28 17:27:26 +02:00
mvdbeek ba1a3b9f69 LazyTool: answer get_panel_section off the entry (no materialise)
AgentTools.get_tool_categories iterates toolbox.tools() and reads
tool.get_panel_section()[1] on every visible tool. get_panel_section was
not on the LazyTool stub surface, so under the permissive default that
sweep materialised the entire toolbox to build the category list (and
under LAZY_TOOL_STRICT=1 it would now raise).

The populator already stamps panel_section_id / panel_section_name onto
ToolIndexEntry, so forward the call off the entry — matching the
(section_id, section_name) / (None, None) contract of
Tool.get_panel_section — and keep the walk lazy.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-28 17:27:26 +02:00
mvdbeek bcd8690a2f CI: run lazy api/integration jobs with LAZY_TOOL_STRICT=1
The lazy dispatch jobs set use_lazy_toolbox=true but left LazyTool in its
permissive default, where an off-surface attribute read materialises the
tool with only a WARN log. An accidental whole-toolbox sweep (a new
caller iterating toolbox.tools() and touching an un-forwarded attr) would
therefore degrade lazy mode to O(N) tool parses without failing CI.

Set LAZY_TOOL_STRICT=1 in the lazy dispatch env so those reads raise
NotImplementedError instead, turning any such regression into a hard
failure on the lazy run. Legitimate materialisation (to_dict, the
execution path) goes through _materialize() directly and is unaffected.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-28 17:27:26 +02:00
mvdbeek 9a12b3aaed remove_tool_by_id: purge every spelling the tool resolves under
test_repository_uninstall kept resurrecting the tool after all
index-level fixes. Local repro + probes showed removal completed on the
right toolbox but two in-memory paths still served the tool:

- The materialise LRU is keyed by whatever id the caller resolved with,
  so the same tool sits under both its guid and its short id; the
  guid-prefix purge left the short-id entry behind. Purge by cached
  tool identity instead.
- _tools_by_old_id is keyed by old_id — the SHORT id for a shed guid —
  and super().remove_tool_by_id deletes from it by object identity
  only, which misses when the bucket holds an earlier registration
  (stub or materialised instance) than _tools_by_id. The leftover
  resurfaced through the eager get_tool fall-through. Scrub every
  object belonging to the removed guid; sibling installs sharing the
  short id survive.

Verified locally: the full TestRepositoryInstallIntegrationTestCase
passes 3/3 consecutive lazy runs (install, uninstall, update).
2026-07-28 17:27:25 +02:00
mvdbeek 39f86185d9 serialize index-cache invalidation against tool removal
remove_tool_by_id runs under app._toolbox_lock, but the invalidation
path (queue-worker reload_tool_source_cache handler — and every
populate broadcasts one) reloaded the index and re-registered stubs
without it. Interleaved with an uninstall, that reloads the pre-removal
index and re-registers the just-removed tool. Take the same RLock.
2026-07-28 17:27:25 +02:00
mvdbeek 2340b9f33c lazy shed install: snapshot panel elems before the conf persist empties them
When the target <section> already exists in the on-disk shed_tool_conf,
add_to_shed_tool_config merges the incoming children into it via lxml
append — which reparents them, leaving the section elements in
elem_list empty. The lazy branch persists the conf BEFORE loading (the
populate needs it on disk), so both the partial populate (no paths →
no enriched entries) and load_item (empty section → tool never enters
the in-memory panel; only the async conf-watcher reload rescued it) ran
against gutted elements. First install into a fresh section was immune,
which is why only the second fastp changeset lost tool_shed_repository
(test_only_latest_version_in_panel_fastp — now reproduced and verified
locally in both lazy and eager modes).

Deep-copy the elements before persisting and drive the populate +
load_item from the snapshot. The eager branch loads before persisting
and stays untouched.
2026-07-28 17:27:25 +02:00
mvdbeek 1b333b7583 shed install populate: resolve new tool paths the way discovery does
The partial populate filters discovered tools on exact path strings.
_collect_new_tool_paths joined the shed conf dict's raw tool_path —
relative for typical shed_tool_conf files — while discovery resolves a
relative tool_path against the conf file's directory and returns an
absolute path. The strings never matched, so the enriched conf-driven
discovery was filtered out and the ad-hoc synthesis populated a
metadata-poor entry instead: the panel served fastp without
tool_shed_repository (test_only_latest_version_in_panel_fastp), and the
store grew a duplicate row under the unresolved path string.

Route both through discover.resolve_tool_path (now public) and thread
each tool's guid so even a genuinely-unreachable path yields a
guid-keyed entry.
2026-07-28 17:27:25 +02:00
mvdbeek 6f0f950ed5 update_index_entry: write through to the per-version map
The data_manager_id stamp landed in index.entries only —
update_index_entry never touched entries_by_version, and the two maps
serialize independently. Job-time materialise passes a tool_version, so
it resolves through the per-version map and got the unstamped entry
back after the next index reload (populate broadcasts one constantly):
exec_after_process still failed with 'Invalid data manager (<guid>)'.

Both backends now route updates through ToolIndex.add_entry, which
keeps the maps in step (and stops an older-version update from
clobbering the newer default entry). create_tool additionally stamps
the in-memory per-version twin, which is a distinct object after a
from_dict reload.
2026-07-28 17:27:25 +02:00
mvdbeek 3aaecbe65d index reload: refresh the entries behind already-registered stubs
_register_new_index_entries_as_stubs skipped every tool already in
_tools_by_id, so a stub minted from an install-time ad-hoc entry kept
serving that metadata-poor entry after the conf-driven populate
enriched the index — the fastp panel payload was missing
tool_shed_repository (test_only_latest_version_in_panel_fastp).

Swap the entry behind the existing stub instead: the panel and
registries hold the stub object, and _overrides survive.
2026-07-28 17:27:25 +02:00
mvdbeek cae636885d create_tool: stamp the data manager conf id the caller already hands us
Shed-installed data managers load their tool at install time, before
shed_data_manager_conf.xml exists — the self-healed entry carries
data_manager_id=None, materialise never restores the conf id, and
DataManagerTool.exec_after_process fails with 'Invalid data manager
(<guid>)' at job finish (all six data-manager integration tests).

DataManager._load_tool already passes data_manager_id through
load_hidden_tool; create_tool now stamps it onto the resolved entry and
persists it, so job handlers materialising from the shared index
resolve the registry correctly.
2026-07-28 17:27:25 +02:00
mvdbeek 0cf910ddac persist tool removal in the index — uninstalls stayed in-memory only
remove_tool_by_id popped the entry from every in-memory map, but the
persisted singleton index still carried it. Every populate broadcasts a
reload_tool_source_cache invalidation, so the very next reload handed
the entry back and the uninstalled tool resolved again
(test_repository_uninstall expected err_msg, got the full tool).

ToolSourceStore grows remove_index_entry — the uninstall counterpart of
update_index_entry — and remove_tool_by_id writes the removal through.
2026-07-28 17:27:24 +02:00
mvdbeek efd598802e materialise data manager tools with their conf id
DataManagerTool resolves the registry by the <data_manager id> conf id,
which may differ from the tool XML id
(test_data_manager_async_submission_with_mismatched_conf_id). Eager
threads it through load_hidden_tool(data_manager_id=...); lazy
materialisation built the tool from the stored source alone, so
DataManagerTool.__init__ fell back to the tool id and
exec_after_process failed with 'Invalid data manager requested'.

Discovery now records the conf id, the index entry carries it, and
_create_tool_from_stored_source restores it at materialise time. Also
stamp is_local from the discovered guid — it defaulted True, which kept
the ToolConfRepository branch of the shed materialise path dead even
with repository metadata present.
2026-07-28 17:27:24 +02:00
mvdbeek f2405eeed1 data manager stubs: match eager hidden and admin-gate semantics
Discovery stamped hidden=True on data manager tools, but eager's
load_hidden_tool only means 'not in the panel' — Tool.hidden stays
falsy, and the flat /api/tools?in_panel=false listing filters hidden
tools, so the lazy listing dropped them
(test_data_manager_async_submission_with_mismatched_conf_id).

Also gate LazyTool.allow_user_access on tool_type 'manage_data' — the
value DataManagerTool actually carries; 'data_manager' never occurs, so
the admin gate on stubs never fired and lazy was more permissive than
eager.
2026-07-28 17:27:24 +02:00
mvdbeek fe5528dfb9 LazyToolboxSearch: search unlimited like the eager panel search
The lazy path capped whoosh results at tool_search_limit (default 20)
while eager ToolPanelViewSearch passes limit=None. Uniform-score matches
truncate in doc-insertion order, so a tag query fanning out to 23 tools
silently lost the last three (test_search_curated_tool_tags).
2026-07-28 17:27:24 +02:00
mvdbeek 428248677e LazyToolBox.create_tool: self-heal an index miss for on-disk files
Shed installs never survived lazy mode: metadata generation
(installed_repository_metadata_manager.get_repository_tools_tups) loads
each freshly cloned tool before add_to_tool_panel persists the conf and
populates the index, so create_tool raised on every install and the
repository landed broken (data managers, workflow test tools, fastp
panel tests).

A miss for a file that exists on disk now populates that single path —
still through the single-writer populator; populate_store_inline
synthesizes a DiscoveredTool for requested paths outside every conf and
threads the caller's guid so the ad-hoc entry is keyed like its eventual
conf-driven replacement — then reloads the index and retries. Misses
for files that don't exist still raise.
2026-07-28 17:27:24 +02:00
mvdbeek a4d24ecad3 tool_source_store: keep one row per (hash, source_path)
Six test/functional tools (upload.xml, export_remote.xml,
ucsc_tablebrowser.xml, parse_values_from_file.xml, catWrapper.xml,
for_tours/filtering.xml) expand to byte-identical content as their
root tools/ counterparts. With hash unique, whichever file populates
first owns the row; the twin's populate is skipped and
get_by_source_path(second path) returns nothing, so LazyToolBox.create_tool
raises and the toolbox silently drops the tool — upload breaks for any
instance whose config discovers both trees (CI integration shards do).

hash becomes a non-unique index (migration amended in place — unreleased),
store() upserts by source_path, the populator skips only when the same
path already holds the same content, and get/exists/delete tolerate
multiple rows per hash.
2026-07-28 17:27:24 +02:00
mvdbeek e16502d77a index carries the panel-entry contract; LazyTool serves it without materialising
ToolIndexEntry gains icon, xrefs, model_class, form_style and
is_workflow_compatible, derived at populate time the same way
Tool.__init__/to_dict derive them (tool_type class registry, root
workflow_compatible attribute, page count). EDAM fields now go through
expand_ontology_data, picking up the curated mapping overrides and
legacy bio.tools xrefs the direct parse_edam_* calls missed.

LazyTool.to_panel_entry emits the full client payload -- including
tool_shed_repository (the field the panel-view integration tests
assert), versions from the registered lineage, uuid, and the
admin-gated config_file from source_path.
2026-07-28 17:27:23 +02:00
mvdbeek 7af0f056cd LazyToolBox: reload must read the index the inline populator just wrote
The populator writes through its own store instances, so after the
inline backfill the toolbox's store still served its cached pre-populate
index. Single-instance that cache is merely stale-but-identical; on a
shared database (the CI integration shards share one postgres DB, and
any multi-server deployment shares one in production) the cached index
is another instance's view and the eager walk drops every tool missing
from it — the post-reload 'no index entry for upload.xml' cascade.

Reload integration tests drive galaxy.queue_worker.reload_toolbox
directly and reproduce the failure with a foreign index row; the earlier
send_control_task-based attempts passed vacuously because the control
message never executes in this harness.
2026-07-28 17:27:23 +02:00
mvdbeek 2a1c8c503d lazy search + data managers: whoosh tool_tags parity, planemo path fallback, _view materialise
- The lazy whoosh schema lacked the tool_tags field the eager schema
  indexes, so a fielded phrase query (tool_tags:"send data") fanned out
  to positionless NGRAMWORDS fields and raised QueryError -> 500. Index
  tool_tags per entry (curated mapping over the entry ids) and search it.
- Data manager tool paths resolve with the same relative-to-conf
  fallback DataManagers.load_from_xml applies for planemo layouts.
- LazyTool consults _MATERIALIZE_OK before the underscore guard and adds
  _view, so admin dependency-management endpoints materialise instead of
  AttributeError.
2026-07-28 17:27:23 +02:00
mvdbeek 4166b53de9 packages: ship galaxy.tool_source_store in galaxy-app; search reads config directly
The packages/app tree links each galaxy subpackage explicitly and
tool_source_store was never added, so the built wheel lacked the package
and per-package mypy resolved its imports as Any (surfacing as
'Returning Any' in LazyToolboxSearch.search).
2026-07-28 17:27:23 +02:00
mvdbeek b3c75062e5 LazyToolBox: answer tool_tags from the curated mapping; index data manager tools
curated_tool_tags sweeps every tool for .tool_tags, which force-
materialised the whole toolbox on first /api/tags/tool_tags request and
500d whenever one indexed tool can't build (filter_data_table's
column="value" dynamic options). The tag mapping is keyed purely by
tool id, so the stub now answers without materialising.

Data manager tools are loaded post-boot via DataManagers ->
load_hidden_tool, which the discover walk missed — create_tool's index
miss aborted DataManager loading and the tools 404d under lazy mode.
discover_tools now walks data_manager_conf/shed_data_manager_conf
(path + guid resolution mirroring DataManager._load_from_element), and
the seam's source-path lookups normalize to absolute paths since data
manager tool files are joined from a possibly relative tool_path.
2026-07-28 17:27:23 +02:00
mvdbeek caea1912b9 Use plain config/attribute access; unify tool-file detection with tool_util
GalaxyAppConfiguration materializes every schema option, so the
hasattr/getattr guards around tool_configs, shed_tool_config_file,
migrated_tools_config, root, tool_search_index_dir and use_lazy_toolbox
were dead (StandaloneInstallationTarget's Config stub gains the
use_lazy_toolbox attribute the real config carries). The
tool_source_url getattr had no matching schema option — the default
sqlalchemy store is now disk-path only; full URLs remain available via
named tool_source_stores entries.

getattr(entry, "source_path") was hiding a real gap: ToolIndexEntry
never carried the field, so LazyTool.config_file always materialised.
The populator now stamps source_path onto every entry.

discover.py's duplicated _looks_like_a_tool is replaced by the real
galaxy.tool_util.loader_directory.looks_like_a_tool (threading
enable_beta_tool_formats), and the works-without-galaxy fallback for
MODEL_TOOLS_PATH is gone — the populator always runs with galaxy
importable.
2026-07-28 17:27:23 +02:00
mvdbeek 5bbdef365a style: apply isort/black (tox -e format) 2026-07-28 17:27:23 +02:00
mvdbeek 6bf468273d DatabaseToolSourceStore: own tool_source_record table, stop sharing tool_source
The job-request path creates and reads tool_source rows whose source
column carries the raw source string; the store was writing dict-shaped
payloads into the same table, and once identity hashes aligned the job
path could resolve a store row and fail pydantic validation (500 on
every request-style tool execution under lazy mode).

Give the store its own content-addressed table with real columns —
get_by_tool_id / get_by_source_path become indexed queries instead of
full-table JSON scans, populator pruning can never delete rows the job
path references, and the identity_hash coupling (and the legacy
__tool_index__ sentinel-row fallback) disappear.
2026-07-28 17:27:22 +02:00
mvdbeek 3eef867da8 mypy: fix type errors flagged by lint + packages jobs
- AbstractToolBox gains a no-op invalidate_index_cache (LazyToolBox
  overrides it); shed installs no longer depend on the subclass type.
- Annotate LazyToolboxSearch.build_index, ToolFileWatcher.observer,
  populator session cast, UUID key coercion in _register_lazy_entry,
  and drop stale type: ignore comments.
2026-07-28 17:27:22 +02:00
mvdbeek fb8b0b8f9e tool_panel_manager: keep upstream install ordering on the eager path
The lazy path persists shed_tool_conf before load_item because the
populator's conf walk needs it on disk; that same reorder broke eager
version-replacement in the panel (test_add_twice). Split the paths:
eager is now byte-identical to upstream, the reorder applies only under
use_lazy_toolbox.
2026-07-28 17:27:22 +02:00
mvdbeek ab80d977a8 test_lazy_tool: replace deprecated datetime.utcnow 2026-07-28 17:27:22 +02:00
mvdbeek fa35f80369 DatabaseToolSourceStore: populate identity_hash + scalar tool columns (new upstream schema) 2026-07-28 17:27:22 +02:00
mvdbeek 3b9bc8823f style: modernize Optional/Union annotations to PEP 604 (ruff UP045/UP007) 2026-07-28 17:27:22 +02:00
mvdbeek 34b9f0d328 tools service: restore FilterFactory + per-hit auth parity for lazy listings and search 2026-07-28 17:27:21 +02:00
mvdbeek eba90fff5a tool_panel_manager: gate shed-install populator behind use_lazy_toolbox 2026-07-28 17:27:21 +02:00
mvdbeek a9e298027e populator: make watchdog import local to watch mode 2026-07-28 17:27:21 +02:00
mvdbeek b863a32340 lazy_toolbox: fix flake8 blank-line violations 2026-07-28 17:27:21 +02:00
mvdbeek d9c5fc86ce migration: re-parent tool_index onto 28885b317f78 after dev rebase 2026-07-28 17:27:21 +02:00
mvdbeek 3d1f2eefb2 toolbox: split panel-view shape from show-endpoint to_dict
The lazy ``to_dict`` fast path was short-circuiting *both* the
panel-view walk (intended) *and* the ``/api/tools/{id}`` show endpoint
(unintended) — they're indistinguishable at the ``to_dict(io_details=
False, link_details=False)`` callsite. ``test_legacy_biotools_xref_
injection`` caught it: the show endpoint returned the entry-shape
dict and no longer carried ``xrefs``.

Split the two call sites at the API rather than the parameter:

* ``Tool.to_panel_entry(trans)`` — new method, returns the cheap
  entry-shape dict (id/name/version/description/labels/edam/panel
  section/link). ``LazyTool`` overrides it to skip materialise.
* ``AbstractToolBox.get_tool_to_dict`` calls ``to_panel_entry`` when
  ``tool_help=False`` (the panel-view default); ``tool_help=True``
  still falls through to ``to_dict`` so the help payload renders.
* ``LazyTool.to_dict`` always materialises now — it's the
  ``/api/tools/{id}`` contract and must ship ``xrefs`` / ``versions``
  / ``is_workflow_compatible`` / ``tool_shed_repository`` / etc. The
  materialise-failure fallback now reuses ``to_panel_entry``.

Net effect: ``/api/tool_panels/{view}`` and ``/api/tools?in_panel=
true`` stay materialise-free for the lazy toolbox; ``/api/tools/{id}``
serves the full show-endpoint dict in both modes.
2026-07-28 17:27:21 +02:00
mvdbeek c9dcdafde9 DatabaseToolSourceStore: isolate reads in a private Session
`LazyTool` materialise during workflow ``inject_all`` calls
``store.get(hash)``; the store ran the SELECT on Galaxy's shared
scoped session and then called ``session.rollback()`` to release the
implicit read transaction. That rollback expired *every*
request-scoped instance attached to the shared session — including the
in-flight ``workflow.steps`` collection. The next access in the same
``populate_module_and_state`` re-fetched a fresh ``WorkflowStep`` list
from the DB, with ``.module`` unset on the new instances, so
``compute_runtime_state``'s ``assert step.module`` raised
``AttributeError: 'WorkflowStep' object has no attribute 'module'``.
Every workflow-invocation test under ``use_lazy_toolbox=true`` 500'd.

Fix: read methods on ``DatabaseToolSourceStore`` (``get``, ``exists``,
``get_by_tool_id``, ``get_by_source_path``, ``count``, ``list_all``,
``load_index``) now open a private ``Session`` bound to the same
engine and close it on exit. The shared scoped session — and its
caller's transaction state — is untouched. The original lock-release
intent (cold-start populator, queue-worker init) still holds because
each private session's read transaction ends when the session closes.

Writes (``store``, ``store_index``, ``delete``, ``update_index_entry``)
still use ``_get_session()`` so they participate in the caller's
transaction and commit in the right context (populator, queue worker,
``LazyToolBox.__init__``).
2026-07-28 17:27:21 +02:00
mvdbeek 8e8f587209 LazyTool: restore entry-only to_dict fast path (no panel materialise)
With ``link_details`` no longer gating any field, the toolbox-internal
panel / listing callers stopped passing it (``f64858bf55e``).
``LazyTool.to_dict`` can now return the entry-shape dict for the
default-call case — every panel-view tool serves from the index, zero
materialise.

Materialise still fires when the caller asks for the parsed-Tool
payload: ``io_details=True`` (the ``/api/tools/{id}`` show endpoint
contract) or ``tool_help=True`` (rendered help). Failure-to-materialise
still falls back to the entry-shape dict so the show endpoint never
returns 500 when a specific tool's XML chokes the parameter factory.

The ``link`` field is derived from ``self.id`` matching what
``Tool.to_dict`` now emits unconditionally — ``/tool_runner?tool_id={id}``
for the standard tool runner.

Tests updated:
- ``test_to_dict_entry_fast_path_does_not_materialise`` asserts the
  default call serves the entry dict and never invokes the callback.
- ``test_to_dict_io_details_materialises`` covers the show contract.
- ``test_to_dict_falls_back_to_entry_when_materialise_fails`` covers the
  parameter-factory-failure case.

23/23 unit tests pass.
2026-07-28 17:27:20 +02:00
mvdbeek 12615a4c45 Tool.to_dict: drop dead min_width / target / uihints fields
These were emitted on every entry of /api/tools but no client component
ever read them. The Vue panel renderer derives the link target from
``model_class == "DataSourceTool"`` via ``getTargetById`` and never reads
``min_width`` at all; the Tool interface declared the fields purely to
match the wire shape. ``Tool.target`` / ``Tool.uihints`` likewise had
``to_dict`` as their sole reader in lib/.

Removed:
- ``self.target`` / ``self.uihints`` attributes and the corresponding
  ``<inputs target>`` and ``<uihints minwidth>`` reads in ``__parse_legacy_features``
- ``DataSourceTool.parse_inputs`` self.target = "_top" override (dead)
- ``min_width`` / ``target`` keys from both the eager ``to_dict`` payload
  and the ``LazyTool.to_dict`` stub

The XSD declarations for ``<uihints minwidth>`` and ``<inputs target>``
stay so existing data-source tool XML continues to validate cleanly.
2026-07-28 17:27:20 +02:00
mvdbeek 84b9c2dbea LazyTool: flip default to permissive — strict via LAZY_TOOL_STRICT=1
After five rounds of CI surfacing missing stub attrs and the latest
round still hitting indirect failures (workflow steps with ``module``
unset because some inner attribute access raised under strict), the
explicit ``_MATERIALIZE_OK`` set has stopped converging. The strict
guard caught the obvious surface and now slows ship velocity without
adding signal.

Flip the default: unknown attr reads on ``LazyTool`` now log a WARNING
and materialise. The strict raise is still available for debugging via
``LAZY_TOOL_STRICT=1`` — the path that surfaces ``add to the stub
surface or _MATERIALIZE_OK`` for new audit work.

``test_strict_getattr_raises_with_clear_message`` now monkey-patches
the module flag to force strict mode for the assertion. Other tests are
unchanged.
2026-07-28 17:27:20 +02:00
mvdbeek 8308bdbf5b LazyTool.to_dict: always materialise; fast path was incomplete
The entry-only fast path was missing fields the ``/api/tools/<id>``
show endpoint serves (``xrefs``, ``inputs`` when ``io_details=False``,
…). The flat ``/api/tools?in_panel=False`` listing — the only caller
that benefits from the fast path — already bypasses this method:
``services/tools.py:list_tools`` walks ``tool_index.list_all()`` and
calls ``entry.to_api_dict()`` directly.

Always materialise here; the entry-only dict survives as the fallback
when materialise itself raises (a tool whose XML the parameter
factory chokes on — ``upload_dataset``, ``column="value"``, …) so the
caller still gets *something* renderable rather than a 500.

Tests updated:
- ``test_to_dict_fast_path_does_not_materialise`` → renamed to
  ``test_to_dict_falls_back_to_entry_dict_when_materialise_fails``.
- ``test_to_dict_link_details_materialises_exactly_once`` → renamed
  for accuracy; behaviour unchanged.
2026-07-28 17:27:20 +02:00
mvdbeek b7eb650188 LazyTool.to_dict: materialise when io_details=True too
``/api/tools/<id>`` show endpoint passes ``io_details=True`` to surface
``inputs`` / ``outputs`` / parameter detail; the previous condition
only materialised on ``link_details=True``, so the listing-shape dict
came back without inputs and ``test_show_repeat`` / ``test_show_multi_data``
asserted on a missing key.

Add ``io_details`` to the materialise predicate. Forward both flags into
``Tool.to_dict`` so the parent picks the right payload variant. The
entry-only fast path now only applies to the truly cheap case (the
``/api/tools?in_panel=False`` listing).

Materialise failure still falls back to the entry-only dict — listing
remains 200 even if a specific tool can't materialise.
2026-07-28 17:27:20 +02:00
mvdbeek 985da318b1 LazyTool: round-5 _MATERIALIZE_OK additions
Seven more execution + introspection-path attrs from the round-5 CI:

- ``outputs`` — output spec read by ``execute`` / display formatters.
- ``tests`` — tool tests definitions surfaced via ``/api/tools/.../tests``.
- ``to_json`` — tool JSON serialisation.
- ``tool_requirements_status`` — ``/api/tools/.../requirements_status``.
- ``provided_metadata_file`` — job-runner metadata path resolution.
- ``test_data_path`` — tool test data lookup.
- ``get_configured_job_handler`` — job-runner handler routing.

Cascading WorkflowStep ``module`` injection errors in the same log
were a downstream effect: workflow steps couldn't resolve their tool
modules because the underlying ``LazyTool`` raised on these reads.
Materialising lets the injector populate ``step.module`` normally.
2026-07-28 17:27:20 +02:00
mvdbeek b71b2847af test_lazy_tool: pick a non-MATERIALIZE_OK attr for strict-raise test
``tool_action`` graduated to ``_MATERIALIZE_OK`` in the previous commit
(it's read on the execute path). The strict-raise assertion needs an
attribute the stub neither forwards nor materialises;
``totally_not_a_tool_attr`` is guaranteed unaccounted-for.
2026-07-28 17:27:20 +02:00
mvdbeek 39c5552f2b LazyTool: round-4 _MATERIALIZE_OK additions
Eight more execution-path attrs surfaced by API tests:

- ``execute`` — the tool execute method itself.
- ``expand_incoming`` — parameter expansion for multi-run / map-over.
- ``params_to_strings`` — parameter serialisation.
- ``tool_action`` — execution action dispatch.
- ``tool_dir`` — on-disk directory accessor (read by some callers).
- ``completed_jobs`` — job-search-by-tool API.
- ``get_default_history_by_trans`` — default-history lookup during execute.
- ``regenerate_imported_metadata_if_needed`` — import/export job path.

All execution-path; correct to materialise on first read.
2026-07-28 17:27:19 +02:00
mvdbeek 09f32c2a15 LazyTool: more execution-path attrs in _MATERIALIZE_OK
Round-2 CI surfaced five more attributes the tool-execution path
hits before the parameter machinery has finished setting up:

- ``check_and_update_param_values`` — parameter validation called from
  ``handle_input``.
- ``wants_params_cleaned`` — parameter scrub before execution.
- ``tool_source`` — raw ``ToolSource`` (some callers walk the XML
  directly).
- ``dynamic_tool`` — dynamic-tool linkage on execution.
- ``produces_entry_points`` — interactive-tool entry-point lookup.

All five materialise correctly when read; adding to ``_MATERIALIZE_OK``
makes the stub's strict ``__getattr__`` route through the cached
``_real`` Tool on first access instead of raising.
2026-07-28 17:27:19 +02:00
mvdbeek ab8090765a LazyTool: add tool-execution path attrs to _MATERIALIZE_OK
CI surfaced six attributes that the tool-execution path
(``/api/tools/{id}`` POST → ``handle_input``) and the job runner read
on the resulting Tool, all of which fundamentally need a parsed
parameter tree:

- ``handle_input`` — entry point for tool execution.
- ``inputs``, ``parameters``, ``new_state`` — the parameter machinery.
- ``input_translator`` — runtime parameter translation.
- ``requires_galaxy_python_environment`` — job-runner environment hint.

Each was raising ``NotImplementedError`` for every tool-execution
request, taking out ~hundreds of API tests at once. Add them to
``_MATERIALIZE_OK`` so the stub materialises on first read and the
real Tool answers the rest of the call.

Materialise on tool execution is exactly the cost model the lazy
toolbox is designed for — pay parse cost only when the tool is
actually used. The stub still surfaces unaccounted-for attribute reads
loudly, which is the contract the user picked.
2026-07-28 17:27:19 +02:00