Three review fixes to the populator:
- Reload broadcast: send_reload_notification does a bare kombu exchange
publish that never declares the active-process queues, so on the kombu
sqlalchemy transport (a common default) the shed-install broadcast never
reaches peers — their cached index stays stale until restart. Add
_broadcast_reload, which routes through send_control_task (correct
declare_queues, derived from the worker_process table) when an app is in
hand; the app-less CLI keeps the best-effort raw publish. Thread app
through populate_store_inline / populate_for_paths / reconcile_index and
the shed-install, ad-hoc self-heal, and reset_shed_tools call sites.
- Macro watch: a changed macros file left its importing tools with stale
expanded content because the watcher filtered "macro" out by filename.
Accept every .xml and, on a <macros> root, re-expand the tool siblings
in its directory.
- Typing: annotate build_index_entry_from_source (ToolSource / stored /
discovered) and drop the redundant hasattr guards the ABC already
guarantees (parse_uuid stays guarded — it is XML-only).
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
ToolPanelViewSearch (eager) and ToolWhooshIndex (store) built the same
field set + boosts twice, inviting drift. Extract build_search_schema in
source_store.search as the single definition; the eager side passes
help_boost to add its help field (the populator has no rendered help text,
so the store schema omits it and help-text queries don't match in lazy
mode). Field boosts are unchanged, so ranking is identical.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Neither is on the production read path. Lazy search goes through the
populator-built whoosh index (LazyToolboxSearch -> ToolWhooshIndex), and
/api/sanitize_allow builds its response from config.sanitize_allowlist
against the toolbox. The two ToolIndex methods were called only from
benchmarks.py and unit tests, and ToolIndex.search's token ranking
diverged from whoosh's BM25F — a correctness trap. Remove both (whoosh
search stays covered by test_multi_store_search.py); keep the reusable
per-entry ToolIndexEntry.to_sanitize_entry projection.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The only local import left is lazy_toolbox's send_control_task, which
is a genuine cycle: galaxy.queue_worker imports LazyToolBox at module
level. Everything else - app boot, queue_worker, tool_panel_manager,
services/tools, search, populator, benchmarks, uses_shed and the test
modules - now imports at the top; the claimed app/tools circularity
never existed (galaxy.app already imports galaxy.tools).
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
walk_tool_directories() in tool_util.toolbox.base is now the single
implementation of the hidden/private-skipping directory walk;
AbstractToolBox.__watch_directory consumes it (keeping per-directory
watcher registration) and discover drops its parallel copy.
Two deliberate behaviour touch-ups on the eager side: entries are now
walked in sorted order (deterministic panel order for tool_dir tags),
and a directory's watcher now registers when any tool in it loaded
rather than only when the last candidate did.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
AbstractToolBox.__resolve_tool_path becomes the module-level
resolve_tool_path() in tool_util.toolbox.base and discover uses it,
dropping its own copy. This also fixes discover's fallback for confs
without a tool_path attribute: it guessed <root>/tools, while the
toolbox has always used config.tool_path; discover_tools_from_config
now takes that default (discover_tools passes config.tool_path).
Also hoists the MODEL_TOOLS_PATH import to module level - the parent
galaxy.tools package is imported with the subpackage anyway, so the
local import bought nothing.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Adopt #23067's store infrastructure wholesale where it is canonical, keeping
only genuine lazy-toolbox additions as deltas on top:
- Store package: adopt #23067's factory.py/interface.py split and facade
__init__.py, and its URL-only SqlAlchemyToolSourceStore. Re-apply lazy-only
deltas — ToolIndexEntry panel-contract fields (icon/xrefs/model_class/
form_style/is_workflow_compatible/source_path) and data_manager_id;
composite per-version index merge; scored multi-store whoosh search
(search_scored + tool_tags field); populator panel-contract derivation via
expand_ontology_data + biotools and data_manager_id stamping; data-manager /
converter discovery in discover.py; benchmarks.py.
- Config: replace tool_source_store + tool_source_disk_path with the single
SQLAlchemy URI tool_source_database_connection (defaulted in config/__init__.py,
validated via try_parsing, schema attr added), adopt #23067's tool_source_stores
wording, and keep the branch-only use_lazy_toolbox / lazy_toolbox_cache_size
options. Regenerated galaxy.yml.sample, galaxy_options.rst, and the schema-type
stub. galaxy_mock uses tool_source_database_connection.
- Docs: adopt #23067's tool_source_storage.rst (admin + dev) as the base and
re-add the lazy sections (LazyToolBox, batch-endpoint integration,
materialisation-count guard, LazyToolboxSearch multi-store search, benchmarks).
- Tests: adopt #23067's store + scripts unit tests; re-add the ours-only
composite entries_by_version merge test, data-manager discovery /
build-index tests, and multi-store search test, all on the URI config.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The store is a rebuildable, content-addressed cache — it does not need
Galaxy's database, and living there was the source of every subtle
lifecycle mechanism in the subsystem: store.commit() committed whatever
request or job transaction was in flight, reads needed a private-Session
isolation hack, the LazyToolBox constructor carried a post-init commit to
avoid queue-worker idle-in-transaction locks, shutdown needed a haltable
ordered before engine disposal, and a sa_session parameter threaded
through the factory, populator, watcher and CLI.
Standalone stores are now the only kind: the default is a SQLite file
under tool_source_disk_path (any SQLAlchemy URL via tool_source_stores
for shared multi-host deployments), committing per write on its own
engine. The tool_source_record/tool_index tables, their migration, the
model classes and the whole session plumbing are gone, and the store is
only initialized when use_lazy_toolbox is enabled — default deployments
no longer touch any of this at boot. The ToolSourceStore ABC loses
commit() (every backend persists per write). The dev/watch-mode watcher
now populates per file without broadcasting and keeps its single
debounced reload notification for the batch.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The populator writes one whoosh index per store, but search only opened
the default store's directory — tools served from a named store
(per-conf store="...") listed and ran fine yet never appeared in
/api/tools?q=. Search each configured store's index and merge hits by
BM25 score (ToolWhooshIndex.search_scored); scores from different
indexes aren't strictly comparable, but interleaving beats invisible
tools, and resolve_search_hit drops ids foreign to this toolbox.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The merge loop copied only entries and by_section, leaving the merged
index's per-version map empty — ToolIndex.get(tool_id, tool_version)
resolves exact versions through entries_by_version, so every non-newest
version of a member-store tool was unreachable (wrong default served,
get_all_versions empty). Merge per (id, version) with the same
earlier-member-wins rule.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Two gaps found by review: (1) invalidate_index_cache was add/update
only — it registered new index entries as stubs and refreshed existing
ones, but never dropped registrations whose ids vanished from the
reloaded index, so a peer-process uninstall left this process serving
the stale stub via the eager get_tool fall-through; (2) remove_tool_by_id
persisted the index removal locally but never broadcast, so peers had no
reason to reload until an unrelated populate ran.
Extract the in-memory removal bookkeeping into _remove_tool_in_memory,
diff the previous index against the reloaded one in
invalidate_index_cache and pop vanished ids (diff-based, so internal
and dynamic tools that never enter the index are untouched), and send
reload_tool_source_cache (noop_self) after persisting a removal.
_register_lazy_entry now registers the old-id bucket even when
old_id == tool_id, matching the eager __add_tool — super's removal
unconditionally pops that bucket.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Commit 4c16f4d819a (drop _init_lazy_toolbox) removed the
LazyLineageMap assignment together with the old init path, silently
downgrading the toolbox to the eager LineageMap. Boot-time lineages
still came out right (the eager walk's register() accumulates versions
by versionless id), but every post-boot lookup — peer installs surfaced
by invalidate_index_cache, reloads — fell back to building a
single-version lineage from one Tool object, hiding the other indexed
versions from get_safe_version and panel version dedup.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The LazyTool docstring claimed strict-by-default with a nonexistent
LAZY_TOOL_PERMISSIVE toggle (the code is permissive by default, toggled
by LAZY_TOOL_STRICT=1). The dev doc described a Redis backend, an API
layer (api/tool_sources.py, ToolAPICache), a _init_lazy_toolbox
mechanism, and an auto-enable behavior that don't exist, plus pre-move
module paths. Also drop change-narration comments referencing removed
intermediate code, an unused SqliteToolSourceStore alias, and hoist the
populator's local os/biotools imports now that the module lives inside
galaxy.tools.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The LAZY_TOOL_STRICT=1 lazy Integration run surfaced three execution-path
attributes that were only materialising via the permissive fallback:
- build_dependency_shell_commands (on __SET_METADATA__, which follows
most jobs) — the runner builds the dependency/env shell commands for
the tool it's about to run
- params_with_missing_data_table_entry / params_with_missing_index_file
(data-manager / index-file tools) — param validation that walks the
parsed parameter tree at job time
All three genuinely need a parsed Tool, so add them to _MATERIALIZE_OK.
Under permissive mode this was a silent WARN + materialise; under strict
it raised NotImplementedError and failed every job-running integration
test. This is exactly the audit strict is meant to force — the fix makes
the materialisation explicit rather than accidental.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Add a materialise counter to LazyToolBox (_lazy_materialize_count, bumped
at the single _create_tool_from_stored_source chokepoint) and an
integration test that hits every batch reader the client uses — flat
listing, panel walk, panel views, tests-summary, all_requirements, and
search — asserting a zero materialise delta across them.
This catches accidental whole-toolbox materialisation that LAZY_TOOL_STRICT
can't: a tool-filter or _MATERIALIZE_OK attribute read inside a full-index
sweep parses tools without ever hitting the strict __getattr__ path. The
test was written against the search-materialisation regression fixed in
the previous commit and fails without it.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
ToolsService.search_tools resolved every whoosh hit through
_get_tool -> toolbox.get_tool, which in lazy mode loads the tool from the
store on demand. A single /api/tools?q= query therefore parsed one Tool
per hit. The populator-owned whoosh index also carries ids that were
never loaded into this toolbox (tool_conf.xml.sample alone lists ~150
legacy tools whose files aren't present), so lazy search both parsed and
returned tools the eager path skips: eager whoosh only holds loaded
tools, and eager _get_tool returns None for un-loaded ids.
Add LazyToolBox.resolve_search_hit, which looks the hit up in the
registered stubs (_tools_by_id, plus the shed short-id map) and returns
None for anything not part of this toolbox — no parse. search_tools uses
it in lazy mode and keeps the eager _get_tool path unchanged. Result:
search stays O(hits) index reads and matches eager scoping.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
AgentTools.get_tool_categories iterates toolbox.tools() and reads
tool.get_panel_section()[1] on every visible tool. get_panel_section was
not on the LazyTool stub surface, so under the permissive default that
sweep materialised the entire toolbox to build the category list (and
under LAZY_TOOL_STRICT=1 it would now raise).
The populator already stamps panel_section_id / panel_section_name onto
ToolIndexEntry, so forward the call off the entry — matching the
(section_id, section_name) / (None, None) contract of
Tool.get_panel_section — and keep the walk lazy.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
test_repository_uninstall kept resurrecting the tool after all
index-level fixes. Local repro + probes showed removal completed on the
right toolbox but two in-memory paths still served the tool:
- The materialise LRU is keyed by whatever id the caller resolved with,
so the same tool sits under both its guid and its short id; the
guid-prefix purge left the short-id entry behind. Purge by cached
tool identity instead.
- _tools_by_old_id is keyed by old_id — the SHORT id for a shed guid —
and super().remove_tool_by_id deletes from it by object identity
only, which misses when the bucket holds an earlier registration
(stub or materialised instance) than _tools_by_id. The leftover
resurfaced through the eager get_tool fall-through. Scrub every
object belonging to the removed guid; sibling installs sharing the
short id survive.
Verified locally: the full TestRepositoryInstallIntegrationTestCase
passes 3/3 consecutive lazy runs (install, uninstall, update).
remove_tool_by_id runs under app._toolbox_lock, but the invalidation
path (queue-worker reload_tool_source_cache handler — and every
populate broadcasts one) reloaded the index and re-registered stubs
without it. Interleaved with an uninstall, that reloads the pre-removal
index and re-registers the just-removed tool. Take the same RLock.
When the target <section> already exists in the on-disk shed_tool_conf,
add_to_shed_tool_config merges the incoming children into it via lxml
append — which reparents them, leaving the section elements in
elem_list empty. The lazy branch persists the conf BEFORE loading (the
populate needs it on disk), so both the partial populate (no paths →
no enriched entries) and load_item (empty section → tool never enters
the in-memory panel; only the async conf-watcher reload rescued it) ran
against gutted elements. First install into a fresh section was immune,
which is why only the second fastp changeset lost tool_shed_repository
(test_only_latest_version_in_panel_fastp — now reproduced and verified
locally in both lazy and eager modes).
Deep-copy the elements before persisting and drive the populate +
load_item from the snapshot. The eager branch loads before persisting
and stays untouched.
The partial populate filters discovered tools on exact path strings.
_collect_new_tool_paths joined the shed conf dict's raw tool_path —
relative for typical shed_tool_conf files — while discovery resolves a
relative tool_path against the conf file's directory and returns an
absolute path. The strings never matched, so the enriched conf-driven
discovery was filtered out and the ad-hoc synthesis populated a
metadata-poor entry instead: the panel served fastp without
tool_shed_repository (test_only_latest_version_in_panel_fastp), and the
store grew a duplicate row under the unresolved path string.
Route both through discover.resolve_tool_path (now public) and thread
each tool's guid so even a genuinely-unreachable path yields a
guid-keyed entry.
The data_manager_id stamp landed in index.entries only —
update_index_entry never touched entries_by_version, and the two maps
serialize independently. Job-time materialise passes a tool_version, so
it resolves through the per-version map and got the unstamped entry
back after the next index reload (populate broadcasts one constantly):
exec_after_process still failed with 'Invalid data manager (<guid>)'.
Both backends now route updates through ToolIndex.add_entry, which
keeps the maps in step (and stops an older-version update from
clobbering the newer default entry). create_tool additionally stamps
the in-memory per-version twin, which is a distinct object after a
from_dict reload.
_register_new_index_entries_as_stubs skipped every tool already in
_tools_by_id, so a stub minted from an install-time ad-hoc entry kept
serving that metadata-poor entry after the conf-driven populate
enriched the index — the fastp panel payload was missing
tool_shed_repository (test_only_latest_version_in_panel_fastp).
Swap the entry behind the existing stub instead: the panel and
registries hold the stub object, and _overrides survive.
Shed-installed data managers load their tool at install time, before
shed_data_manager_conf.xml exists — the self-healed entry carries
data_manager_id=None, materialise never restores the conf id, and
DataManagerTool.exec_after_process fails with 'Invalid data manager
(<guid>)' at job finish (all six data-manager integration tests).
DataManager._load_tool already passes data_manager_id through
load_hidden_tool; create_tool now stamps it onto the resolved entry and
persists it, so job handlers materialising from the shared index
resolve the registry correctly.
remove_tool_by_id popped the entry from every in-memory map, but the
persisted singleton index still carried it. Every populate broadcasts a
reload_tool_source_cache invalidation, so the very next reload handed
the entry back and the uninstalled tool resolved again
(test_repository_uninstall expected err_msg, got the full tool).
ToolSourceStore grows remove_index_entry — the uninstall counterpart of
update_index_entry — and remove_tool_by_id writes the removal through.
DataManagerTool resolves the registry by the <data_manager id> conf id,
which may differ from the tool XML id
(test_data_manager_async_submission_with_mismatched_conf_id). Eager
threads it through load_hidden_tool(data_manager_id=...); lazy
materialisation built the tool from the stored source alone, so
DataManagerTool.__init__ fell back to the tool id and
exec_after_process failed with 'Invalid data manager requested'.
Discovery now records the conf id, the index entry carries it, and
_create_tool_from_stored_source restores it at materialise time. Also
stamp is_local from the discovered guid — it defaulted True, which kept
the ToolConfRepository branch of the shed materialise path dead even
with repository metadata present.
Discovery stamped hidden=True on data manager tools, but eager's
load_hidden_tool only means 'not in the panel' — Tool.hidden stays
falsy, and the flat /api/tools?in_panel=false listing filters hidden
tools, so the lazy listing dropped them
(test_data_manager_async_submission_with_mismatched_conf_id).
Also gate LazyTool.allow_user_access on tool_type 'manage_data' — the
value DataManagerTool actually carries; 'data_manager' never occurs, so
the admin gate on stubs never fired and lazy was more permissive than
eager.
The lazy path capped whoosh results at tool_search_limit (default 20)
while eager ToolPanelViewSearch passes limit=None. Uniform-score matches
truncate in doc-insertion order, so a tag query fanning out to 23 tools
silently lost the last three (test_search_curated_tool_tags).
Shed installs never survived lazy mode: metadata generation
(installed_repository_metadata_manager.get_repository_tools_tups) loads
each freshly cloned tool before add_to_tool_panel persists the conf and
populates the index, so create_tool raised on every install and the
repository landed broken (data managers, workflow test tools, fastp
panel tests).
A miss for a file that exists on disk now populates that single path —
still through the single-writer populator; populate_store_inline
synthesizes a DiscoveredTool for requested paths outside every conf and
threads the caller's guid so the ad-hoc entry is keyed like its eventual
conf-driven replacement — then reloads the index and retries. Misses
for files that don't exist still raise.
Six test/functional tools (upload.xml, export_remote.xml,
ucsc_tablebrowser.xml, parse_values_from_file.xml, catWrapper.xml,
for_tours/filtering.xml) expand to byte-identical content as their
root tools/ counterparts. With hash unique, whichever file populates
first owns the row; the twin's populate is skipped and
get_by_source_path(second path) returns nothing, so LazyToolBox.create_tool
raises and the toolbox silently drops the tool — upload breaks for any
instance whose config discovers both trees (CI integration shards do).
hash becomes a non-unique index (migration amended in place — unreleased),
store() upserts by source_path, the populator skips only when the same
path already holds the same content, and get/exists/delete tolerate
multiple rows per hash.
ToolIndexEntry gains icon, xrefs, model_class, form_style and
is_workflow_compatible, derived at populate time the same way
Tool.__init__/to_dict derive them (tool_type class registry, root
workflow_compatible attribute, page count). EDAM fields now go through
expand_ontology_data, picking up the curated mapping overrides and
legacy bio.tools xrefs the direct parse_edam_* calls missed.
LazyTool.to_panel_entry emits the full client payload -- including
tool_shed_repository (the field the panel-view integration tests
assert), versions from the registered lineage, uuid, and the
admin-gated config_file from source_path.
The populator writes through its own store instances, so after the
inline backfill the toolbox's store still served its cached pre-populate
index. Single-instance that cache is merely stale-but-identical; on a
shared database (the CI integration shards share one postgres DB, and
any multi-server deployment shares one in production) the cached index
is another instance's view and the eager walk drops every tool missing
from it — the post-reload 'no index entry for upload.xml' cascade.
Reload integration tests drive galaxy.queue_worker.reload_toolbox
directly and reproduce the failure with a foreign index row; the earlier
send_control_task-based attempts passed vacuously because the control
message never executes in this harness.
- The lazy whoosh schema lacked the tool_tags field the eager schema
indexes, so a fielded phrase query (tool_tags:"send data") fanned out
to positionless NGRAMWORDS fields and raised QueryError -> 500. Index
tool_tags per entry (curated mapping over the entry ids) and search it.
- Data manager tool paths resolve with the same relative-to-conf
fallback DataManagers.load_from_xml applies for planemo layouts.
- LazyTool consults _MATERIALIZE_OK before the underscore guard and adds
_view, so admin dependency-management endpoints materialise instead of
AttributeError.
The packages/app tree links each galaxy subpackage explicitly and
tool_source_store was never added, so the built wheel lacked the package
and per-package mypy resolved its imports as Any (surfacing as
'Returning Any' in LazyToolboxSearch.search).
curated_tool_tags sweeps every tool for .tool_tags, which force-
materialised the whole toolbox on first /api/tags/tool_tags request and
500d whenever one indexed tool can't build (filter_data_table's
column="value" dynamic options). The tag mapping is keyed purely by
tool id, so the stub now answers without materialising.
Data manager tools are loaded post-boot via DataManagers ->
load_hidden_tool, which the discover walk missed — create_tool's index
miss aborted DataManager loading and the tools 404d under lazy mode.
discover_tools now walks data_manager_conf/shed_data_manager_conf
(path + guid resolution mirroring DataManager._load_from_element), and
the seam's source-path lookups normalize to absolute paths since data
manager tool files are joined from a possibly relative tool_path.
GalaxyAppConfiguration materializes every schema option, so the
hasattr/getattr guards around tool_configs, shed_tool_config_file,
migrated_tools_config, root, tool_search_index_dir and use_lazy_toolbox
were dead (StandaloneInstallationTarget's Config stub gains the
use_lazy_toolbox attribute the real config carries). The
tool_source_url getattr had no matching schema option — the default
sqlalchemy store is now disk-path only; full URLs remain available via
named tool_source_stores entries.
getattr(entry, "source_path") was hiding a real gap: ToolIndexEntry
never carried the field, so LazyTool.config_file always materialised.
The populator now stamps source_path onto every entry.
discover.py's duplicated _looks_like_a_tool is replaced by the real
galaxy.tool_util.loader_directory.looks_like_a_tool (threading
enable_beta_tool_formats), and the works-without-galaxy fallback for
MODEL_TOOLS_PATH is gone — the populator always runs with galaxy
importable.
The job-request path creates and reads tool_source rows whose source
column carries the raw source string; the store was writing dict-shaped
payloads into the same table, and once identity hashes aligned the job
path could resolve a store row and fail pydantic validation (500 on
every request-style tool execution under lazy mode).
Give the store its own content-addressed table with real columns —
get_by_tool_id / get_by_source_path become indexed queries instead of
full-table JSON scans, populator pruning can never delete rows the job
path references, and the identity_hash coupling (and the legacy
__tool_index__ sentinel-row fallback) disappear.
- AbstractToolBox gains a no-op invalidate_index_cache (LazyToolBox
overrides it); shed installs no longer depend on the subclass type.
- Annotate LazyToolboxSearch.build_index, ToolFileWatcher.observer,
populator session cast, UUID key coercion in _register_lazy_entry,
and drop stale type: ignore comments.
The lazy path persists shed_tool_conf before load_item because the
populator's conf walk needs it on disk; that same reorder broke eager
version-replacement in the panel (test_add_twice). Split the paths:
eager is now byte-identical to upstream, the reorder applies only under
use_lazy_toolbox.
The lazy ``to_dict`` fast path was short-circuiting *both* the
panel-view walk (intended) *and* the ``/api/tools/{id}`` show endpoint
(unintended) — they're indistinguishable at the ``to_dict(io_details=
False, link_details=False)`` callsite. ``test_legacy_biotools_xref_
injection`` caught it: the show endpoint returned the entry-shape
dict and no longer carried ``xrefs``.
Split the two call sites at the API rather than the parameter:
* ``Tool.to_panel_entry(trans)`` — new method, returns the cheap
entry-shape dict (id/name/version/description/labels/edam/panel
section/link). ``LazyTool`` overrides it to skip materialise.
* ``AbstractToolBox.get_tool_to_dict`` calls ``to_panel_entry`` when
``tool_help=False`` (the panel-view default); ``tool_help=True``
still falls through to ``to_dict`` so the help payload renders.
* ``LazyTool.to_dict`` always materialises now — it's the
``/api/tools/{id}`` contract and must ship ``xrefs`` / ``versions``
/ ``is_workflow_compatible`` / ``tool_shed_repository`` / etc. The
materialise-failure fallback now reuses ``to_panel_entry``.
Net effect: ``/api/tool_panels/{view}`` and ``/api/tools?in_panel=
true`` stay materialise-free for the lazy toolbox; ``/api/tools/{id}``
serves the full show-endpoint dict in both modes.
`LazyTool` materialise during workflow ``inject_all`` calls
``store.get(hash)``; the store ran the SELECT on Galaxy's shared
scoped session and then called ``session.rollback()`` to release the
implicit read transaction. That rollback expired *every*
request-scoped instance attached to the shared session — including the
in-flight ``workflow.steps`` collection. The next access in the same
``populate_module_and_state`` re-fetched a fresh ``WorkflowStep`` list
from the DB, with ``.module`` unset on the new instances, so
``compute_runtime_state``'s ``assert step.module`` raised
``AttributeError: 'WorkflowStep' object has no attribute 'module'``.
Every workflow-invocation test under ``use_lazy_toolbox=true`` 500'd.
Fix: read methods on ``DatabaseToolSourceStore`` (``get``, ``exists``,
``get_by_tool_id``, ``get_by_source_path``, ``count``, ``list_all``,
``load_index``) now open a private ``Session`` bound to the same
engine and close it on exit. The shared scoped session — and its
caller's transaction state — is untouched. The original lock-release
intent (cold-start populator, queue-worker init) still holds because
each private session's read transaction ends when the session closes.
Writes (``store``, ``store_index``, ``delete``, ``update_index_entry``)
still use ``_get_session()`` so they participate in the caller's
transaction and commit in the right context (populator, queue worker,
``LazyToolBox.__init__``).
mvdbeek