Pure refactor: the conf walker now lives at
``lib/galaxy/tool_source_store/discover.py`` and is importable via
``from galaxy.tool_source_store.discover import discover_tools``. No
sys.path injection needed for in-process callers (cold-start
auto-populate, shed-install reroute, test driver in upcoming commits).
``populate_store.py`` updates its two import sites; behaviour unchanged.
Round out the index discovery so the lazy toolbox boots against the full
shipping tool set:
- ``scripts/tool_source/_discover.py`` now handles ``<tool_dir>`` panel
entries and reads YAML tool ids alongside XML, so multi-version tool
directories and YAML tools land in the index.
- ``LazyToolBox`` extends the bootstrap path to register every variant
discovered for a single tool id (was previously dropping siblings) and
uses the shared YAML id reader when chasing direct ``<tool file=...>``
references from the panel config.
- ``DatabaseToolSourceStore.store_index`` accepts the richer per-id
payload from the discovery script.
Test coverage is extended in ``test_tool_source_storage`` to assert that
the bootstrapped index sees both directory-based and YAML-defined tools.
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink. Also drop an unused back-compat alias and
fix stale doc references.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Introduces galaxy.tool_source_store: a queryable store of parsed,
macro-expanded tool sources plus a pre-computed ToolIndex of the metadata
batch consumers need (panel section, labels, EDAM, requirements, test
counts, shed provenance).
- Backends: database (store-owned tool_source_record + tool_index tables,
migration f5a73c8b9d12), sqlalchemy (read-only SQLite bundles, e.g.
CVMFS-shipped), and a composite that layers named stores over the
default with read fall-through.
- scripts/tool_source/populate_store.py walks tool_conf files and writes
sources + index + whoosh search index; --watch updates the store on
file changes and broadcasts reload_tool_source_cache. The populator is
the single writer; consumers are read-only.
- Per-conf opt-in via store="name" on a toolbox conf root, resolved
against the tool_source_stores catalog in galaxy.yml.
- The tool_source_record table is deliberately separate from tool_source,
whose rows belong to the job-request path and carry a raw-string
payload contract.
Nothing reads from the store yet — an on-demand-loading toolbox consuming
it is follow-up work (#22633). Extracted from that PR.
Flip the relationship between directives.yml and the Python validator: the YAML
is now authoritative and scripts/markdown_directives_doc.py generates everything
downstream of it:
- lib/galaxy/managers/_markdown_directives.py - generated VALID_ARGUMENTS,
EMBED_CAPABLE_DIRECTIVES, SHARED_ARGUMENTS, DynamicArguments; markdown_parse
re-exports these, staying self-contained (no YAML at runtime) so it remains
reusable outside Galaxy (e.g. gxformat2).
- client .../Utilities/requirements.yml - generated from each directive's
'requires', removing the hand-maintained duplicate.
- directives.md - unchanged generator, now fed by the same source.
Add _shared_arguments to directives.yml (homes 'collapse' as data). Replace the
now-tautological yml-vs-parser cross-check with internal yml validation plus a
dispatch-coverage check against markdown_util.py (the gap that hid the dead
instance_citation_bibtex directive). Guard all three generated artifacts in
test_markdown_directives_doc.py and via make client-gen-markdown-directives.
Emit the directives.md tables pre-formatted to match prettier (column padding),
so the generated reference is prettier-stable and needs no ignore entry; drop the
ineffective client/.prettierignore line for it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enrich directives.yml into the source of truth for directive docs: per-param
type/context/default/description (via shared _parameter_sets), embeddable,
requires, category, renders; add missing instance_*_link and visualization
entries.
Add scripts/markdown_directives_doc.py to render directives.md and cross-check
the yml against markdown_parse.py (VALID_ARGUMENTS, EMBED_CAPABLE_DIRECTIVES,
SHARED_ARGUMENTS) + requirements.yml. Guard via test_markdown_directives_doc.py
and `make client-gen-markdown-directives`, mirroring config-rebuild. directives.md
is generated, so prettier-ignore it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop Python 3.8 support in 5 Pulsar-compatible packages
(job_metrics, tool_util, tool_util_models, util, objectstore)
and run pyupgrade --py310-plus on their source code.
- Bump requires-python from >=3.8 to >=3.10 in all 5 packages
- Remove Python 3.8 and 3.9 classifiers
- Remove ruff per-file-ignores for UP rules on these paths
- Remove backports.zoneinfo conditional dependency
- Remove pydyf<0.11 pin from conditional-requirements.txt
- Update Makefile pyupgrade target (remove PY38_PYUPGRADE_PATHS)
- Update CI workflow to test with Python 3.10 instead of 3.8
Clean up unused deprecated typing imports after pyupgrade
Remove now-unused typing imports (Dict, List, Optional, Set, Tuple,
Type, Union) that became dead after pyupgrade --py310-plus converted
annotations to use built-in types and | syntax.
Also run ruff check --fix --select=UP007,UP045 across the entire
codebase to convert remaining Optional[X] -> X | None and
Union[X, Y] -> X | Y patterns.
Enable ruff UP007/UP045 for Python 3.10 union syntax
Remove UP007 (Union[X,Y] -> X | Y) and UP045 (Optional[X] -> X | None)
from the ruff ignore list and convert all type aliases across the
codebase. These rules were deferred while Python 3.9 was supported;
requires-python is now >=3.10.
A custom script was used because neither ruff --fix nor
pyupgrade --py310-plus converts Optional[X]/Union[X,Y] in type alias
positions (e.g. X = Union[A, B]) — they only handle annotation
positions (e.g. def f(x: Optional[int])). All 167 violations were
module-level type aliases. A few edge cases were fixed manually:
single-element Union[X,], typing.Union qualified refs, runtime
Optional[type] calls, and Annotated[Optional[...]] pydantic fields.
Fix UP007 autofix regression with string forward reference type aliases
Commit fa6bd955a0 enabled ruff UP007/UP045 and auto-fixed
module-level type aliases using Union with string forward
references, producing invalid 'str | str' expressions.
This was a known ruff bug (charliermarsh/ruff#826) that has since
been fixed in later ruff versions, but this codebase was
converted before the fix was in place.
Revert to Union syntax and restore TYPE_CHECKING imports that
ruff's TCH rule cleaned up as a side effect when it thought the
forward references were unused.
Skips the physical file removal process for datasets stored in user-defined object stores. These storage locations cannot be resolved by the cleanup script, so the database records are updated while the external files remain intact to prevent potential errors or data loss.
- Extract FavoritesManager (lib/galaxy/managers/favorites.py): owns
normalize / resolve / persist / order logic; controllers in users.py
are now 2 lines each, and trans.sa_session.commit moves out of the
controller layer. Drops the redundant re-normalize on every write
and replaces FavoriteObjectType.X.value == .value defensive dispatch
with enum-identity comparison. Renames the set_favorite_order
endpoint summary away from the ambiguous "top-level" wording.
- Move curated_tool_tags_by_id off the toolbox into ToolsService;
the controller now routes through the service instead of poking
trans.app.toolbox directly. Raw aggregates (curated_tool_tags,
tool_edam_operations, tool_edam_topics) remain on the toolbox since
they're derived from per-tool data.
- Rename /api/tools/tags -> /api/tags/tool_tags (operation_id
tags__tool_tags) to disambiguate from /api/tools/{id}; update the
client store, tests, and the hand-edited OpenAPI schema entry.
(Run `make update-client-api-schema` before pushing to regenerate
the schema in its canonical alphabetical position.)
- Promote inline imports to module-level (ontology_data.py logging,
app/__init__.py configure_tool_tag_mapping); replace
getattr(self.config, "tool_tag_mappings_file", None) with direct
attribute access now that the option is registered in the schema.
- Convert ontology_data's import-time bundled-mapping reads to
lru_cache so the YAML / TSV files load lazily; fix the
Tool._setup_id comment to point at the actual lowercasing site
(Tool.parse around `self_ids = [self.id.lower()]`).
- doc/source/admin/tool_panel.rst: 26.0 -> 26.1.
- scripts/extract_tool_sections_from_api.py: annotate the public
helpers the test suite imports.
- test_to_dict_cache_drops_when_tool_removed now asserts observable
to_dict() output before/after remove, not private cache membership.
- Split test_tool_discovery_landing into focused tests; extract a
helper that opens the discovery view and types the standard filter.
- Move the Playwright drag helper from test_tool_panel_search.py into
TestWithSeleniumMixin.playwright_drag_item_above so future tests
reuse it.
- Client tests: drop the full-string Whoosh equality in
utilities.test.ts; hard-code expected Whoosh strings in
ToolsList.test.ts (drop createWhooshQuery as both sides of the
assertion) and remove the redundant injectTestRouter mount option
(ToolsList.vue only uses the useRouter composable); drop the
FontAwesome data-icon/iconName assertions in ToolPanel.test.ts in
favor of icon-existence checks; replace three near-identical
add/remove favorite tests in ToolsListCard.test.ts with an it.each;
export quoteToolTagValue from Panels/utilities so
filterConversion.test.js imports the real helper instead of
reconstructing it inline; delete the spy-only "loads the curated
tag mapping" test in ToolBoxSearch.test.ts (the gate is covered by
ToolPanel.test.ts's negative case and the favorite-tag rendering
tests cover the positive case end-to-end).
Two related fixes around the admin-facing helper script and admin docs:
* Make the helper script work from any cwd and against any Galaxy
server: `--api-url`, `--output`, `--timeout` argparse flags; raise
`RuntimeError` on transport / decode failures (was a silent no-op);
anchor the default output path on the script location so
`python scripts/extract_tool_sections_from_api.py` works without a
prior `cd`. Default output now points at `config/tool_tag_mappings.yml`
to match the new admin override path.
* Add a "Curated tool tags" section to `doc/source/admin/tool_panel.rst`
documenting the new `tool_tag_mappings_file` option, the new
`tag:"…"` / `ontology:"…"` search syntax, the `?section=` URL
deprecation, and the reindex requirement after changing the mapping.
Nest the prebuilt/hash-check branches inside `if [ $SKIP_CLIENT_BUILD
-eq 0 ]` so the shape mirrors the subsequent build-or-install block.
Per Nicola's review.
The prebuilt client path (GALAXY_INSTALL_PREBUILT_CLIENT=1, used by
planemo) now pulls the matching galaxy-web-client wheel from PyPI and
lets webapp.py pick it up via `import galaxy.web_client`. The
root-level pnpm install is no longer needed for this flow -- the
Galaxy client isn't a JS library, so shipping it as an npm package
was always the wrong shape.
Dev versions don't have a published wheel, so we bail early with a
message pointing users at the source build path.
No root pnpm-lock.yaml is shipped, so `pnpm install --frozen-lockfile`
fails with ERR_PNPM_NO_LOCKFILE (hit locally via planemo). The dev
branch is unaffected -- client/pnpm-lock.yaml exists.
only directories in `lib/galaxy` & `packages/*/galaxy`
All other files removed or moved to their own module
All references to those files updated, especially galaxy/version.py
Replaces yarn with pnpm throughout the Makefile and common_startup.sh.
The startup script now installs pnpm via conda, corepack, or npm
depending on the environment.
by runnig:
```shell
make pyupgrade
ruff check --fix .
make format
```
Also update ruff configuration to:
- Enable UP006 and UP035 error codes for PEP 585 type annotations
- Ignore UP045 for the packages which need to stay compatible with Python 3.8
- Add tool-util-models paths to those that need to stay compatible with Python 3.8
Updates repository_metadata.tools with new and correct file locations.
The test toolshed moved around `file_path` multiple times since its
inception, and old revisions might point at old location. This is a
simple script that fixes the locations. It has a dry run mode that I've
run successfully:
```
----------------------------------------------------------------------
Dry run complete - no changes made to database
======================================================================
SUMMARY
======================================================================
Total RepositoryMetadata records processed: 25414
RepositoryMetadata with tools: 22497
Total tools examined: 40470
Tools with correct paths (unchanged): 3898
Tools with paths updated: 36572
- Tool file exists on disk: 33297
- Tool file missing (removed in later changeset): 3275
Tools with path construction errors: 0
======================================================================
```
it also write out json files for each repository as backups should
this be needed.
Closes https://github.com/galaxyproject/galaxy/issues/21454
mvdbeek