The LazyTool docstring claimed strict-by-default with a nonexistent
LAZY_TOOL_PERMISSIVE toggle (the code is permissive by default, toggled
by LAZY_TOOL_STRICT=1). The dev doc described a Redis backend, an API
layer (api/tool_sources.py, ToolAPICache), a _init_lazy_toolbox
mechanism, and an auto-enable behavior that don't exist, plus pre-move
module paths. Also drop change-narration comments referencing removed
intermediate code, an unused SqliteToolSourceStore alias, and hoist the
populator's local os/biotools imports now that the module lives inside
galaxy.tools.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The job-request path creates and reads tool_source rows whose source
column carries the raw source string; the store was writing dict-shaped
payloads into the same table, and once identity hashes aligned the job
path could resolve a store row and fail pydantic validation (500 on
every request-style tool execution under lazy mode).
Give the store its own content-addressed table with real columns —
get_by_tool_id / get_by_source_path become indexed queries instead of
full-table JSON scans, populator pruning can never delete rows the job
path references, and the identity_hash coupling (and the legacy
__tool_index__ sentinel-row fallback) disappear.
- doc/source/admin/tool_source_storage.rst: operator-facing guide
covering backend choice, populate_store usage, watch mode, and the
per-conf composite story.
- doc/source/dev/tool_source_storage.rst: developer-facing
architecture overview of the store, LazyToolBox, and the batch
endpoint integration.
Ownership-only check rejected published/shared workflows that a normal
invocation would accept. Defer to get_stored_accessible_workflow, update
docs, add API tests for published + inaccessible cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a `github` file source plugin backed by fsspec's built-in GithubFileSystem,
usable as a user-instantiated "Bring Your Own Storage" template. Each instance
exposes a single repository (org/repo at a branch or commit) for browsing and
download; users create one instance per repository. Optional write support commits
uploaded files to the branch via the GitHub Contents API with an automatic commit
message (gated by `writable`).
Authentication uses Galaxy's OAuth2 flow (Dropbox-style): an admin registers a
GitHub App and provides its client id/secret; each user authorizes Galaxy and the
per-user refresh token is stored in the vault. The access token is sent to GitHub
as a Bearer credential via a WritableGithubFileSystem `kw` override.
Supporting GitHub required a few framework changes to the shared OAuth2 code:
- OAuth2 token requests always send `Accept: application/json` so GitHub's token
endpoint returns JSON (it otherwise returns form-encoded data). This is correct for
every provider per RFC 6749, so it is sent universally rather than per-provider.
- `read_oauth2_info_from_configuration` now expands only the oauth2 client fields, so
a template whose configuration references user variables (org/repo) can be read at
authorize time before those variables are collected.
- The oauth2 refresh flow now persists a rotated refresh token when the provider
returns one. GitHub rotates and invalidates the refresh token on every use; other
providers omit it and are unaffected.
- The create-instance UI treats `github` as an OAuth2 type so it triggers the
authorization redirect.
Registers the `github` OAuth2 template type, ships a production example, and
documents the GitHub App setup in admin/data.md. No new dependency: fsspec is core
and ships the github implementation.
Shorten the tool_source_database_connection description, drop the
'lazy toolbox' lingo in favor of a link to the admin documentation,
lead the storage doc with the benefits, use sqlite in the shared-store
examples, stop framing watch mode as development-only, and replace the
nonexistent use_lazy_toolbox option in the setup recipe.
The store is a rebuildable, content-addressed cache — it does not need
Galaxy's database, and living there was the source of every subtle
lifecycle mechanism in the subsystem: store.commit() committed whatever
request or job transaction was in flight, reads needed a private-Session
isolation hack, and a sa_session parameter threaded through the factory,
populator, watcher and CLI.
Standalone stores are now the only kind: the default is a SQLite file
under tool_source_disk_path (any SQLAlchemy URL via tool_source_stores
for shared multi-host deployments), committing per write on its own
engine. The tool_source_record/tool_index tables, their migration, the
model classes and the whole session plumbing are gone. The ToolSourceStore
ABC loses commit() (every backend persists per write). The dev/watch-mode
watcher now populates per file without broadcasting and keeps its single
debounced reload notification for the batch.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink. Also drop an unused back-compat alias and
fix stale doc references.
Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
Introduces galaxy.tool_source_store: a queryable store of parsed,
macro-expanded tool sources plus a pre-computed ToolIndex of the metadata
batch consumers need (panel section, labels, EDAM, requirements, test
counts, shed provenance).
- Backends: database (store-owned tool_source_record + tool_index tables,
migration f5a73c8b9d12), sqlalchemy (read-only SQLite bundles, e.g.
CVMFS-shipped), and a composite that layers named stores over the
default with read fall-through.
- scripts/tool_source/populate_store.py walks tool_conf files and writes
sources + index + whoosh search index; --watch updates the store on
file changes and broadcasts reload_tool_source_cache. The populator is
the single writer; consumers are read-only.
- Per-conf opt-in via store="name" on a toolbox conf root, resolved
against the tool_source_stores catalog in galaxy.yml.
- The tool_source_record table is deliberately separate from tool_source,
whose rows belong to the job-request path and carry a raw-string
payload contract.
Nothing reads from the store yet — an on-demand-loading toolbox consuming
it is follow-up work (#22633). Extracted from that PR.
The custom tool authoring agent (galaxy.agents.custom_tool) gains container
resolution and producer-output safeguards. It consumes the mulled recommender
(galaxy.tool_util.deps.mulled.recommend):
- Container resolution: a dedicated container critic infers the conda packages
from the tool's shell_command/configfiles, the recommender resolves a verified
quay.io image, and a deterministic gate applies it -- overriding an
exact-version match, a verifiably-absent biocontainer tag, or a non-biocontainer
image, while leaving a present pinned biocontainer alone. Runs independently of
the quality critic.
- Producer reliability: inputs/outputs are required in the LLM authoring view so
grammar-constrained models can't drop them; UserToolSource stays lenient for the
API and stored tools.
- The quality critic supplies deterministic field edits (labels/help/...) applied
in place, falling back to a full re-roll only for structural changes.
- Reject a shell_command that runs a script by name with no configfile to create
it, driving a producer retry.
- Render saved tool YAML with exclude_defaults so it carries only authored fields.
Admin config docs (config_schema.yml / galaxy.yml.sample) describe the
quality_critic_enabled and container_recommendation_enabled flags.
Tests cover the authoring-view required fields, container override paths,
deterministic critic edits, and the script check.
A reusable resolver (galaxy.tool_util.deps.mulled.recommend) that picks the best
quay.io/biocontainers image for a set of conda packages, verified against quay.io
rather than guessed:
- recommend_container() resolves single-package and mulled-v2 (multi-package)
images, distinguishing an exact-version hit from a name-only fallback and
resolving unpinned versions against tags that are actually built.
- biocontainer_tag_built() verifies whether a specific image tag exists.
- Shared lookup helpers move into mulled/util.py and are reused by the existing
container resolver (container_resolvers/mulled.py) to avoid duplication.
Exposed as the mulled-recommend console script:
mulled-recommend samtools=1.17
mulled-recommend bwa,samtools
mulled-recommend --json "samtools=1.3.1,bedtools=2.26.0"
Missing containers and transient network errors degrade gracefully (no exception).
Documented under admin special topics (mulled containers).
@ElectronicBlueberry left the Freiburg Galaxy Team for greener pastures
a few months ago, thanks for all your contributions to Galaxy and the
community!
Update Galaxy's Rucio test and optional dependency declarations to require rucio-clients 40.2.0.
Adjust the Rucio object store for client 40 behavior, including checksum import changes, ingest checksum handling, Rucio config cache invalidation, and isolated download directories.
Run Rucio integration tests against the Savannah Rucio 40.2.0 container by default while keeping an environment override for the image.
Point-in-time counts (SSE connections, control-queue depth, worker
processes) were emitted via statsd timing(), so telegraf/influxdb stored
them as timing aggregations (mean/median/percentiles) instead of a single
value. Add a gauge() method to the statsd client and emit these as gauges.
The SSEConnectionManager is per-process state registered only on the
webapp, so sampling it from the Celery beat task saw an empty manager and
always reported ~0. Move SSE connection sampling into each web worker via
a per-worker daemon thread (SSEConnectionGaugeEmitter), tagging samples
with server_name so per-worker series don't collide. It is opt-in via the
new enable_sse_connection_metrics config (default off) and only runs when
statsd is actually configured.
mvdbeek