Cover how the SSE pipeline is wired (HistoryAuditMonitor +
galaxy-sse-monitor + control-task fan-out + per-worker connection
manager), how to configure it, what statsd metrics to watch, and
how to set up nginx so /api/events/stream is not buffered.
Wire the new page into the admin toctree and add cross-references
from the nginx and scaling guides where SSE has operational impact.
Replace enable_sse_history_updates and enable_sse_entry_point_updates
with a single enable_sse_updates. The two flags shared the same SSE
connection plumbing (events/stream + dispatcher), and only the
HistoryAuditMonitor lifecycle has real per-process cost — entry-point
dispatch is event-driven and effectively free. One knob admins can flip.
Updates schema, sample config, server (app, configuration manager,
history_audit_monitor, sse_monitor), client stores and their tests,
integration tests, and the regenerated admin docs.
Three related additions to the SSE notification pipeline, bundled here
because they share ``SSEEventDispatcher._send`` as their modification
point:
1. Interactive-tool entry-point SSE channel
- ``entry_point_update`` dispatcher method + queue-worker handler.
- ``InteractiveToolManager.configure_entry_points`` dispatches a
wake-up event after the DB commit; the client refetches
``/api/entry_points`` on receipt (no payload).
- Frontend: new SSE event type, store subscription with XOR polling
fallback via ``enable_sse_entry_point_updates`` config flag.
- Integration + selenium tests.
2. Queue and SSE observability metrics
- Counters, timers, and periodic gauges for SSE dispatch, control-queue
task execution, control-queue depth (via kombu passive declare),
active SSE connections, dropped events, and active WorkerProcess
rows. Flow through the existing ``galaxy_statsd_client`` — no new
infra. Gauges are scheduled via Celery beat at ``queue_metrics_interval``
seconds (default 15). All instrumentation no-ops when statsd isn't
configured.
- Sub-emitter failures log once at WARNING and bump a
``galaxy.queue_metrics.error`` counter tagged by emitter so broken
emitters are visible in metrics without log spam.
3. Active-worker control-queue cache
- 30 s TTL cache on ``all_control_queues_for_declare`` with RLock
stampede protection. At 1000+ events/s this eliminates ~30 DB
round-trips/s per webapp for data that only changes on 60 s
heartbeat cadence. Empty results are not cached — would otherwise
silently drop every SSE event during the startup window.
Add real-time notification delivery via SSE to replace the 30-second
polling interval. The SSE endpoint streams notification_update,
broadcast_update, and notification_status events to connected clients.
Backend:
- New SSEConnectionManager (lib/galaxy/managers/sse.py) maps user IDs
to asyncio queues with thread-safe push via call_soon_threadsafe
- SSE streaming endpoint at GET /api/notifications/stream with
Last-Event-ID catch-up support and 30s keepalive
- Kombu control tasks (notify_users, notify_broadcast) fan out events
across all Galaxy worker processes
- Existing polling API unchanged for backward compatibility
Frontend:
- New useNotificationSSE composable using EventSource with auto-reconnect
- notificationsStore tries SSE first, falls back to polling after 5+
consecutive errors
Tests:
- API integration tests for SSE event delivery, broadcasts, and reconnect
- Selenium E2E tests for notification appearance and bell indicator
Add SSE-based real-time history update notifications
Replace aggressive 3-second history polling with Server-Sent Events
driven by database change detection, configurable via admin setting.
Backend:
- Add pg_notify() to PostgreSQL audit triggers for instant LISTEN/NOTIFY
- New HistoryAuditMonitor: PG LISTEN/NOTIFY with SQLite polling fallback
- New /api/events/stream SSE endpoint (uses StructuredApp, not MinimalManagerApp)
- Kombu control task "history_update" with message TTL (expiration=10s)
- Config: enable_sse_history_updates, history_audit_monitor_poll_interval
Frontend:
- Generalize useNotificationSSE → useSSE composable with event type filtering
- historyStore connects SSE for history_update events, triggers immediate refresh
- notificationsStore updated to use /api/events/stream and useSSE
- Polling kept as fallback at existing intervals
Tests:
- 5 integration tests: endpoint, dataset upload event, history ID in payload,
cross-user isolation, polling backward compatibility
Documents the 7-column `huggingface` data table schema, column
conventions, controlled vocabularies for pipeline_tag/domain, XML
filter patterns, and an example .loc entry.
Announced in galaxyproject/galaxy-hub#3923.
Introduce lib/galaxy/tool_util_models/yaml_parameters.py with a narrow
YamlGalaxyToolParameter union (extra=forbid) used for UserToolSource.inputs
and ToolSourceBase.inputs. v1 supports boolean, integer, float, text, select,
color, data, data_collection, conditional, repeat, section. XML-only fields
(truevalue, falsevalue, argument, is_dynamic, hidden, parameter_type) and
unsupported types (hidden, drill_down, data_column, genomebuild, group_tag,
baseurl, rules, directory) now reject at parse time.
to_internal() maps each YAML model onto the existing internal parameter
metamodel so no downstream pipeline needs to change shape. Production path
keeps re-parsing via YamlToolSource (plan option a).
UserToolSource envelope also gets extra=forbid - stray top-level keys like
argument: at the tool level fail fast. Regenerated ToolSourceSchema.json is
32KB (was 77KB); Monaco no longer autocompletes the leaked fields.
runtimeify() gains yaml_origin flag; evaluation.py passes it based on
tool_source.parse_class(). assert_yaml_v1_parameters() hard-fails on any
out-of-v1 type inside a YAML-origin bundle, recursing through whens/repeat/
section. XML path untouched.
doc/source/admin/user_defined_tools.md documents the supported set and
rejected types. 34 new tests in test_yaml_parameters.py cover red cases,
green round-trips, structural groups, the runtimeify guard, and a snapshot
blacklist on the published schema. test_parsing.py still green (83 tests).
No compat shim for existing beta DB rows that contain XML-only fields -
those will now fail validation on load and need to be recreated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a celery beat for renewal of Hashicorp Vault tokens.
This eliminates the need for manual token rotation when
using short-lived renewable tokens.
The approach:
- HashicorpVault checks token renewable status on startup (warning
if not renewable)
- New `renew_vault_token` Celery Beat task calls renew-self
- Configured via `vault_token_renewal_interval` in galaxy.yml
(default 0 = disabled)
- Requires Celery Beat to be running
Fixes https://github.com/galaxyproject/galaxy/issues/22187
Separate from the server-side sentry_traces_sample_rate so admins can
independently control browser performance trace sampling. Defaults to 0
(no client performance traces).
Dead option - nothing reads it, pause module always available regardless of setting. Add deprecation handler so existing configs don't error on upgrade.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deprecated ai_api_key/ai_model/ai_api_base_url keys were front and
center in the original doc. This flips the emphasis to inference_services
as the recommended config path, demoting the deprecated keys to a note.
Also removes dataset_analyzer (no implementation exists) and the agents
enabled/disabled config section (not wired up yet).
More portable and idiomatic.
Also:
- Replace `os.path.abspath(f"{dir}/../..")` with `os.path.dirname()`
calls.
- Add test for `get_alternative_versions()` with `recipes_path`
argument specified.
Document configuration options needed for handling large history exports:
- Celery result_expires setting to prevent task results from expiring
- Short-term storage duration settings for export file retention
- Combined configuration example for large history export scenarios
Fixes: https://github.com/galaxyproject/galaxy/issues/21701
Implement full Workflow Execution Service (WES) API support for running Galaxy
workflows via the GA4GH standard. Includes complete core functionality with run
submission, status tracking, cancellation, input/output handling, and task logs.
Core WES Features:
- 6 WES API endpoints (service-info, runs CRUD, cancel, status)
- Support for both workflow_url and workflow_attachment input methods
- Support for gxworkflow:// URI scheme for direct database workflow references
- Automatic history creation with optional custom naming
- Full workflow support: gx_workflow_ga and gx_workflow_format2
- DRS URI generation for all workflow outputs
- State mapping between Galaxy invocation states and WES states
- Cursor-based pagination for run listings
Task Log Features:
- /api/jobs/{job_id}/stdout - Job stdout as plain text
- /api/jobs/{job_id}/stderr - Job stderr as plain text
- /ga4gh/wes/v1/runs/{run_id}/tasks - Paginated task list
- /ga4gh/wes/v1/runs/{run_id}/tasks/{task_id} - Task details
- Build TaskLogs from workflow invocation steps
- Populate RunLog with task_logs list and task_logs_url
Implementation Highlights:
- Auto-generated Pydantic models from GA4GH WES OpenAPI spec
- FastAPI CBV router following Galaxy API patterns
- Service layer with full workflow submission/execution pipeline
- Shared GA4GH utilities for DRS code reuse
- Reduced test duplication with helper functions
Files Added:
- lib/galaxy/schema/wes/__init__.py - Generated Pydantic models
- lib/galaxy/webapps/galaxy/api/wes.py - WES API router
- lib/galaxy/webapps/galaxy/services/wes.py - WesService
- lib/galaxy/webapps/galaxy/services/ga4gh.py - Shared GA4GH utilities
- WES_PLAN.md - Implementation plan
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
mvdbeek