Mypy on CI flagged ``second_run_state.get_prior(...).content`` because
``get_prior`` returns Optional[AgentResponse]. The preceding ``is not
None`` assert doesn't narrow the next call's return type. Bind the
result first, assert non-None, then access .content.
The orchestrator was passing prior agent output to the next agent by
text-concatenating "Previous analysis from {name}: ..." onto the user's
query, capped at 2000 chars. error_analysis then sniffed for the literal
string "Previous analysis" to know it had upstream context. That coupling
is brittle, lossy, and hides what's flowing between agents.
This adds a small AgentRunState dataclass (prior_responses dict plus
record/get_prior helpers) that the orchestrator instantiates per user
query and threads through each sequential agent's context. Each agent
gets the original query verbatim and reads upstream AgentResponses
structurally from run_state. error_analysis now pulls the prior history
response from run_state and appends its full content as "Context from
history analysis:" alongside the existing job_id-driven augmentation.
Parallel execution doesn't need run_state and isn't touched.
Tests cover the dataclass round-trip, that sequential agents share one
run_state instance and see the original query (no "Previous analysis
from..." mutation), and that error_analysis incorporates a recorded
history response into its prompt.
Owner dispatch only delivers history_update events to the user who owns
the changed history. Watching a shared or pinned non-owned history
(e.g. via the multi-history view) silently received nothing under SSE
because the dispatch path keys on history ownership and there was no
way for a viewer to opt in.
Add an additive viewer-subscription channel: the client POSTs the
history ids it wants pushes for to /api/events/history-subscriptions,
the dispatcher fans the record out to every webapp worker over Kombu,
and history_update fanout pushes to subscribed viewers in addition to
the owner. Subscriptions are refcounted on the client and replayed on
every EventSource onopen so reconnects don't drop them. Per-worker
subscription state is purged when a user's last connection on that
worker closes; the client's onopen replay re-asserts the desired set.
No authorization check on the dispatch path: SSE events only carry
history ids, the client follows up with a normal access-controlled
REST fetch, and leaking "history changed at T" pings is acceptable.
MultipleViewItem now subscribes whenever it renders a history the
current user does not own and SSE is enabled; owned histories continue
through owner dispatch with zero subscribe round-trip.
Tests: integration test exercising subscribe -> mutate -> receive ->
unsubscribe -> stop receiving end-to-end, plus vitest for the client
refcount and replay logic.
The history-store SSE handler only refreshed the current history. With
polling disabled under enable_sse_updates=true, other histories visible
in the multi-history view stayed stale until the user switched into
them. Iterate over all changed history ids in the SSE event and refresh
each one already tracked in storedHistories via updateContentStats and
historyItemsStore.fetchHistoryItems, so multi-view panels for
non-current histories pick up uploads as they happen.
Adds an integration_selenium test that uploads to a non-current history
through the API and asserts the new item renders in the multi-view
without a current-history switch — exercising the SSE-driven refresh
path end-to-end.
Mypy on CI runs against test files too (not just lib/), and list
invariance tripped it up: ``history = [ModelRequest(...), ModelResponse(...)]``
gets inferred as ``list[ModelRequest]``, which can't be passed to a
function expecting ``list[ModelMessage]``. Annotate the four test
fixtures explicitly.
CLAUDE.md is explicit that test files shouldn't use inline imports.
The pydantic_ai.messages and truncate_message_history imports were
duplicated across seven new tests; move them to module level (under
the existing pytest.importorskip guard so the skip behavior is
preserved when pydantic_ai isn't installed).
The chat router was flattening prior conversation messages into a text
blob via _build_query_with_context, which loses the role/content
semantics chat models already know how to use. The structured pydantic-ai
message format already exists -- ChatManager.get_chat_history has a
format_for_pydantic_ai flag returning ModelMessage objects -- but the
chat API was calling it with False and the router was re-flattening the
result.
Flips the API call to format_for_pydantic_ai=True, plumbs the structured
list[ModelMessage] through to agent.run(message_history=...), and adds
truncate_message_history that keeps the first message + last
MAX_HISTORY_MESSAGES (40) so original intent is preserved across long
conversations. Tighter cap (8) for sub-agent calls from inside
@agent.tool since tool-context turns burn token budget faster.
Drops the magic max_history=6 from router._build_query_with_context and
the 200-char-per-message text-concat from BaseGalaxyAgent's
_call_agent_from_tool.
Notifications SSE is now gated on enable_sse_updates; without it the
frontend falls back to polling and __galaxy_sse_connected never flips,
timing out the connect wait.
Replace enable_sse_history_updates and enable_sse_entry_point_updates
with a single enable_sse_updates. The two flags shared the same SSE
connection plumbing (events/stream + dispatcher), and only the
HistoryAuditMonitor lifecycle has real per-process cost — entry-point
dispatch is event-driven and effectively free. One knob admins can flip.
Updates schema, sample config, server (app, configuration manager,
history_audit_monitor, sse_monitor), client stores and their tests,
integration tests, and the regenerated admin docs.
Add _decode_id(encoded_id) and _user_id_for_api_key(api_key) to
IntegrationInstance so SSE integration tests stop duplicating
self._app.security.decode_id and ad-hoc /api/users/current calls.
Refactor test_entry_point_sse (integration + selenium) to inherit.
Move urlparse, sqlalchemy.create_engine/text, lagom.UnresolvableType,
galaxy.managers.queue_metrics, and galaxy.managers.sse to module top in
galaxy.celery.tasks. None of them protected against circular deps or
optional drivers — they were plain lazy. Hoist galaxy.model imports out
of method bodies in the selenium SSE test.
Justified inline imports kept in place: notification.py circular dep,
psycopg/psycopg2 driver fallback in history_audit_monitor, sse_monitor
entry-point lazy app boot.
Dispatch tests had to monkeypatch module-level globals (ControlTask and
all_control_queues_for_declare in galaxy.managers.sse_dispatch) because
the dispatcher constructed its ControlTask and called the queues helper
directly — the tests were effectively testing their own patch setup.
Make both seams injectable on SSEEventDispatcher:
- control_task_factory: Callable[[GalaxyQueueWorker], ControlTask],
default ControlTask.
- queues_provider: Callable[[], list[Queue]], default closes over
application_stack and delegates to all_control_queues_for_declare
(webapp_only=True) so production behavior is unchanged.
test_sse_dispatch.py and test_sse_dispatch_cache.py drop every
monkeypatch.setattr and pass FakeControlTask / a counting provider via
the constructor. 9/9 unit tests still pass and the module-global
knowledge leak from tests into production code is gone.
Architecture/DI reviews flagged lib/galaxy/webapps/galaxy/metrics as a
misplaced module — it has no webapp concern, is called from a Celery
beat task, samples DB rows and broker state, and reached into
StructuredApp as a service locator (app.amqp_internal_connection_obj,
app.application_stack, app.model, app.execution_timer_factory,
app[SSEConnectionManager]).
Move queue_metrics.py under lib/galaxy/managers where its role — a
periodic sampler mirroring the new sse.py/sse_dispatch.py — fits
cleanly. Drop the webapps/galaxy/metrics directory entirely.
Change emit_control_queue_depth, emit_worker_process_gauge and
emit_queue_metrics to accept the narrow collaborators they actually
need (Connection, ApplicationStack, GalaxyModelMapping,
VanillaGalaxyStatsdClient, SSEConnectionManager). The Celery task in
celery/tasks.py becomes the composition root that resolves those
narrow deps from the app and passes them in.
Type the send_control_task / send_local_control_task return values,
type _run(fn: Callable[[], None]), and replace StructuredApp's
execution_timer_factory: Any annotation with a TYPE_CHECKING-guarded
ExecutionTimerFactory reference so the interface is strongly typed.
Tests migrate to the new signatures and drop the _ContainerApp fake.
- test_notification_sse: connect to the unified /api/events/stream rather
than the deleted /api/notifications/stream. The old URL was matching
FastAPI's /api/notifications/{notification_id} route and failing the
encoded-id length check with HTTP 400.
- test_history_sse: add SSELineListener.wait_for_event_where(predicate)
and use it in both history-update tests. wait_for_event returned after
the first history_update of any kind, so a background update for an
unrelated history could land before the test's own upload dispatched,
causing the assertion to fail even though the event was in flight.
wait_for_selector_visible dispatches as a CSS selector on both
backends; `text=` is a Playwright engine prefix that Selenium's
css-selector path rejects with InvalidSelectorException.
wait_for_xpath_visible works on both backends.
Three related fixes to make the SSE selenium tests pass end-to-end:
- Swap ``self.driver.execute_script`` for ``self.execute_script`` so the
tests work with both the Selenium and Playwright backends; the former
raises ``NotImplementedError`` under Playwright.
- Switch user-info lookup from ``self._get("users/current")`` (API
interactor's default key) to ``self.api_get("users/current")`` (browser
cookie). ``ensure_registered`` creates a distinct Selenium user, so the
API-key user and the browser-SSE user were different and
``push_to_user`` landed in no queue.
- Add ``interactivetools_enable`` and ``enable_sse_entry_point_updates``
to the entry-point test config — App.vue won't start watching entry
points, and the store won't open an SSE connection, without both.
- Use ``self.get(path)`` (goes through ``build_url``) instead of raw URL
concatenation, and pass wait timeouts in seconds (not milliseconds);
the timeout helper multiplies by 1000 internally.
- Target ``.nav-indicator`` on the activity-bar item rather than a
``.indicator`` class that the bell component no longer renders.
Three related additions to the SSE notification pipeline, bundled here
because they share ``SSEEventDispatcher._send`` as their modification
point:
1. Interactive-tool entry-point SSE channel
- ``entry_point_update`` dispatcher method + queue-worker handler.
- ``InteractiveToolManager.configure_entry_points`` dispatches a
wake-up event after the DB commit; the client refetches
``/api/entry_points`` on receipt (no payload).
- Frontend: new SSE event type, store subscription with XOR polling
fallback via ``enable_sse_entry_point_updates`` config flag.
- Integration + selenium tests.
2. Queue and SSE observability metrics
- Counters, timers, and periodic gauges for SSE dispatch, control-queue
task execution, control-queue depth (via kombu passive declare),
active SSE connections, dropped events, and active WorkerProcess
rows. Flow through the existing ``galaxy_statsd_client`` — no new
infra. Gauges are scheduled via Celery beat at ``queue_metrics_interval``
seconds (default 15). All instrumentation no-ops when statsd isn't
configured.
- Sub-emitter failures log once at WARNING and bump a
``galaxy.queue_metrics.error`` counter tagged by emitter so broken
emitters are visible in metrics without log spam.
3. Active-worker control-queue cache
- 30 s TTL cache on ``all_control_queues_for_declare`` with RLock
stampede protection. At 1000+ events/s this eliminates ~30 DB
round-trips/s per webapp for data that only changes on 60 s
heartbeat cadence. Empty results are not cached — would otherwise
silently drop every SSE event during the startup window.
Move send_notification_internal from NotificationService to
NotificationManager so non-web callers (psa_authnz, sharable service)
can dispatch notifications without constructing a web-layer service —
which is why the sse_manager constructor drift was caught by mypy in
the first place.
Relocate async_task_summary to galaxy.celery.helpers so the manager
can call it without reaching into galaxy.webapps.galaxy.services.base;
existing service callers keep working via an explicit re-export.
Also narrow Optional api_key before SSELineListener in the SSE
integration tests, and call wait_for_selector_visible on self rather
than self.driver in the selenium SSE test.
Backend
- Narrow SSEEventDispatcher's dependencies. Previous constructor took
the whole app and reached for `queue_worker`, `application_stack`,
`config.server_name` off it (via send_control_task) — a service-
locator pattern where the `MinimalManagerApp` annotation didn't even
cover the accessed fields. Inject `queue_worker: Optional[
GalaxyQueueWorker]` and `application_stack: ApplicationStack`
directly and publish via `ControlTask(qw).send_task(...)` so the
dispatcher is unit-testable without a full app.
- Move user-id resolution, notifications-enabled guard, and catch-up
wiring out of `api/events.py` and `api/notifications.py`.
Controllers are now pure `StreamingResponse` wrappers around
`EventsService.open_stream(...)` / `NotificationService.open_stream
(...)`, matching the three-layer architecture.
- Drop the starlette `Request` leak from `managers/sse.py`.
`SSEConnectionManager.stream` now takes an `IsDisconnected`
(async callable) so the manager stays framework-agnostic; services
pass `request.is_disconnected` in.
- Centralise SSE event-id generation. New `make_event_id()` /
`parse_event_id()` helpers in `managers/sse.py` use
`galaxy.model.orm.now` (timezone-naive UTC, matching the rest of
Galaxy's DB timestamps) instead of `datetime.utcnow()` scattered
across three callers — previous code risked a tz-aware/naïve mix
that would silently drop Last-Event-ID catch-up.
- Drop dead `app.sse_connection_manager` attribute and
`StructuredApp.sse_connection_manager` field. Nothing reads them;
the manager is consumed via Lagom (`depends(SSEConnectionManager)` /
`app[SSEConnectionManager]`).
- Type-annotate the public control-task surface: `send_control_task`,
`ControlTask.send_task`, `all_control_queues_for_declare`, SSE
control-task handlers (`notify_users` / `notify_broadcast` /
`history_update`), and both new stream endpoints. Tighten
`SSEEventDispatcher.history_update` to `dict[str, list[int]]`.
Tests
- Replace the substring `data` matches with `json.loads(data)["content"]
["subject"]` so a regression in the NotificationResponse envelope
(renamed keys, missing fields) actually fails. Confirmed shape
against `NotificationResponse.model_dump_json()`.
- Assert the *absence* of pre-Last-Event-ID subjects in the catch-up
test. Without this, a server that replays everything on every
reconnect silently passes.
- Delete smoke tests that only checked content-type + 200 (subsumed by
the functional tests) and the two "existing polling API still works"
tests that didn't exercise SSE at all.
- Add `window.__galaxy_sse_last_event_ts` to the `useSSE` composable
and gate the selenium bell/notification tests on it advancing past
a baseline. The previous 15s wait would silently become a polling
test if the poll interval ever dropped — this makes "update came
from SSE" a positive assertion.
- Vitest: add a shared `_testing/sseStoreSupport.ts` helper for the
two store tests to share a `useSSE` mock that captures the
`onEvent` callback. Use it to synthesize real `MessageEvent`s and
assert store-state changes for `notification_update`,
`notification_status`, and `history_update` — the most
consequential handlers were previously uncovered. Also save/restore
`document.visibilityState` so the patching doesn't leak across
tests, drop the dead `mockSseConnected` ref, tighten the
idempotency assertion to advance time by one interval, and collapse
the repeated orchestration into a `primeStore(...)` helper.
Add real-time notification delivery via SSE to replace the 30-second
polling interval. The SSE endpoint streams notification_update,
broadcast_update, and notification_status events to connected clients.
Backend:
- New SSEConnectionManager (lib/galaxy/managers/sse.py) maps user IDs
to asyncio queues with thread-safe push via call_soon_threadsafe
- SSE streaming endpoint at GET /api/notifications/stream with
Last-Event-ID catch-up support and 30s keepalive
- Kombu control tasks (notify_users, notify_broadcast) fan out events
across all Galaxy worker processes
- Existing polling API unchanged for backward compatibility
Frontend:
- New useNotificationSSE composable using EventSource with auto-reconnect
- notificationsStore tries SSE first, falls back to polling after 5+
consecutive errors
Tests:
- API integration tests for SSE event delivery, broadcasts, and reconnect
- Selenium E2E tests for notification appearance and bell indicator
Add SSE-based real-time history update notifications
Replace aggressive 3-second history polling with Server-Sent Events
driven by database change detection, configurable via admin setting.
Backend:
- Add pg_notify() to PostgreSQL audit triggers for instant LISTEN/NOTIFY
- New HistoryAuditMonitor: PG LISTEN/NOTIFY with SQLite polling fallback
- New /api/events/stream SSE endpoint (uses StructuredApp, not MinimalManagerApp)
- Kombu control task "history_update" with message TTL (expiration=10s)
- Config: enable_sse_history_updates, history_audit_monitor_poll_interval
Frontend:
- Generalize useNotificationSSE → useSSE composable with event type filtering
- historyStore connects SSE for history_update events, triggers immediate refresh
- notificationsStore updated to use /api/events/stream and useSSE
- Polling kept as fallback at existing intervals
Tests:
- 5 integration tests: endpoint, dataset upload event, history ID in payload,
cross-user isolation, polling backward compatibility
ILIKE '%term%' on unindexed columns is linear in table size, so every raw
term in a whitespace-split search string is one full scan. Terms shorter
than 4 characters ('of', 'and', 'by', ...) match too much to be useful
and, once joined with all the others, dominate query time. Cap raw
terms at 7 and drop any shorter than 4; filtered terms (tag:foo) and
quoted raw terms ('ab') are preserved as explicit user intent.
In a FastAPI request, UrlBuilder's NoMatchFound fallback called
web.url_for without the popped qualified flag, and routes.url_for has
no host available in an ASGI context. The admin resend-activation
endpoint hits this path (since /user/activate is a WSGI-only
@web.expose route) and produced links like http:///user/activate?... .
Latent in the FastAPI endpoint since 72d5cceda8 ("Add mail activation
operation to api"); surfaced for users after c9b849d934 bumped
fastapi 0.135.3 -> 0.136.0.
Qualify the fallback URL with request.base_url when qualified=True.
Integration test posts to /api/users/{user_id}/send_activation_email as
admin with mock SMTP capture, then asserts the captured email body
contains a fully-qualified activation link with a non-empty host. Would
have caught the regression that produced http:///user/activate?... .
Add a `RetrySession` subclass of `Session` that configures a urllib3
`Retry` adapter restricted to idempotent HTTP methods
(`Retry.DEFAULT_ALLOWED_METHODS`). `total`, `backoff_factor`, and any
additional `Retry` keyword arguments can be passed to `RetrySession.__init__()`,
defaulting to 3 retries and a 0.1s backoff factor.
Callers that need retry behaviour on transient failures (e.g. internal
Galaxy-to-Galaxy calls, object store access) can opt in explicitly by
using `RetrySession` instead of plain `Session`.
Also update `url_get()` in `galaxy.util` to use `RetrySession` instead
of manually constructing a `Retry` adapter and mounting it.
- Replace `default_user_agent_decorator` with a cleaner `_request_decorator`
that creates a `Session` instance per call, removing the need for `cast`,
`TypeVar`, and per-request header manipulation.
- Make `Session` a proper subclass of `requests.Session` that sets the
Galaxy user-agent header in `__init__`, so all session-level operations
(including streaming) automatically carry the correct header.
- Fix uses of deprecated `session()` function (replaced with `Session()`)
in `mulled_build_channel.py`, `util.py`, and `container_resolution.py`.
- Slightly refactor `url_get()` to use a context manager for the session.
- Fix type-ignore comment on the `Retry` import in `galaxy.util.__init__`.
- Add tests for user-agent injection and session factory.
Ensures all notification channel settings are respected by verifying
that only enabled channels invoke their plugins. Prevents regressions
where default channel values might be ignored.
guerler