fix: reserve chat hook dispatch capacity for running turns (#27656)

## Context

Follow-up fix from live UAT of the merged chat lifecycle hooks stack
(#27430). Its companion UAT fix (#27655) has merged, so this targets
`main` directly.

## Why?

UAT measured a burst of 1,500 concurrent chat creations against a
consumer with 1.2s latency. 255 were admitted and 1,245 got `502
hook_dispatch_failed (over_capacity)`, which is correct fail-closed
behavior. The collateral wasn't: the same burst failed 24 `stop`
dispatches, parking chats that had already been admitted and had already
executed tools. One 256-slot semaphore served every event, so new-work
admission could take every slot and kill turns in flight.

Callers now classify each dispatch as admission or generation, and
admission draws from a 192-slot gate held *before* the shared pool. At
least 64 shared slots stay reachable only by dispatches for work a chat
already admitted. The dispatcher is per `coderd` replica, so these
limits are per replica, not deployment-wide, and the docs say so.

**The caller classifies, not the event type.** Event type isn't a
reliable proxy in either direction: a subagent spawn dispatches
`user_prompt_submit` from inside a running turn, and the edit path
dispatches `session_start` at admission time. `CapacityClassUnset` is
rejected in `Dispatch`, so a new call site fails closed rather than
silently inheriting a share.

**Acquisition order is load-bearing.** Admission takes its own gate
first. Taking a shared slot first would let admissions queued on the
gate occupy the very capacity the reserve protects. `acquireCapacity` is
the only path that takes either pool, so the order can't be bypassed.

## What this does not guarantee

Nothing bounds how many turns generate concurrently, so the 192/64 split
is a judgement call, not a derived ceiling. This stops an *admission*
burst from consuming every slot; it does not make the remainder
sufficient. A large enough generation load can still exhaust the reserve
and error a running chat. The docs say so explicitly rather than
promising a guarantee the code doesn't deliver.

Generation can now take all 256 slots, so generation traffic starves
admission harder than before. That's the intended priority: rejecting a
new prompt is recoverable, ending a turn that already ran tools is not.

## Testing

Red-green proved both new tests. Removing the release-on-failure path
fails `RefusedSharedAcquireReleasesAdmission` deterministically;
removing the expired-deadline check fails
`ExpiredDeadlineRefusesFreeSlot` in 18/30 runs.

That deadline check fixes a real race found in review. `acquire`
previously shared one `time.Timer` across both acquires. Because
`select` picks a ready case at random, an admission dispatch could take
a slot after its capacity deadline had passed. Measured over 300 trials:
135 late acquisitions, worst overshoot 2.1ms. `acquire` now takes an
absolute deadline and refuses an expired one before selecting, which
measures 0/300.

Go: `coderd/x/agenthooks/...` and `coderd/x/chatd/...`, plus `-race
-count=3` on the dispatcher.

> Mux opened this PR on Mike's behalf.
This commit is contained in:
Michael Suchacz
2026-08-03 19:04:54 +02:00
committed by GitHub
parent 7bd9f5ec93
commit fc24c27dfd
10 changed files with 224 additions and 43 deletions
+7
View File
@@ -167,6 +167,13 @@ Coder checks admission before dispatching, but concurrent requests can still fai
The consumer then observes an event for a request that Coder rejects, and the rejected request doesn't persist a prompt or tool result.
Treat events as attempt notifications rather than proof of a committed operation, and key idempotent tool-event processing on `tool_use_id`.
Each `coderd` replica runs at most 256 dispatches at once and waits up to 250 ms for a free slot; a dispatch that waits out that limit fails as over capacity.
The limit is per replica rather than deployment-wide, so size the consumer for 256 concurrent requests per replica.
Slow consumer responses hold slots for longer, so a slow consumer turns a burst of chat activity into over-capacity failures.
Prompt admission (creating, sending, or editing a message) can hold at most 192 of a replica's slots, so at least 64 stay reachable only by dispatches for work a chat already admitted.
That bound stops a burst of new submissions from consuming every slot, but it doesn't make the remaining slots sufficient: a saturated dispatcher can still fail a dispatch for a running chat and leave that chat in the error state.
Watch `coderd_chatd_hook_dispatches_total{result="over_capacity"}` to see whether the consumer's latency is turning normal traffic into rejections.
Delivery is best-effort and can duplicate.
Coder never queues a failed dispatch for redelivery, so plan for duplicates without assuming every event arrives.
Coder retries one connection failure per dispatch with the same JWT, so use `dispatch_id` to recognize a repeated HTTP attempt and return the same response.