Limits concurrent chat generation on capped deployments to 5 root chats
and 10 delegated subagent chats. The pools are deployment-wide and
independent, so delegated work can continue while root capacity is full.
The default caps live in AGPL code. Enterprise contributes only a
licensing unlock, so unlicensed deployments stay capped and cannot fail
open. Licensed deployments are uncapped while Agent Hours usage stays
below an explicit hard limit. Deployments without a hard limit remain
uncapped, and reaching the Agent Hours allocation only triggers
warnings.
Admission happens before a worker takes chat ownership. Capped
deployments serialize admission across replicas with a
transaction-scoped advisory lock and derive active and queued state from
current ownership plus fresh runner heartbeats, rather than persisted
queue markers or per-replica state. The acquisition query returns a
bounded, pool-interleaved candidate set instead of ranking the whole
backlog; a migration replaces the acquisition index with a pool-aware
one. Refused chats stay running but unowned, and interrupt requests
bypass admission so users can stop queued or over-cap chats.
The single-chat API derives `queued_for_capacity` from live pool state;
list endpoints do not report it. The UI polls that value every 5 seconds
while a chat is running and shows a callout when the chat is waiting for
capacity.
Updates the administrator documentation and deployment-wide Prometheus
gauges for active and queued agents. Replica-level values must be
aggregated with `max`, not `sum`.
> Mux updated this PR on Mike's behalf.
Fixescoder/internal#1519
Fixes CODAGT-353
These nine tests were skipped pending a chatd notification flow refactor
that would let workers distinguish stale control `NOTIFY` messages from
real interrupts. They now pass consistently, so this drops the `t.Skip`
calls and the now-stale `TODO(CODAGT-353)` blocks.
While rerunning the package after unskipping them,
`TestNewReplicaRecoversStaleChatFromDeadReplica` also surfaced as flaky
on `main` because it asserted transient ownership state. This PR keeps
that server-level test as a stable end-to-end recovery check and adds a
deterministic worker-level stale reacquisition test so we still directly
cover lease takeover behavior.
## Tests unskipped
- `coderd`: `TestPatchChatMessage/ChangesModel`
- `coderd/x/chatd`:
- `TestExploreChatSendMessageCannotMutateMCPSnapshot`
- `TestAutoPromoteQueuedMessagesPreservesPerTurnModelOrder`
- `TestSignalWakeSendMessage`
- `TestAdvisorChainMode_SnapshotKeepsFullHistory`
- `TestOpenAIResponsesNoStaleWebSearchReplay`
- `TestOpenAIResponsesFullReplayPairsReasoningAndWebSearch`
- `TestOpenAIResponsesChainModeSkipsWhenLocalCallPending`
- `TestOpenAIResponsesChainModeStillFiresForProviderExecutedOnly`
## Stale recovery follow-up
- `coderd/x/chatd`: `TestNewReplicaRecoversStaleChatFromDeadReplica` now
waits for the stable `waiting` and unowned end state after recovery.
- `coderd/x/chatd`: `TestWorker_ReacquiresStaleOwnedChat` blocks the
runner after reacquisition and directly asserts the new worker
ownership, new runner ID, and fresh heartbeat.
I stress-ran the stale recovery tests locally with repeated plain and
race runs, and re-ran the nine unskipped tests plus
`TestPatchChatMessage/ChangesModel` after these follow-up changes.
Michael Suchacz