## Problem
CODAGT-906: chats using OpenAI-compatible backends (e.g. poolside)
through the AI Bridge persist token usage inflated 105x-640x, which
falsely triggers automatic chat compaction on every turn.
The chat-completions streaming interceptor summed usage across every SSE
chunk of one upstream stream and rewrote each relayed usage-bearing
chunk with that running sum. Spec-compliant OpenAI emits usage once
(final chunk with `stream_options.include_usage`), so the sum equals the
final value. vLLM-style backends emit cumulative usage snapshots on
every chunk, so the relayed final usage becomes roughly `N_chunks x
prompt_tokens` (e.g. 417,012 persisted for a ~6,000-token context).
chatd persists that value per assistant message and its compaction
trigger reads it as context occupancy.
## Fix
Track the latest usage-bearing chunk's raw usage (last-wins) in the
stream processor, updating only when a chunk actually carries usage so a
trailing usage-less chunk cannot zero it. `marshalChunk` relays that
value and `recordTokenUsage` records the same value, unifying relayed
and recorded usage. Last-wins is correct for both shapes: a single final
usage chunk, and cumulative snapshots where each snapshot already
includes all prior tokens.
Per-iteration semantics are unchanged: each tool-loop iteration has its
own processor, and the final iteration's usage is what the client sees.
## Tests
- `TestStreamProcessorUsage` (internal): cumulative snapshots with a
trailing usage-less chunk, and the spec-compliant final-only shape;
asserts relayed and recorded usage equal the last snapshot.
- New txtar fixture `streaming_cumulative_usage_injected_tool.txtar`
with per-chunk cumulative usage plus an injected tool call; asserts
client-visible final usage through the full interceptor.
- Red-green verified: with the fix reverted, the internal test reports
zeroed usage (trailing chunk overwrite) and the fixture test reports
18000 summed prompt tokens instead of 6000.
The blocking (non-streaming) path deliberately keeps its cross-iteration
summation for external clients and is untouched. Remote dogfood UAT
validated chat streaming, tool calls, plausible usage numbers, and zero
spurious compactions.
> Mux acted on Mike's behalf to author this change.
## Summary
The Responses interceptor previously recorded only `function_call` and `custom_tool_call` output items, so interceptions that did real work via built-in tools (`web_search_call`, `computer_call`, `shell_call`, `mcp_call`, etc.) recorded no tool usage at all.
`recordNonInjectedToolUsage` now whitelists every tool-call output type and records it, with the tool name falling back to the item type when none is set.
`ToolUsageRecord` also gains an `ItemID` field so the two distinct Responses identifiers are captured without conflation (addresses review feedback on coder/aibridge#273):
- `ItemID`: the output item's unique `id` (always present).
- `ToolCallID`: the `call_id` correlation id (empty for hosted tools the provider runs server-side).
## Tests
- Extends `TestRecordToolUsage` with cases for the new hosted/agentic tool types.
- Adds blocking and streaming `web_search` fixtures (scrubbed of credentials and identifying metadata) plus `TestResponsesOutputMatchesUpstream` cases asserting a hosted tool records with an empty `ToolCallID` and a populated `ItemID`.
Linear: AIGOV-96
---
*This PR was produced by opencode (agent) using the `anthropic/claude-opus-4-8` model, under human direction and review.*
The agentic loop has a race between the main goroutine and the `Start`
goroutine on the shared `ResponseWriter`. When an iteration's response
contains only injected-tool events (no text to relay), `Start` may not
have called `InitiateStream` by the time main reaches the `IsStreaming`
check on the next iteration. The `IsStreaming` check then returns false,
main writes a JSON error via `writeUpstreamError`, and `Start` later
writes SSE headers and events on top, producing a malformed JSON+SSE
response:
```
{\"error\":{\"message\":\"all configured keys are rate-limited\",\"type\":\"rate_limit_error\"},\"request_id\":\"\",\"type\":\"error\"}event: message_start\n..."
```
Fix: explicitly call `events.InitiateStream(w)` at the agentic
continuation point so the SSE stream is committed before the next
iteration runs. Keeps `messages` consistent with the pattern already
used in `chatcompletions/streaming.go`. `sync.Once` makes the double
call safe.
Related: coder/internal#1524
Related: coder/coder#25654
Closes:
https://linear.app/codercom/issue/AIGOV-336/flake-teststreaminginterception-agenticloopfailoveragentic-all-keys
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by
@ssncferreira
## Description
Adds automatic key failover for centralized OpenAI provider, covering both chat completions and responses APIs. Same shape as the Anthropic PR: each upstream call walks the configured key pool, keys are marked **temporary** on 429 (with cooldown from `Retry-After`) and **permanent** on 401/403. Each agentic-loop iteration gets its own fresh walker so a tool-call continuation can fail over independently of the initial request.
BYOK is unchanged: BYOK requests run as a single attempt with no failover.
## Changes
- `config.OpenAI` carries a `KeyPool`. `Key` remains for BYOK Authorization Bearer set per interception.
- Chat completions blocking interceptor: walks the pool via `newChatCompletionWithKeyFailover`, marks keys on key-specific failures, returns on first success or non-failover error.
- Chat completions streaming interceptor: per-iteration walker. Pre-stream failures fail over to the next key; mid-stream errors are relayed as SSE events.
- Responses blocking interceptor: extracts `newResponseWithKeyFailover` parallel to chatcompletions.
- Responses streaming interceptor: per-iteration walker, retains the existing buffer-then-forward design.
## Related Issues
Related to: https://github.com/coder/internal/issues/1446
Related to: https://linear.app/codercom/issue/AIGOV-197/aibridge-automatic-key-failover-for-bridged-and-passthrough-routes
## Follow-up PRs
- Bedrock multi-key support.
- Refactor provider vs interceptor config separation.
- Record the actually-used key in the interception credential hint after failover.
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by @ssncferreira
This PR merges code from `coder/aibridge` repository into `coder/coder`.
It was split into 4 PRs for easier review but stacked PRs will need to
be merged into this PR so all checks pass.
* https://github.com/coder/coder/pull/24190 -> raw code copy (this PR,
before merging PRs on top of it, it was just 1 commit:
https://github.com/coder/coder/commit/70d33f33200c7e77df910957595715f81f9bec24)
* https://github.com/coder/coder/pull/24570 -> update imports in
`coder/coder` to use copied code
* https://github.com/coder/coder/pull/24586 -> linter fixes and CI
integration (also added README.md)
* https://github.com/coder/coder/pull/24571 -> added exclude to
scripts/check_emdash.sh check
Original PR message (before PR squash):
Moves coder/aibridge code into coder/coder repository.
Omitted files:
- `go.mod`, `go.sum`, `.gitignore`, `.github/workflows/ci.yml,`
`Makefile`, `LICENSE`, `README.md` (modified README.md is added later)
- `.github`, `example`, `buildinfo,` `scripts` directories
Simple verification script (will list omitted files)
```
tmp=$(mktemp -d)
echo "$tmp"
git clone --depth=1 https://github.com/coder/aibridge "$tmp/aibridge"
git clone --depth=1 --branch pb/aibridge-code-move https://github.com/coder/coder "$tmp/coder"
diff -rq --exclude=.git "$tmp/aibridge" "$tmp/coder/aibridge"
# rm -rf "$tmp"
```
Michael Suchacz