mirror of
https://github.com/coder/coder.git
synced 2026-09-22 05:05:20 +08:00
blockgroot
7a545b35ed
_Disclosure: investigated and drafted with Claude Opus 5. I reviewed the
change and ran the tests locally._
Streaming Responses interceptions recorded no token usage when the
bridge was built with a nil `mcp.ServerProxier`, because
`recordTokenUsage` was called from inside the `i.mcpProxy != nil` branch
in `aibridge/intercept/responses/streaming.go`. Requests completed
normally and returned `200`, so traffic was served but metered as zero,
with no error surfaced. Upstream reports usage on the
`response.completed` event independently of tool injection, so the
proxier is not a valid precondition for recording it.
That state is reachable in production:
`coderd/aibridged/pool.go:255-265` treats proxier construction failure
as non-fatal ("MCP server injection can gracefully degrade") and caches
the resulting bridge via `SetWithTTL`, so one transient config-retrieval
error suppressed usage recording for every streaming Responses request
served by that bridge until its TTL expired.
This records usage for every completed response, guarded only on
`completedResponse`, matching `responses/blocking.go` and both
`chatcompletions` implementations. Per-iteration semantics are preserved
for the inner agentic loop.
The integration harness substituted a non-nil noop manager whenever no
proxier was supplied (`setupbridge.go:153-155`), so the nil path was
never exercised. `withoutMCP()` covers it.
Verification, with the fix reverted:
```
--- FAIL: TestResponsesStreamingRecordsTokenUsageWithoutMCP/without_mcp_proxy
Error: "[]" should have 1 item(s), but has 0
--- PASS: TestResponsesStreamingRecordsTokenUsageWithoutMCP/with_noop_mcp_proxy
```
and with it applied:
```
--- PASS: TestResponsesStreamingRecordsTokenUsageWithoutMCP/without_mcp_proxy
--- PASS: TestResponsesStreamingRecordsTokenUsageWithoutMCP/with_noop_mcp_proxy
--- PASS: TestResponsesStreamingRecordsTokenUsagePerAgenticIteration
```
The per-iteration case asserts exactly two records for the injected-tool
fixture, so decoupling the call from the proxier does not double-count
when the agentic loop iterates. `go test -race ./aibridge/...` passes
across all 15 packages.
Fixes #27885
One caveat on verification: I was unable to run `make gen` / `make
pre-commit` locally, as I do not have the full mise toolchain installed.
The change touches no codegen inputs (no SQL, protos, mocks, or
TypeScript), so I do not expect generated-file drift, but flagging it
rather than leaving it implied.
aibridge
aibridge provides an HTTP handler that intercepts AI client requests bound for upstream AI providers (Anthropic, OpenAI, Copilot). It records token usage, prompts, and tool invocations per user. Optionally supports centralized MCP tool injection with allowlist/denylist filtering.
The handler is mounted by a host process. Today that host is coderd, which mounts the handler at /api/v2/ai-gateway/<provider>/*. Running aibridge as a separate process is planned for the future.
Architecture
┌─────────────────┐ ┌───────────────────────────────────────────┐
│ AI Client │ │ aibridge │
│ (Claude Code, │────▶│ ┌─────────────────┐ ┌─────────────┐ │
│ Cursor, etc.) │ │ │ RequestBridge │───▶│ Providers │ │
└─────────────────┘ │ │ (http.Handler) │ │ (Anthropic │ │
│ └─────────────────┘ │ OpenAI) │ │
│ └──────┬──────┘ │
│ │ │
│ ▼ │ ┌─────────────┐
│ ┌─────────────────┐ ┌─────────────┐ │ │ Upstream │
│ │ Recorder │◀───│ Interceptor │─── ───▶│ API │
│ │ (tokens, tools, │ │ (streaming/ │ │ │ (Anthropic │
│ │ prompts) │ │ blocking) │ │ │ OpenAI) │
│ └────────┬────────┘ └──────┬──────┘ │ └─────────────┘
│ │ │ │
│ ▼ ┌──────▼──────┐ │
│ ┌ ─ ─ ─ ─ ─ ─ ─ ┐ │ MCP Proxy │ │
│ │ Database │ │ (tools) │ │
│ └ ─ ─ ─ ─ ─ ─ ─ ┘ └─────────────┘ │
└───────────────────────────────────────────┘
Components
- RequestBridge: The main
http.Handlerthat routes requests to providers - Provider: Defines bridged routes (intercepted) and passthrough routes (proxied)
- Interceptor: Handles request/response processing and streaming
- Recorder: Interface for capturing usage data (tokens, prompts, tools)
- MCP Proxy (optional): Connects to MCP servers to list tool, inject them into requests, and invoke them in an inner agentic loop
Request Flow
- Client sends request to
/anthropic/v1/messagesor/openai/v1/chat/completions - Actor extraction: Request must have an actor in context (via
AsActor()). The host is responsible for authenticating the caller before invoking the handler. - Upstream call: Request forwarded to the AI provider
- Response relay: Response streamed/sent to client
- Recording: Token usage, prompts, and tool invocations recorded
With MCP enabled: Tools from configured MCP servers are centrally defined and injected into requests (prefixed bmcp_). Allowlist/denylist regex patterns control which tools are available. When the model selects an injected tool, the gateway invokes it in an inner agentic loop, and continues the conversation loop until complete.
Passthrough routes (/v1/models, /v1/messages/count_tokens) are reverse-proxied directly.
Observability
Prometheus Metrics
Create metrics with NewMetrics(prometheus.Registerer):
| Metric | Type | Description |
|---|---|---|
interceptions_total |
Counter | Intercepted request count |
interceptions_inflight |
Gauge | Currently processing requests |
interceptions_duration_seconds |
Histogram | Request duration |
passthrough_total |
Counter | Non-intercepted requests forwarded to the upstream |
prompts_total |
Counter | User prompt count |
tokens_total |
Counter | Token usage (input, output, cache read/write, provider extras) |
injected_tool_invocations_total |
Counter | Injected MCP tool invocations performed by the handler |
non_injected_tool_selections_total |
Counter | Client-defined tool selections returned by the model |
circuit_breaker_state |
Gauge | Circuit breaker state per provider/endpoint (0=closed, 0.5=half, 1=open) |
circuit_breaker_trips_total |
Counter | Times the circuit breaker transitioned to open |
circuit_breaker_rejects_total |
Counter | Requests rejected due to an open circuit breaker |
Recorder Interface
Implement Recorder to persist usage data to your database:
aibridge_interceptions- request metadata (provider, model, initiator, timestamps)aibridge_token_usages- input/output and cache read/write token counts per responseaibridge_user_prompts- user promptsaibridge_tool_usages- tool invocations (injected and client-defined)aibridge_model_thoughts- model reasoning content (thinking, reasoning summaries, commentary)
type Recorder interface {
RecordInterception(ctx context.Context, req *InterceptionRecord) error
RecordInterceptionEnded(ctx context.Context, req *InterceptionRecordEnded) error
RecordTokenUsage(ctx context.Context, req *TokenUsageRecord) error
RecordPromptUsage(ctx context.Context, req *PromptUsageRecord) error
RecordToolUsage(ctx context.Context, req *ToolUsageRecord) error
RecordModelThought(ctx context.Context, req *ModelThoughtRecord) error
}
Supported Routes
Each provider instance is mounted under /api/v2/ai-gateway/<name>, where <name> is the provider's configured name. For example, with an Anthropic provider named my-anthropic, its /messages endpoint would be reachable at /api/v2/ai-gateway/my-anthropic/v1/messages.
If a name is not set, the route path defaults to the provider's type: anthropic, openai, or copilot. The table below uses the default names.
(/*) denotes a route that handles both the exact path and any subpaths. A trailing /* denotes subpaths only.
| Provider | Route | Type |
|---|---|---|
| Anthropic | /anthropic/v1/messages |
Bridged (intercepted) |
| Anthropic | /anthropic/v1/messages/count_tokens |
Passthrough |
| Anthropic | /anthropic/v1/models(/*) |
Passthrough |
| Anthropic | /anthropic/api/event_logging/* |
Passthrough |
| OpenAI | /openai/v1/chat/completions |
Bridged (intercepted) |
| OpenAI | /openai/v1/responses |
Bridged (intercepted) |
| OpenAI | /openai/v1/responses/* |
Passthrough |
| OpenAI | /openai/v1/conversations(/*) |
Passthrough |
| OpenAI | /openai/v1/models(/*) |
Passthrough |
| Copilot | /copilot/chat/completions |
Bridged (intercepted) |
| Copilot | /copilot/responses |
Bridged (intercepted) |
| Copilot | /copilot/models(/*) |
Passthrough |
| Copilot | /copilot/agents/* |
Passthrough |
| Copilot | /copilot/mcp/* |
Passthrough |
| Copilot | /copilot/.well-known/* |
Passthrough |