mirror of
https://github.com/coder/coder.git
synced 2026-10-31 11:02:50 +08:00
Michael Suchacz
8eb1e43f0b
## Problem
Selecting a reasoning effort on a Google (Gemini) chat model either got
dropped or broke the request outright, and even when the request
succeeded, Gemini chats never showed thinking blocks:
- The native `fantasygoogle` path had no case in `applyReasoningEffort`
at all, so effort was silently ignored. With only `include_thoughts`
configured, Gemini 3+ models default to no thinking, so chats produced
zero reasoning tokens.
- The path chat generation actually uses today routes Google providers
through aibridge as openai-compat clients and forwarded the raw effort
string as `reasoning_effort`. Google's OpenAI-compatible endpoint
validates that value against the model's supported thinking levels
instead of clamping, so out-of-range efforts fail the whole request with
HTTP 400. Verified against the live Gemini API: `gemini-3.1-pro-preview`
accepts low/medium/high and rejects none/minimal/xhigh/max. Because the
config's default effort is sent whenever the user selects nothing, a
config whose default falls outside the model's supported set failed
every no-selection message.
- Google's OpenAI-compatible endpoint never emits thought text for
`reasoning_effort` requests. Thoughts require
`extra_body.google.thinking_config` with `include_thoughts`, which is
mutually exclusive with `reasoning_effort`, and even then the thought
deltas come back inline in `content` (marked
`extra_content.google.thought`, wrapped in `<thought>` markers) rather
than in the `reasoning_content` field OpenAI-compatible clients parse.
So no reasoning parts were ever produced, persisted, or rendered for
Gemini chats (verified live in UAT).
## Fix
Reasoning effort:
- Map the global reasoning effort scale onto Google's `thinking_level`
in `applyReasoningEffort`, gated to Gemini 3+ models and clamped to each
model's supported subset (3.0 Pro launched with LOW/HIGH only, 3.1 Pro
added MEDIUM, Flash accepts all four through 3.6 while 3.7 and the
latest flash aliases drop MINIMAL, image models accept HIGH plus MINIMAL
for flash). Pre-Gemini-3 models reject `thinking_level`, so effort
degrades to a no-op for them and a config-pinned level is dropped at
call time instead of failing every generation.
- Add `thinking_level` to `ChatModelGoogleThinkingConfig` so admins can
pin a level for configs without user-selectable effort. A resolved
per-turn effort overrides the pinned level; an explicit
`thinking_budget` wins over both since the Google API rejects requests
that set `thinking_budget` and `thinking_level` together, and config
validation now rejects that combination up front. The admin panel picks
up the new field automatically through the generated model-options
schema.
Thinking blocks on the openai-compat path (the path chat generation
uses), translated in both directions at the existing Gemini transport
seam:
- chatd's OpenAI-compat request patch swaps `reasoning_effort` for an
equivalent `extra_body.google.thinking_config` with `include_thoughts`
enabled: `thinking_level` on Gemini 3+ (clamped to the model's supported
subset), `thinking_budget` on earlier models (effort `none` maps to
budget 0, which disables thinking). Non-Gemini models and non-Google
routes are untouched.
- The same transport rewrites Google's responses so thought output
surfaces as `reasoning_content`, which fantasy already converts into
reasoning parts: streaming bodies are rewritten per SSE line (preserving
streaming latency), JSON bodies in place, with the
`<thought>`/`</thought>` markers stripped at the transitions.
- aibridge's chat-completions interception preserves the `extra_body`
passthrough that openai-go's typed params silently drop, forwarding it
to Google upstreams only, so the chatd -> aibridge -> Google route
carries the thinking config end to end.
## Validation
- `go test ./coderd/x/chatd/... ./aibridge/... ./internal/googleopenai
./codersdk` pass; golangci-lint clean on touched packages.
- New unit tests cover the effort-to-level mapping for every effort
value, per-model clamping on both paths, the effort-to-thinking_config
request swap (level, budget, pinned-config precedence, non-Gemini
passthrough), SSE and JSON thought-response rewriting (fixtures mirror
live captures, including tool-call and no-thought streams), an
end-to-end fake-SSE test through the fantasy pipeline proving reasoning
deltas emerge with markers stripped, and aibridge `extra_body`
preservation (including proof the typed round trip drops it).
- Verified live against `generativelanguage.googleapis.com` on
`gemini-3-flash-preview` and `gemini-2.5-flash` through the real
chatprovider stack: reasoning deltas stream through with markers
stripped and answer text intact.
- Live UAT on a dogfood deployment (remote Coder Agents chat, 4 rounds):
effort clamping verified end to end (rounds 1-2), the missing-thoughts
limitation isolated (round 3), and the fix validated (round 4):
expandable, readable thinking blocks render for
`gemini-3.1-pro-preview`, `gemini-3-flash-preview`, and
`gemini-2.5-flash` at effort=high; no `<thought>` marker leaks;
effort=none produces no block and no 400; multi-turn history replay
works with a thinking block on each turn.
- `make gen` regenerated `typesGenerated.ts` and
`chatModelOptionsGenerated.json`; `pnpm -C site lint:types` passes.
> 🤖 Mux acted on Mike's behalf to create this pull request.
<!-- mux-attribution: model=claude-opus-4-6 thinking=high -->
aibridge
aibridge provides an HTTP handler that intercepts AI client requests bound for upstream AI providers (Anthropic, OpenAI, Copilot). It records token usage, prompts, and tool invocations per user. Optionally supports centralized MCP tool injection with allowlist/denylist filtering.
The handler is mounted by a host process. Today that host is coderd, which mounts the handler at /api/v2/ai-gateway/<provider>/*. Running aibridge as a separate process is planned for the future.
Architecture
┌─────────────────┐ ┌───────────────────────────────────────────┐
│ AI Client │ │ aibridge │
│ (Claude Code, │────▶│ ┌─────────────────┐ ┌─────────────┐ │
│ Cursor, etc.) │ │ │ RequestBridge │───▶│ Providers │ │
└─────────────────┘ │ │ (http.Handler) │ │ (Anthropic │ │
│ └─────────────────┘ │ OpenAI) │ │
│ └──────┬──────┘ │
│ │ │
│ ▼ │ ┌─────────────┐
│ ┌─────────────────┐ ┌─────────────┐ │ │ Upstream │
│ │ Recorder │◀───│ Interceptor │─── ───▶│ API │
│ │ (tokens, tools, │ │ (streaming/ │ │ │ (Anthropic │
│ │ prompts) │ │ blocking) │ │ │ OpenAI) │
│ └────────┬────────┘ └──────┬──────┘ │ └─────────────┘
│ │ │ │
│ ▼ ┌──────▼──────┐ │
│ ┌ ─ ─ ─ ─ ─ ─ ─ ┐ │ MCP Proxy │ │
│ │ Database │ │ (tools) │ │
│ └ ─ ─ ─ ─ ─ ─ ─ ┘ └─────────────┘ │
└───────────────────────────────────────────┘
Components
- RequestBridge: The main
http.Handlerthat routes requests to providers - Provider: Defines bridged routes (intercepted) and passthrough routes (proxied)
- Interceptor: Handles request/response processing and streaming
- Recorder: Interface for capturing usage data (tokens, prompts, tools)
- MCP Proxy (optional): Connects to MCP servers to list tool, inject them into requests, and invoke them in an inner agentic loop
Request Flow
- Client sends request to
/anthropic/v1/messagesor/openai/v1/chat/completions - Actor extraction: Request must have an actor in context (via
AsActor()). The host is responsible for authenticating the caller before invoking the handler. - Upstream call: Request forwarded to the AI provider
- Response relay: Response streamed/sent to client
- Recording: Token usage, prompts, and tool invocations recorded
With MCP enabled: Tools from configured MCP servers are centrally defined and injected into requests (prefixed bmcp_). Allowlist/denylist regex patterns control which tools are available. When the model selects an injected tool, the gateway invokes it in an inner agentic loop, and continues the conversation loop until complete.
Passthrough routes (/v1/models, /v1/messages/count_tokens) are reverse-proxied directly.
Observability
Prometheus Metrics
Create metrics with NewMetrics(prometheus.Registerer):
| Metric | Type | Description |
|---|---|---|
interceptions_total |
Counter | Intercepted request count |
interceptions_inflight |
Gauge | Currently processing requests |
interceptions_duration_seconds |
Histogram | Request duration |
passthrough_total |
Counter | Non-intercepted requests forwarded to the upstream |
prompts_total |
Counter | User prompt count |
tokens_total |
Counter | Token usage (input, output, cache read/write, provider extras) |
injected_tool_invocations_total |
Counter | Injected MCP tool invocations performed by the handler |
non_injected_tool_selections_total |
Counter | Client-defined tool selections returned by the model |
circuit_breaker_state |
Gauge | Circuit breaker state per provider/endpoint (0=closed, 0.5=half, 1=open) |
circuit_breaker_trips_total |
Counter | Times the circuit breaker transitioned to open |
circuit_breaker_rejects_total |
Counter | Requests rejected due to an open circuit breaker |
Recorder Interface
Implement Recorder to persist usage data to your database:
aibridge_interceptions- request metadata (provider, model, initiator, timestamps)aibridge_token_usages- input/output and cache read/write token counts per responseaibridge_user_prompts- user promptsaibridge_tool_usages- tool invocations (injected and client-defined)aibridge_model_thoughts- model reasoning content (thinking, reasoning summaries, commentary)
type Recorder interface {
RecordInterception(ctx context.Context, req *InterceptionRecord) error
RecordInterceptionEnded(ctx context.Context, req *InterceptionRecordEnded) error
RecordTokenUsage(ctx context.Context, req *TokenUsageRecord) error
RecordPromptUsage(ctx context.Context, req *PromptUsageRecord) error
RecordToolUsage(ctx context.Context, req *ToolUsageRecord) error
RecordModelThought(ctx context.Context, req *ModelThoughtRecord) error
}
Supported Routes
Each provider instance is mounted under /api/v2/ai-gateway/<name>, where <name> is the provider's configured name. For example, with an Anthropic provider named my-anthropic, its /messages endpoint would be reachable at /api/v2/ai-gateway/my-anthropic/v1/messages.
If a name is not set, the route path defaults to the provider's type: anthropic, openai, or copilot. The table below uses the default names.
(/*) denotes a route that handles both the exact path and any subpaths. A trailing /* denotes subpaths only.
| Provider | Route | Type |
|---|---|---|
| Anthropic | /anthropic/v1/messages |
Bridged (intercepted) |
| Anthropic | /anthropic/v1/messages/count_tokens |
Passthrough |
| Anthropic | /anthropic/v1/models(/*) |
Passthrough |
| Anthropic | /anthropic/api/event_logging/* |
Passthrough |
| OpenAI | /openai/v1/chat/completions |
Bridged (intercepted) |
| OpenAI | /openai/v1/responses |
Bridged (intercepted) |
| OpenAI | /openai/v1/responses/* |
Passthrough |
| OpenAI | /openai/v1/conversations(/*) |
Passthrough |
| OpenAI | /openai/v1/models(/*) |
Passthrough |
| Copilot | /copilot/chat/completions |
Bridged (intercepted) |
| Copilot | /copilot/responses |
Bridged (intercepted) |
| Copilot | /copilot/models(/*) |
Passthrough |
| Copilot | /copilot/agents/* |
Passthrough |
| Copilot | /copilot/mcp/* |
Passthrough |
| Copilot | /copilot/.well-known/* |
Passthrough |