Implements
https://linear.app/codercom/issue/AIGOV-286/add-interception-cost-calculation-to-aibridge-token-usages
Adds spend attribution to AI Gateway. After the upstream response, each
token-usage record now captures the user's effective group, the
per-token prices in effect at that moment, and a computed cost — so
spend is recorded as an immutable, point-in-time snapshot.
Concretely, `aibridge_token_usages` gains `effective_group_id`,
`input_price_micros`, `output_price_micros`, `cache_read_price_micros`,
`cache_write_price_micros`, and `cost_micros`. When a usage record is
written, the effective group is resolved (per-user override, else the
deployment budget policy), the `(provider, model)` price is looked up
and snapshotted onto the row, and cost is computed from the
provider-reported token counts. A model that isn't in the price table
records its tokens with a `NULL` cost; any *other* resolution failure
fails the write, so a `NULL` cost unambiguously means "model not priced"
rather than "lookup errored."
All values are stored in micro-units (1 unit = 1,000,000 micro-units;
Phase 1 assumes USD, so 1 micro-unit = $0.000001). Prices are quoted per
million tokens.
This also grants the AI Bridge RBAC subject `read` on `ai_model_prices`
(the per-interception price lookup needs it; it previously only had
`update` for the startup seeder).
## Cost precision
Cost is computed per token category as `tokens × price / 1_000_000` with
integer division, then the four categories are summed. The division is
done **per category** (not once over the summed numerator) on purpose:
it keeps the per-category line items summing exactly to the stored total
— no "the parts don't add up to the whole" in reporting).
Integer division truncates sub-micro-unit fractions. For example, a
cheap model at $0.10 per million tokens is a price of `100_000`; 9
tokens cost `9 × 100_000 / 1_000_000 = 900_000 / 1_000_000 = 0` (the
true 0.9 micro-units floors to 0). At real list prices this rarely bites
— $3/M input is a price of `3_000_000`, so even a single token is 3
micro-units. The per-record under-count is bounded below 1 micro-unit
per category, so under $0.000004 total across the four categories, which
is acceptable for list-price-based cost approximation.
## Overflow safety
`cost_micros` is a `BIGINT` (int64), and the largest intermediate value
is a single category's `tokens × price` before division. int64's ceiling
is ≈ `9.223e18`.
- At a steep $75/M model (price `75_000_000`), overflow would require
~123 billion tokens in one response: `123e9 × 75e6 = 9.225e18`, just
over the limit. `122e9` stays under at `9.15e18`.
- A realistically maxed-out Opus 4.8 response (≈1M input + 128K output
at list prices) costs about $15, with a numerator around `1.5e13` —
roughly six orders of magnitude below the ceiling.
So overflow is unreachable from real token counts.
### Multi-currency support
In the future, we may encounter issues with multi-currency support,
especially when dealing with currencies that have very large exchange
rates relative to USD, for example:
IRR: ~1,300,000 IRR ≈ 1 USD
VND: ~26,000 VND ≈ 1 USD
For currencies with such large denominations, numeric overflow is
technically possible, considering that we have only about six orders of
magnitude of headroom before reaching the limit (see above).
## `effective_group_id` has no foreign key
`effective_group_id` records the group a spend was attributed to, as an
immutable historical fact. It is intentionally **not** a foreign key, so
the record survives deletion of the group.
Alternatives were considered and rejected:
- **`ON DELETE SET NULL`** would mutate an "immutable" record — deleting
a group silently erases that interception's attribution and under-counts
the group's historical spend.
- **`RESTRICT` / `NO ACTION`** would block group deletion entirely
(groups are hard-deleted).
- **`CASCADE`** would delete spend history when a group is deleted — the
worst outcome for an audit record.
There is also no insert-time check that the group still exists: the id
comes from a budget that was just resolved, meaning it was valid at some
point.
## Open question: group name snapshotting
Should we also snapshot the group *name* onto each record? Two options:
- **Denormalize it now** — readable in historical reports even after a
group is deleted, but the snapshot can drift from the current name on
rename, raising a "show point-in-time vs. current name" question.
- **Postpone until needed** — it's a purely additive column later, and
the name is display-only (not correctness-bearing like the price). The
cost: names of groups deleted before the column is added can't be
backfilled.
Leaning toward postponing until a concrete reporting need settles the
drift question.
Closes
https://linear.app/codercom/issue/AIGOV-287/add-effective-group-resolution
Implements the effective AI budget resolution from the AI Governance
cost-controls RFC: for a given user, a `user_ai_budget_overrides` row
wins if present, otherwise the deployment budget policy (`highest`)
picks the largest group budget across the user's groups, ties broken
alphabetically.
For now, I keep the logic under `coderd/aibridge/budget`, but that may
change during the implementation of budget enforcement.
Allows an `api_key_id` to be passed from a trusted in-memory transport
(currently: `chatd`) to `aibridged` for use in authenticating LLM
requests.
This value can _only_ be passed via context, and all users of the
in-memory transport _must_ provide it.
It can be used in conjunction with BYOK headers.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
### TL;DR
Introduces an in-process `TransportFactory` for aibridge so that chatd (coder-agent LLM traffic) can route requests through the aibridged handler without crossing the HTTP route or requiring a license entitlement check.
### What changed?
- Added a new `coderd/aibridge` package with a `TransportFactory` interface and a `Source` type for tagging the call site on request contexts. `SourceAgents` is defined as the constant for coder-agent traffic.
- Implemented `NewTransportFactory` in `coderd/aibridged/transport.go`, which returns an `http.RoundTripper` that dispatches requests to the aibridged handler in-process. The response body is streamed through an `io.Pipe` so SSE/NDJSON/chunked responses propagate token-by-token. Handler panics are recovered and surfaced as 500 responses, and context cancellation closes the pipe with the appropriate error.
- `RegisterInMemoryAIBridgedHTTPHandler` now also constructs a `TransportFactory` from the registered handler and stores it on `API.AIBridgeTransportFactory` (an `atomic.Pointer`), making it available to chatd without going through the license-gated HTTP route.
- Added `API.AIBridgeTransportFactory` as a public `atomic.Pointer[aibridge.TransportFactory]` field on `coderd.API`.
### How to test?
- `coderd/aibridged/transport_test.go` covers: transport creation, nil-handler errors, source attachment to context, header/status passthrough, streaming (SSE-style chunked writes visible before handler completion), context cancellation closing the body with an error, concurrent requests, handler panics producing 500s, and handlers that return without writing.
- `coderd/aibridge_test.go` verifies that `AIBridgeTransportFactory` starts as nil on AGPL coderd, can be stored and loaded atomically, and that the stored factory correctly dispatches requests through the stub handler.
### Why make this change?
Chatd needs to send LLM requests through aibridge in-process rather than via the external HTTP route, which is license-gated. The `TransportFactory` abstraction provides a clean seam: the entitlement check remains on the HTTP route for external callers, while in-process coder-agent traffic bypasses it through the factory. The `Source` type allows downstream handlers and logs to attribute traffic without gating behavior on the caller identity.
# Summary
Implements
https://linear.app/codercom/issue/AIGOV-282/add-ai-model-price-table-and-seed-generator
This PR lays the groundwork for AI Bridge cost controls (per the AI
Governance RFC). It adds the foundation needed for future cost tracking:
a place to store per-model token prices, a way to keep those prices in
sync with upstream pricing data, and a startup mechanism that ensures
every deployment has prices loaded before AI Bridge starts processing
requests.
The price data comes from [models.dev](https://models.dev/), a
community-maintained catalogue of AI provider pricing. A generator
script fetches the latest prices, filters to Anthropic and OpenAI for
now, and produces a seed file checked into the repository.
On every server startup the seed is applied to the database, so new
releases automatically pick up any price corrections that landed since
the previous one. Existing rows are overwritten with the latest prices;
rows for models no longer in the seed are left untouched.
# Batching the AI model price seed: three approaches
Context: at server startup we seed the `ai_model_prices` table from an
embedded JSON price book (~70 rows today, will grow as we add providers,
potentially 4000+).
Each row is:
```text
(provider, model, input_price, output_price, cache_read_price, cache_write_price)
```
Any of the four price columns can be:
- `NULL` → “price unknown for this dimension”
- explicit `0` → “free”
The batch must be an UPSERT so re-running is idempotent and existing
rows pick up new prices.
We considered three implementations.
---
## Approach 1 — Per-row UPSERT in a Go loop
```go
for _, row := range rows {
if err := db.UpsertAIModelPrice(ctx, database.UpsertAIModelPriceParams{
Provider: row.Provider,
Model: row.Model,
InputPrice: nullInt64(row.InputPrice),
// ...
}); err != nil {
return err
}
}
```
### Pros
- Trivial.
- NULL handling falls out naturally from `sql.NullInt64`.
### Cons
- `N` round-trips per seed.
- With ~70 rows that means ~70 statement executions on every startup,
even inside a transaction.
- Doesn't scale gracefully as the price book grows, potentially 4000+.
---
## Approach 2 — `UNNEST` with parallel arrays
Pass each column as a separate Go slice. Postgres unnests them in
parallel into a virtual table, then `INSERT ... SELECT`.
```sql
INSERT INTO ai_model_prices (
provider,
model,
input_price,
output_price,
cache_read_price,
cache_write_price
)
SELECT
UNNEST(@providers::text[]),
UNNEST(@models::text[]),
NULLIF(UNNEST(@input_prices::bigint[]), -1),
NULLIF(UNNEST(@output_prices::bigint[]), -1),
NULLIF(UNNEST(@cache_read_prices::bigint[]), -1),
NULLIF(UNNEST(@cache_write_prices::bigint[]), -1)
ON CONFLICT (provider, model) DO UPDATE SET
input_price = EXCLUDED.input_price,
output_price = EXCLUDED.output_price,
cache_read_price = EXCLUDED.cache_read_price,
cache_write_price = EXCLUDED.cache_write_price,
updated_at = NOW();
```
Go side: flatten rows into six parallel slices.
Use a sentinel (`-1`) for “missing”, since `lib/pq` can't encode `NULL`
into a `bigint[]` element.
```go
providers := make([]string, len(rows))
models := make([]string, len(rows))
inputs := make([]int64, len(rows))
outputs := make([]int64, len(rows))
cacheR := make([]int64, len(rows))
cacheW := make([]int64, len(rows))
for i, r := range rows {
providers[i] = r.Provider
models[i] = r.Model
inputs[i] = -1
if r.InputPrice != nil {
inputs[i] = *r.InputPrice
}
outputs[i] = -1
if r.OutputPrice != nil {
outputs[i] = *r.OutputPrice
}
cacheR[i] = -1
if r.CacheReadPrice != nil {
cacheR[i] = *r.CacheReadPrice
}
cacheW[i] = -1
if r.CacheWritePrice != nil {
cacheW[i] = *r.CacheWritePrice
}
}
return db.UpsertAIModelPrices(ctx, database.UpsertAIModelPricesParams{
Providers: providers,
Models: models,
InputPrices: inputs,
OutputPrices: outputs,
CacheReadPrices: cacheR,
CacheWritePrices: cacheW,
})
```
### Pros
- Single round-trip.
### Cons
- The generated `sqlc` params become plain `[]int64`, which can't
represent `NULL`.
---
## Approach 3 — `jsonb_array_elements` over a single `@seed::jsonb`
(chosen)
Pass the raw seed JSON as one parameter; let Postgres expand and parse
it.
```sql
INSERT INTO ai_model_prices (
provider,
model,
input_price,
output_price,
cache_read_price,
cache_write_price
)
SELECT
elem->>'provider',
elem->>'model',
(elem->>'input_price')::bigint,
(elem->>'output_price')::bigint,
(elem->>'cache_read_price')::bigint,
(elem->>'cache_write_price')::bigint
FROM jsonb_array_elements(@seed::jsonb) AS elem
ON CONFLICT (provider, model) DO UPDATE SET
input_price = EXCLUDED.input_price,
output_price = EXCLUDED.output_price,
cache_read_price = EXCLUDED.cache_read_price,
cache_write_price = EXCLUDED.cache_write_price,
updated_at = NOW();
```
Go side reduces to:
```go
return db.UpsertAIModelPrices(ctx, seedJSON)
```
### Pros
- Single round-trip.
- NULLs fall out naturally:
- `(elem->>'cache_write_price')::bigint` becomes `NULL`
- no sentinels
- The seed is already JSON:
- Existing precedent:
- `jsonb_array_elements` is already used elsewhere in the codebase
### Cons
- Less type-safe at the SQL boundary than `UNNEST`
- Slightly less standard than `UNNEST`
- Readers need familiarity with:
- `jsonb_array_elements`
- `->>` extraction syntax
- Postgres pays JSON parse cost
- negligible at our scale
---
---
# Decision
We picked Approach 3.
It collapses the round-trips like `UNNEST` does, but without:
- nullable-array workarounds
- sentinel values
Registers a new aibridge provider for ChatGPT by reusing the existing
OpenAI provider with a different `Name` and `BaseURL`
(https://chatgpt.com/backend-api/codex). The ChatGPT backend API is
OpenAI-compatible, so no new provider type is needed.
ChatGPT authenticates exclusively via per-user OAuth JWTs (BYOK mode) —
no centralized API key is configured. The OpenAI provider already
handles this: when no key is set, it falls through to the bearer token
from the request's Authorization header.
Depends on #23811
## Description
Adds support for multiple Copilot provider instances to route requests to different Copilot upstreams (individual, business, enterprise). Each instance has its own name and base URL, enabling per-upstream metrics, logs, circuit breakers, API dump, and routing.
## Changes
* Add Copilot business and enterprise provider names and host constants
* Register three Copilot provider instances in aibridged (default, business, enterprise)
* Update `defaultAIBridgeProvider` in `aibridgeproxy` to route new Copilot hosts to their corresponding providers
## Related
* Depends on: https://github.com/coder/aibridge/pull/240
* Closes: https://github.com/coder/aibridge/issues/152
Note: documentation changes will be added in a follow-up PR.
_Disclaimer: initially produced by Claude Opus 4.6, heavily modified and reviewed by @ssncferreira ._
## Problem
`aibridgeproxyd` sends `X-AI-Bridge-Request-Id` on every MITM request to
`aibridged` for cross-service log correlation, but aibridged never reads
it. The header is silently forwarded to upstream LLM providers.
## Changes
* Renamed the header to `X-Coder-AI-Governance-Request-Id` to match the
existing `X-Coder-AI-Governance-*` convention.
* `aibridged` now extracts the header, logs it and strips it before
forwarding upstream.
* Added `TestServeHTTP_StripInternalHeaders` to verify no `X-Coder-*`
headers leak to upstream
### Changes
**coder/coder:**
- `coderd/aibridge/aibridge.go` — Added `HeaderCoderBYOKToken` constant,
`IsBYOK()` helper, and updated `ExtractAuthToken` to check the BYOK
header first.
- `enterprise/aibridged/http.go` — BYOK-aware header stripping: in BYOK
mode only the BYOK header is stripped (user's LLM credentials
preserved); in centralized mode all auth headers are stripped.
<hr/>
**NOTE**: `X-Coder-Token` was removed! As of now `ExtractAuthToken`
retrieves token either from `X-Coder-AI-Governance-BYOK-Token` or from
`Authorization`/`X-Api-Key`.
---------
Co-authored-by: Susana Ferreira <susana@coder.com>
Co-authored-by: Danny Kopping <danny@coder.com>
## Description
Introduces a new `X-Coder-Token` header for authenticating requests from
AI Proxy to AI Bridge. Previously, the proxy overwrote the
`Authorization` header with the Coder token, which prevented the
original authentication headers from flowing through to upstream
providers.
With this change, AI Proxy sets the Coder token in a separate header,
preserving the original `Authorization` and `X-Api-Key` headers. AI
Bridge uses this header for authentication and removes it before
forwarding requests to upstream providers. For requests that don't come
through AI Proxy, AI Bridge continues to use `Authorization` and
`X-Api-Key` for authentication.
## Changes
* Add `HeaderCoderAuth` constant and update `ExtractAuthToken` to check
headers in the following order: `X-Coder-Token` > `Authorization` >
`X-Api-Key`
* Update AI Proxy to set `X-Coder-Token` instead of overwriting
`Authorization`
* Remove `X-Coder-Token` in AI Bridge before forwarding to upstream
providers
* Add tests for header handling and token extraction priority
Related to: https://github.com/coder/internal/issues/1235
## Summary
This adds configurable overload protection to the AI Bridge daemon to
prevent the server from being overwhelmed during periods of high load.
Partially addresses coder/internal#1153 (rate limits and concurrency
control; circuit breakers are deferred to a follow-up).
## New Configuration Options
| Option | Environment Variable | Description | Default |
|--------|---------------------|-------------|---------|
| `--aibridge-max-concurrency` | `CODER_AIBRIDGE_MAX_CONCURRENCY` |
Maximum number of concurrent AI Bridge requests. Set to 0 to disable
(unlimited). | `0` |
| `--aibridge-rate-limit` | `CODER_AIBRIDGE_RATE_LIMIT` | Maximum number
of AI Bridge requests per second. Set to 0 to disable rate limiting. |
`0` |
## Behavior
When limits are exceeded:
- **Concurrency limit**: Returns HTTP `503 Service Unavailable` with
message "AI Bridge is currently at capacity. Please try again later."
- **Rate limit**: Returns HTTP `429 Too Many Requests` with
`Retry-After` header.
Both protections are optional and disabled by default (0 values).
## Implementation
The overload protection is implemented as reusable middleware in
`coderd/httpmw/ratelimit.go`:
1. **`RateLimitByAuthToken`**: Per-user rate limiting that uses
`APITokenFromRequest` to extract the authentication token, with fallback
to `X-Api-Key` header for AI provider compatibility (e.g., Anthropic).
Falls back to IP-based rate limiting if no token is present. Includes
`Retry-After` header for backpressure signaling.
2. **`ConcurrencyLimit`**: Uses an atomic counter to track in-flight
requests and reject when at capacity.
The middleware is applied in `enterprise/coderd/aibridge.go` via
`r.Group` in the following order:
1. Concurrency check (faster rejection for load shedding)
2. Rate limit check
**Note**: Rate limiting currently applies to all AI Bridge requests,
including pass-through requests. Ideally only actual interceptions
should count, but this would require changes in the aibridge library.
## Testing
Added comprehensive tests for:
- Rate limiting by auth token (Bearer token, X-Api-Key, no token
fallback to IP)
- Different tokens not rate limited against each other
- Disabled when limit is zero
- Retry-After header is set on 429 responses
- Concurrency limiting (allows within limit, rejects over limit,
disabled when zero)
Yevhenii Shcherbina