## Description
Adds Prometheus metrics for AI budget cost control, emitted by the
aibridged server under the `cost_control` subsystem (full names are
prefixed `coder_ai_gateway_`).
- `blocked_requests_total` (counter) — labels: `group_id`
- `blocked_users` (gauge) — labels: `group_id`
- `unpriced_requests_total` (counter) — labels: `provider`, `model`
- `enforcement_duration_seconds` (histogram) — labels: `outcome`
## Changes
- Add `GetOverBudgetUsersPerGroup` query (plus dbauthz/dbmetrics/dbmock
wiring) to count over-budget users per effective group.
- Add a background collector that refreshes the `blocked_users` gauge on
an interval, started only when Prometheus is enabled.
- Wire `Metrics` through the aibridged server, coderd API,
`cli/server.go`, and the enterprise AI gateway handler; recording is
nil-safe when metrics are unset.
Closes
https://linear.app/codercom/issue/AIGOV-296/add-prometheus-metrics-for-cost-control
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by
@ssncferreira
Implements:
https://linear.app/codercom/issue/AIGOV-289/notify-users-and-admins-on-budget-warning-and-limit-reached
Notify users when their AI spend crosses a budget threshold for their
effective group. Two thresholds are covered: a warning at 85%, and a
limit-reached notification at 100%.
Detection runs on the post-response path, right after the interception's
cost is added to the user's daily spend. It reads the user's AI spend on
the same transaction where token usage is recorded and AI daily spend is
incremented, and derives the pre-interception total by subtracting this
interception's cost. In case of `oldSpend < threshold && newSpend >=
threshold` - notification is sent. A single interception that crosses
both thresholds enqueues both notifications.
Detection and delivery are best-effort: a failure is logged and never
fails usage recording. The payload uses only stable values (the
threshold percentage and the spend limit, not the exact spend), so
duplicate enqueues are deduplicated by the notification system.
The two templates are added via migration and appear in each user's
notification settings under the "AI Budget" group.
Admin notifications (owners and user admins) are a follow-up: #27415.
## Screenshots:
<img width="1102" height="252" alt="image"
src="https://github.com/user-attachments/assets/62291510-09ca-4cdf-a1f5-4bdc11a1db4b"
/>
<img width="466" height="384" alt="image"
src="https://github.com/user-attachments/assets/030460ff-6fe2-4d59-b247-3550c543ef30"
/>
---------
Co-authored-by: Cian Johnston <cian@coder.com>
## Description
Read the AI budget period from the deployment config on both the RPC server and the `/users/{user}/ai/spend` endpoint, instead of hardcoding `month`. Drops the `period_start` RPC parameter that was incorrectly introduced in #26915: the period should have been derived from the deployment config from the start.
## Changes
- Add `codersdk.NewAIBudgetPeriodFromString`, mirroring `NewAIBudgetPolicyFromString`.
- `aibridgedserver.Server` takes a `quartz.Clock`, reads `BudgetPeriod` from the deployment config at construction, and computes the period window inside `IsBudgetExceeded`.
- Remove `period_start` from `IsBudgetExceededRequest` and stop sending it from the daemon.
- The `userAISpendStatus` endpoint reads the period from `AIBridgeConfig.BudgetPeriod` instead of hardcoding month.
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by @ssncferreira
## Why
PR #26797 was accidentally merged into the stale `graphite-base/26797`
branch instead of `main` (Graphite picked the wrong base), so its
changes never landed on `main`. This PR re-lands that work as a clean
cherry-pick onto the current `main`.
## What
Adds a `WatchAIProviders` streaming RPC to the `ProviderConfigurator`
service so a running standalone AI Gateway refetches its provider set
when the provider configuration changes. The server subscribes to
`AIProvidersChangedChannel` (published by the provider CRUD endpoints)
and forwards each event as a payload-free signal, plus one signal on
subscribe; the gateway calls `GetAIProviders` on each signal to rebuild
its pool. The aibridged API is bumped to v1.2.
Env-seeded providers don't need a signal: seeding finishes before coderd
serves the gateway connection, so the gateway's initial fetch already
reflects the seeded set.
## For reviewers
The change is split into two commits to make review easy:
1. **`feat: synchronise provider changes with WatchAIProviders`** is a
faithful cherry-pick of #26797, identical to the originally reviewed PR.
It is committed without pre-commit hooks because it does not build
against current `main` on its own.
2. **`fix: resolve cherry-pick conflicts against main`** contains only
the deltas needed to re-land on current `main`, and passes the full
pre-commit suite:
- `coderd/aibridged/proto/aibridged.pb.go` regenerated via the proto
make target (the cherry-picked copy was generated against the older
proto).
- `enterprise/cli/aigatewaystart.go` import block unioned; `main` added
`os` and `strings` while the PR added `sync`.
- Three `aibridgedserver.NewServer` test call sites that landed on
`main` after the original branch diverged now pass the new `pubsub`
argument.
Refs https://linear.app/codercom/issue/AIGOV-465
*This PR was produced by opencode (agent) using the
`anthropic/claude-opus-4-8` model, under human direction and review.*
> AI Tools were used to produce this PR
This PR adds `coder ai-gateway start` command that runs the AI Gateway
as an independent process.
- Standalone process doesn't have access to DB. Uses DRPC services under
`/api/v2/ai-gateway/serve`for auth, recording and provider
initialization.
- It only handles LLM traffic, other endpoints (eg. `/sessions`) are
only available though `coderd`.
- The standalone gateway reuses applicable flags from AI Gateway
deployment options. Provider-seeding and coderd-only options are
excluded.
- Only added to fat build, the slim build stub rejects the command.
Some wiring used by this new command is added.
**`NewWebsocketDialer`** - implements the standalone gateway's
connection to coderd's `/api/v2/ai-gateway/serve` endpoint. It upgrades
to a WebSocket, multiplexes with yamux, and wires all DRPC services.
**`AIGatewayDataPlaneMiddleware`** - extracts the per-request middleware
chain (concurrency limiting, rate limiting, BYOK gating) into a shared
function used by both the embedded route and the standalone gateway.
**`RootCmd.ResolveClientConnection`** - resolve the deployment URL and
builds an HTTP transport without requiring a session token. Used in
`ai-gateway start`command as it authenticates using different credential
type.
---------
Co-authored-by: Danny Kopping <danny@coder.com>
Hides UI, CLI and API related to AI Gateway key management +
`/api/v2/ai-gateway/serve` endpoint.
API endpoints and CLI commands are still working they are just not
visible.
Closes [AIGOV-455](https://linear.app/codercom/issue/AIGOV-455/extend-drpc-with-buildproviders).
## Why
The AI Gateway (`aibridged`) is being split into a standalone process that must not touch the database. `coderd` stays the source of truth and seeds the `ai_providers` / `ai_provider_keys` tables from the environment. This PR adds a DRPC call so the gateway fetches provider config from `coderd` instead of reading the DB, for both the embedded and standalone daemons.
## What
- **Proto:** new `ProviderConfigurator` service with a unary `GetAIProviders` RPC, plus `AIProvider` / `AIProviderBedrock` messages. `CurrentMinor` bumped to 1 (additive).
- **Server (`coderd/aibridgedserver`):** `GetAIProviders` runs a read-only `InTx` under `LockIDAIProvidersEnvSeed` so it never returns a mid-seed snapshot, reads providers (incl. disabled) plus keys for enabled ones, and maps to proto under `dbauthz.AsAIBridged`. Unmappable rows are skipped and logged; plaintext keys and Bedrock secrets are never logged.
- **Client:** `DRPCProviderConfiguratorClient` wired into the client union, `dialer.go`, and `CreateInMemoryAIBridgeServer`.
- **cli:** `BuildProvidersFromProto` maps the response through the existing DB-neutral `buildProvider`. A shared `poolRPCReloader` does the fetch/build/replace for both daemons: the embedded daemon reloads on every `ai_providers` change and fails startup if it cannot subscribe; the standalone gateway drives the same reloader once at startup, retrying until success and staying interruptible.
- **Dead code removed:** `BuildProvidersFromConfig`, `ProvidersFromConfig`, `AIProviderFromConfig`, and the DB-read `BuildProviders` path.
Renames the `last_used_at` column to `last_heartbeat_at` in `ai_gateway_keys` table.
`ai_gateway_keys` table has not been released yet.
All references updated.
Adds a new enterprise-only `GET /api/v2/ai-gateway/serve` endpoint that standalone AI Gateway replicas use to connect to `coderd` over a DRPC-over-WebSocket transport, mirroring the existing in-memory path used by the embedded AI Bridge daemon.
- The endpoint upgrades the HTTP connection to a WebSocket, multiplexes it with yamux, and finally serves the three DRPC services (Recorder, MCPConfigurator, Authorizer).
- The `X-AI-Governance-Gateway-Key` header is used for authentication.
- The key is looked up by its hashed secret
- Missing or revoked keys return `401`.
- API version negotiation is enforced via a new `aibridged/proto` version (`v1.0`).
- Incompatible versions return `400`.
- `FeatureAIBridge` entitlement is required.
- Key liveness (`last_used_at`) is recorded immediately on connection and refreshed every 60 seconds while the session remains open.
- When key liveness detects the key was deleted (no rows where updated) session is closed.
#### Small refactors
* The three DRPC service registrations are extracted into `aibridgedserver.Register`, shared by both the in-memory and WebSocket paths.
* The literal `256 * 1024` used as the yamux-aligned WebSocket read limit is replaced with the named constant `drpcsdk.YamuxDefaultStreamWindowSize` in all call sites.
* as noted in review comment https://github.com/coder/coder/pull/26506#discussion_r3461905223 order of `SetReadLimit` and `WebsocketNetConn` calls was fixed.
Susana Ferreira