## Problem
Local tool results were persisted and replayed to the model verbatim,
with no size cap. A single oversized result, most often a multi-megabyte
response from an MCP tool, overflows the prompt on the next request.
Every retry rebuilds the same history and fails the same way, leaving
the chat wedged in `error`. Auto-compaction is reactive (token usage is
only known after a response), so it can't catch a single result that
blows the very next request.
## Fix
Cap every locally-executed tool result at its single choke point,
`executeSingleTool` in `chatloop`, so the cap covers built-in tools,
**global (deployment-pinned) MCP**, **workspace MCP**, and provider
runners uniformly. Because this runs before the result is published to
the live stream and before it is committed, the SSE preview, the
persisted message, and the model replay all see the same bounded output.
The budget is token-aware: a single tool result may use at most half the
model's context window (`~4 bytes/token`), with a `16KB` floor and a
`64KB` default when the window is unknown. Truncation keeps the head and
tail of the output and replaces the middle with a marker telling the
model how much was removed and to narrow its query; it is UTF-8 safe and
never exceeds the budget. Binary media `Data` is passed through
untouched (only the text payload is bounded).
A `coderd_chatd_tool_result_truncated_total{provider,model,tool_name}`
counter and a warning log record each truncation.
## Out of scope
- Provider-executed results (e.g. web search) arrive via the stream, not
`executeSingleTool`.
- Dynamic/external tool results submitted through the `/tool-results`
API are validated as JSON elsewhere.
- Cumulative growth across many results is still handled by context
compaction; this change only bounds any single result.
<details>
<summary>Implementation notes</summary>
- New `coderd/x/chatd/chatloop/tooltruncate.go`:
`toolResultByteBudget(contextLimitTokens)` and
`truncateToolResultText(text, maxBytes)` (pure, unit-tested).
- `chatloop.go`: added `ContextLimit` to `ExecuteLocalToolsOptions`;
threaded a computed byte budget through `executeTools` into
`executeSingleTool`, where `resp.Content` is capped for the text,
media-text, and error branches.
- `generation.go`: passes `ContextLimit: prepared.ContextLimitFallback`
(the model's configured context limit).
- `metrics.go`: new `ToolResultTruncatedTotal` counter +
`RecordToolResultTruncated`.
- Tunable knobs live as constants in `tooltruncate.go`
(`toolResultContextDivisor = 2`, `bytesPerTokenEstimate`,
`minToolResultBytes`, `defaultToolResultBytes`).
Verified: `go build ./coderd/x/chatd/...`, `go test
./coderd/x/chatd/chatloop/...`, and the `chatd` test binary compiles.
</details>
---
Resolves CODAGT-678
Generated by Coder Agents on behalf of @kylecarbs.
Closes [DOCS-415](https://linear.app/codercom/issue/DOCS-415).
## TL;DR
Repoints 7 broken links across 5 docs files that the 2026-06-22 weekly
`check-docs.yml` Linkspector run flagged. Two other links from the same
run (the dead `nix` ref and the dead `reflectoring.io` ref in
`CONTRIBUTING.md`) were already folded into
[#26341](https://github.com/coder/coder/pull/26341).
## Why
Broken external and anchor links degrade reader trust, leak SEO juice,
and make the docs look stale. The weekly `check-docs` job exists
precisely to catch this kind of rot before customers do; the
surfacing-to-fix turnaround on these 7 is one PR. Run that surfaced
them: [actions/runs/27948011619 job
82697664858](https://github.com/coder/coder/actions/runs/27948011619/job/82697664858).
## Scope
| File | Line(s) | Old target | New target | Why |
|------|---------|-----------|------------|-----|
| `docs/tutorials/best-practices/organizations.md` | 62 | anchor
`#update-template-metadata-by-id` | `#update-template-settings-by-id` |
API endpoint renamed in
[#19228](https://github.com/coder/coder/pull/19228) (Aug 2025). New
heading at line 1105 of `docs/reference/api/templates.md`. |
| `docs/install/registry-mirror-artifactory.md` | 197 | JFrog
`terraform-registry` |
`terraform-opentofu-and-terraform-backend-repositories` | JFrog
consolidated their Terraform / OpenTofu / Backend docs into a single
page. |
| `docs/admin/templates/extending-templates/modules.md` | 76, 206 |
JFrog `set-up-a-terraform-module/provider-registry` and
`terraform-registry` | same consolidated JFrog page (root, no anchor) |
Same JFrog consolidation. Anchor dropped, see decision log. |
| `docs/admin/integrations/dx-data-cloud.md` | 84 |
`https://help.getdx.com/en/` | `https://docs.getdx.com/` | DX migrated
their help center to a separate docs domain. |
| `docs/about/contributing/frontend.md` | 37, 71 |
`https://reactrouter.com/en/main` | `https://reactrouter.com/` | React
Router dropped the `/en/main` prefix. |
## Validation
- All 7 replacement URLs return HTTP 200 (manual `curl -L -o /dev/null
-w '%{http_code}'` per URL; linkspector's puppeteer crashed in the agent
env, so it was run case-by-case)
- `make lint/markdown lint/emdash` clean locally
- Pre-commit hook (`scripts/githooks/pre-commit` -> `make
pre-commit-light`) clean
- No `/docs/` route changes; pure markdown content
## Not triggering `/coder-agents-review`
Docs-only markdown edit, no CI or build config changes; per `AGENTS.md`
the bot review is reserved for product / CI changes. `doc-check` handles
this category.
## Pre-mortem
| Concern | Mitigation |
|---|---|
| Replacement URL also turns out to be broken later | All 7 verified
HTTP 200 today; next weekly `check-docs` run will catch any future
regression. |
| JFrog anchor drop on `modules.md` (76, 206) loses navigation context |
Verified the consolidated JFrog page has no clean section anchor for the
original target; linking the root page is the honest fix. If JFrog ships
a better TOC anchor later, a follow-up can reattach. |
| Anchor rename in `organizations.md` was actually a different rename |
Confirmed via PR #19228 (Aug 2025) which is the exact rename that
produced `## Update template settings by ID`. |
<details>
<summary>Decision log</summary>
**Why drop the anchor on the JFrog `modules.md` links (76 + 206)**:
JFrog's new consolidated page
(`/terraform-opentofu-and-terraform-backend-repositories`) doesn't
expose the original `set-up-a-terraform-module/provider-registry`
section as a fragment-link target. The honest fix is to link the page
root; readers can scroll. The `registry-mirror-artifactory.md:197`
reference uses the same root link for symmetry.
**Why DX `docs.getdx.com` over `help.getdx.com`**: DX's help center at
`help.getdx.com/en/` now returns 404. They moved to a separate
`docs.getdx.com` domain with a different content structure. Linking the
docs root is the closest analog to the original "browse our docs"
intent.
**Why React Router root over `/en/main`**: React Router unified their
docs under the root URL. The `/en/main` prefix is no longer routable.
The root URL is the canonical successor.
</details>
<details>
<summary>CI: <code>audit-docs-paths</code> failure (pre-existing,
unrelated)</summary>
The `audit-docs-paths` job in `.github/workflows/weekly-docs.yaml` fails
on this PR because its `Fetch redirects.json` step issues an
unauthenticated `curl` to a file in private `coder/coder.com` and gets a
404 (exit code 22). Same failure on every recent PR in this repo.
Tracked in [DOCS-409](https://linear.app/codercom/issue/DOCS-409) and
fixed in [#26571](https://github.com/coder/coder/pull/26571), which
authenticates the fetch through the Contents API. My changes are
docs-content only (5 markdown files, 7 line changes) and don't touch the
TS/TSX paths or `redirects.json` that the audit examines, so this is a
pre-existing CI break, not a regression introduced here.
</details>
---
*Generated by Coder Agents on @nickvigilante's behalf.*
Closes [DOCS-351](https://linear.app/codercom/issue/DOCS-351).
> [!WARNING]
> **DO NOT MERGE** until
[DOCS-349](https://linear.app/codercom/issue/DOCS-349)
([coder.com#877](https://github.com/coder/coder.com/pull/877)) has
shipped to production and baked for at least one Vercel cycle.
>
> Without DOCS-349, the relative links in this PR resolve to broken
docs-route URLs (`/docs/helm/coder/values.yaml` -> 404) instead of
GitHub URLs tagged with the displayed docs version. DOCS-349 fixes the
rewriter to classify these as GitHub blob/tree URLs with the page's
resolved ref.
## TL;DR
Converts 121 absolute
`https://github.com/coder/coder/(blob|tree)/main/<path>` links across 39
docs markdown files to relative paths. After this lands AND DOCS-349
deploys, every one of these links will follow the displayed docs version
(mainline tag on bare URLs, explicit tag on `/@vX.Y.Z/`, `main` on
`/@main/`) instead of always pointing to `main`.
## Why
Today a reader on `/docs/@v2.30.0/install/docker` follows a
`compose.yaml` link and arrives at `main`'s `compose.yaml`, which
doesn't necessarily match what the docs page describes. Helm values,
Terraform templates, and source-code references in particular drift
across versions. The fix is to let the coder.com rewriter substitute the
page's resolved ref into the URL; that only works on relative links.
## Example payoff (post-DOCS-349)
| URL | Today (absolute, always `main`) | After (relative + rewriter) |
|---|---|---|
| `/docs/install/docker` |
`https://github.com/coder/coder/blob/main/compose.yaml` |
`https://github.com/coder/coder/blob/v2.34.1/compose.yaml` (today's
mainline) |
| `/docs/@v2.30.0/install/docker` | same as above |
`https://github.com/coder/coder/blob/v2.30.0/compose.yaml` |
| `/docs/@main/install/docker` | same as above |
`https://github.com/coder/coder/blob/main/compose.yaml` |
## Scope
- **121 conversions** across **39 files**.
- Verb breakdown: `tree/main` (directories) and `blob/main` (files),
both flipped to relative paths.
- Line anchors (`#L23-L24`) and query strings preserved verbatim.
- Conversion is mechanical: relative path computed from the doc file's
directory to the target via `os.path.relpath`. Any path starting at the
same directory or below gets a `./` prefix; otherwise `../` chains.
## Rebased on main
The branch was rebased onto `main` after the DOCS-350 hotfix
([#26339](https://github.com/coder/coder/pull/26339)) merged. The hotfix
repointed 3 `docs-backend-contrib-guide` refs in `backend.md` to `main`,
which then needed the same `main` -> relative conversion this PR is
doing for the other 121 links. The conflict was resolved by reapplying
the mechanical conversion to `backend.md` after taking the hotfix's
content. Net result: those 3 links land here as relative, same as
everything else. New HEAD `3f501cb622`.
## Inline fix folded in: dead `nix` link
- `docs/about/contributing/CONTRIBUTING.md:7` -> `../../../nix`
The original absolute URL `https://github.com/coder/coder/tree/main/nix`
already returned 404 today. Repointed to `flake.nix` (modern Nix
entrypoint, what the prose "Nix environment" semantically refers to).
Closes [DOCS-357](https://linear.app/codercom/issue/DOCS-357) here since
the `check-docs` Linkspector job surfaced it during rebase; cheaper to
fix inline than in a separate single-line PR.
## Out of scope (filed separately)
- [DOCS-350](https://linear.app/codercom/issue/DOCS-350): 3 dead
`docs-backend-contrib-guide` branch refs in `backend.md`
([#26339](https://github.com/coder/coder/pull/26339), merged).
- [DOCS-352](https://linear.app/codercom/issue/DOCS-352): 10 SHA-pinned
`(blob|tree)/<sha>` links pending intent review.
- [DOCS-355](https://linear.app/codercom/issue/DOCS-355): code-server
analog (4 absolute `(blob|tree)/main` links in `coder/code-server`).
- [DOCS-356](https://linear.app/codercom/issue/DOCS-356): 2 upstream
content bugs in `coder/code-server/docs/CONTRIBUTING.md` (independent of
this PR).
## Not triggering `/coder-agents-review`
Docs-only edit; per `AGENTS.md` the bot review is reserved for
product/CI changes.
## Pre-mortem
| Concern | Mitigation |
|---|---|
| Merging before DOCS-349 deploys regresses ~120 currently-working links
into 404s on coder.com | Clear DO-NOT-MERGE banner; tracked as blocker
in Linear. |
| Relative path computed incorrectly (off-by-one `..`) | Verified all
114 newly-relative non-md/non-image paths resolve to existing files in
the repo (only exception is the pre-existing dead `nix` link above). |
| Line anchors stripped during conversion | Preserved by the
substitution regex; verified `#L<n>-L<m>` cases in `airgap.md` and
`speed-up-templates.md`. |
| Future code reorgs change file locations | Relative links will start
pointing to nothing. Same failure mode as absolute links pointing to
renamed files; can be caught with a future link-checker job. |
## Validation
```
$ grep -rE 'github\.com/coder/coder/(blob|tree)/main' docs --include="*.md" | wc -l
0
$ git diff --stat origin/main | tail -1
39 files changed, 118 insertions(+), 118 deletions(-)
```
114 newly-relative paths verified to resolve to existing repo files
(Python `os.path.exists` check on each computed target).
<details>
<summary>Decision log + planning context</summary>
**Why relative over `(blob|tree)/{{currentDocsVersion}}/...`
templating**: relative paths require zero markdown-system support and
zero upstream churn beyond this one PR. Templating would require a
preprocessor on `coder.com` side AND a convention upstream authors have
to remember; relative paths just work in a plain editor and
`github.com`'s own renderer too.
**Why `./` prefix on same-directory targets**: makes the conversion
grep-able later (`grep -E '\((\.\./|\./)'`).
**Why preserve `#L<n>-L<m>` anchors verbatim**: the anchor is meaningful
to the linked file's content, not to the URL form; keeping it as-is
preserves authorial intent. If the file later changes such that the line
range drifts, that's a different problem the SHA-pin audit
([DOCS-352](https://linear.app/codercom/issue/DOCS-352)) will surface.
</details>
---
*Generated by Coder Agents on @nickvigilante's behalf.*
## Drive-by external link fix folded in
`docs/about/contributing/CONTRIBUTING.md:296` cited
`https://reflectoring.io/meaningful-commit-messages/` which is returning
HTTP 503 (the host appears to be down site-wide right now). `check-docs`
Linkspector flagged it after the rebase. Replaced with
`https://cbea.ms/git-commit/` (Chris Beams' canonical "If applied, this
commit will..." article, confirmed 200), which is the original source of
the rule the prose recites anyway.
> AI Tools where used in this request.
Registers `coder_aibridged_*` and `coder_aibridgeproxyd_*` metrics under
new prefixes: `coder_ai_gateway_*` and `coder_ai_gateway_proxy_*`.
Old prefix is still exported. Will be removed in later release.
Also updated the `metricsdocgen` static fixture. Added 4
previously-undocumented metrics `key_pool_state`,
`key_pool_state_transitions_total`, `key_pool_exhaustions_total`,
`key_pool_failover_attempts` added the `client` label to the existing
interception, prompt, and token counter samples.
Updated AI Gateway documentation.
Relates to CODAGT-115
Adds metric `coderd_api_websocket_probes_total`. Every successful
heartbeat for a given path will increment the metric.
Comparing this with `coderd_api_concurrent_websockets` will give an
indication of how many websocket connections are open but in a 'wedged'
state (when heartbeats stopped versus when we closed the connection).
Add metrics for `aibridged` and `aibridgeproxyd`'s provider statuses. AI providers can be modified, and possibly misconfigured, at runtime. These metrics help operators understand the state of these provider definitions in case unexpected behaviour is observed.
## Summary
Three small docs fixes:
- **`docs/admin/integrations/oauth2-provider.md`**: Replace broken
relative link to `scripts/oauth2/README.md` with an absolute GitHub URL.
The previous link escaped the `docs/` tree
(`../../../scripts/oauth2/README.md`) and does not resolve in the
published docs site.
- **`docs/install/releases/feature-stages.md`**: Point the "Coder
documentation" link to `docs/about/contributing/documentation.md`. The
previous `../../README.md` target does not exist under `docs/`.
- **`docs/manifest.json`**: Add the missing `users oidc-claims` entry
alongside the other `users` CLI subcommands so the generated reference
page (`docs/reference/cli/users_oidc-claims.md`) is reachable from the
sidebar.
## Validation
- Confirmed each new link target exists on `main`
(`docs/about/contributing/documentation.md`, `scripts/oauth2/README.md`,
`docs/reference/cli/users_oidc-claims.md`).
- Pre-commit hooks pass (`fmt/markdown`, `lint/markdown`, `lint/emdash`,
`lint/typos`, etc.).
---
_This PR was prepared by a [Coder Agents](https://coder.com/) session on
behalf of @nickvigilante. Human review requested since this is a
docs-only change._
Moves the `coderd_agents_first_connection_seconds` histogram from the
polling-based `prometheusmetrics.Agents()` loop to the event-driven
`agentConnectionMonitor.init()` path. The metric is now recorded exactly
once when an agent first connects over the RPC websocket, instead of
being retroactively computed each polling tick.
The `username` and `workspace_name` labels are removed to reduce
cardinality; only `template_name` and `agent_name` are retained.
Adds unit tests covering both the happy path (first connection recorded)
and the negative-duration guard (clock skew logs a warning, no sample
emitted).
When OpenAI's Responses API returns `Previous response with id ... not
found` for a chained turn, classify it as a `ChainBroken` retry, clear
`previous_response_id`, exit chain mode, reload full history, and let
`chatretry` retry. Self-heals chats whose anchor was poisoned before
#25074 stopped truncated streams from being persisted as a successful
turn with a stored response id.
The new state is exposed via the existing
`coderd_chatd_stream_retries_total` counter as a
`chain_broken="true"|"false"` label. Aggregating queries (`sum`, `rate`
over `provider`/`model`/`kind`) keep working without changes; raw-series
matchers without aggregation will now see two series per `(provider,
model, kind)` where they previously saw one. The metric is internal-only
so the blast radius should be small, but if you have dashboards that
index by exact label matchers without aggregation they will need an
extra `sum` or an explicit `chain_broken` selector.
> 🤖 This PR was created with the help of Coder Agents, and was reviewed by a human 🧑💻
Adds production-observability metrics to coderd/x/chatd/ for
model-level correlation and a chatStreams memory-leak investigation.
- Label per-request chatd metrics (steps_total, message_count,
prompt_size_bytes, tool_result_size_bytes, ttft_seconds,
compaction_total) with `model` and enrich the per-turn logger
with provider/model.
- Add `coderd_chatd_stream_retries_total{provider, model, kind}`
counter incremented in chatloop before OnRetry.
- Register a prometheus.Collector exposing `streams_active`,
`stream_buffer_size_max`, `stream_buffer_events`,
`stream_subscribers` from p.chatStreams.
- Add `coderd_chatd_stream_buffer_dropped_total` counter,
incremented per publishToStream drop independently of the
existing log-rate-limited bufferDropCount.
- Snapshot logger/model before the title-generation goroutine to
avoid a data race with the logger/model rebind below it.
> 🤖
_Disclaimer: produced by Claude Opus 4.6_
Adds a `coder_build_info` metric which allows operators to see which
versions of Coder are currently running.
---------
Signed-off-by: Danny Kopping <danny@coder.com>
## Summary
Add `coderd_agents_first_connection_seconds` histogram metric that
records the
duration from workspace agent creation to first connection. This fills
an
observability gap — provisioner job timings and startup script metrics
exist,
but the agent connection phase (which can take several minutes) was not
exposed
to Prometheus.
Closes https://github.com/coder/coder/issues/21282
## Changes
- **`coderd/prometheusmetrics/prometheusmetrics.go`** — Define and
register a
`HistogramVec` in the existing `Agents()` polling loop. Observe
`first_connected_at - created_at` exactly once per agent via a
deduplication
map, pruned each tick to prevent unbounded memory growth.
- **`coderd/prometheusmetrics/prometheusmetrics_test.go`** — Update
`TestAgents`
to set `first_connected_at` on the test agent and assert the histogram
is
collected with correct labels, sample count, and sample sum.
- **`docs/admin/integrations/prometheus.md`**,
**`scripts/metricsdocgen/generated_metrics`** —
Auto-generated documentation updates from `make gen`.
## Metric details
| Property | Value |
|---|---|
| Name | `coderd_agents_first_connection_seconds` |
| Type | histogram |
| Labels | `template_name`, `agent_name`, `username`, `workspace_name` |
| Buckets | 1s, 10s, 30s, 1m, 2m, 5m, 10m, 30m, 1h |
## Example PromQL
```promql
# P95 agent connection time by template
histogram_quantile(0.95,
sum(rate(coderd_agents_first_connection_seconds_bucket[1h])) by (le, template_name)
)
```
<details>
<summary>Implementation notes</summary>
### Design decisions
- **Histogram over gauge**: Enables `histogram_quantile()` for
percentile queries.
- **Observe in `Agents()` polling loop**: All required data is already
fetched by
`GetWorkspaceAgentsForMetrics()` — no new DB queries.
- **Dedup via `map[uuid.UUID]struct{}`**: Prevents re-observing the same
agent
across polling ticks. Pruned each cycle to bound memory.
- **Buckets**: Aligned with
`coderd_provisionerd_workspace_build_timings_seconds`
range (1s–1h).
### Overhead at scale (100k active workspaces)
The deduplication map (`observedFirstConnection`) and per-tick pruning
map
(`currentAgentIDs`) are both `map[[16]byte]struct{}`. At 100k agents:
- **Memory**: ~2.25 MB persistent + ~2.25 MB transient per tick = **~4.5
MB peak**.
- **CPU**: ~25 ms of map operations per tick (one tick per minute) =
**<0.05% of one core**.
Both are negligible relative to the existing cost of the `Agents()` loop
(the DB
query, per-agent `GetWorkspaceAppsByAgentID` calls, and coordinator node
lookups
dominate).
</details>
> 🤖 Generated by Coder Agents
Go's html/template has a built-in security filter (urlFilter) that only
allows http, https, and mailto URL schemes. Any other scheme gets
replaced with #ZgotmplZ.
The OAuth2 app's callback URL uses custom URI scheme which the filter
considers unsafe. For example the Coder JetBrains plugin exposes a
callback URI with the scheme jetbrains:// - which was effectively
changed by the template engine into #ZgotmplZ. Of course this is not an
actual callback. When users clicked the cancel button nothing happened.
The fix was simple - we now wrap the apps registered callback URI into
htmltemplate.URL. Usually this needs some validation otherwise the
linter will complain about it. The callback URI used by the Cancel logic
is actually validated by our backend when the client app
programmatically registered via the dynamic OAuth2 registration
endpoints, so we refactored the validation around that code and re-used
some of it in the Cancel handling to make sure we don't allow URIs like
`javascript` and `data`, even though in theory these URIs were already
validated.
In addition, while testing this PR with
https://github.com/coder/coder-jetbrains-toolbox/pull/209 I discovered
that we are also not compliant with
https://www.rfc-editor.org/rfc/rfc6749#section-4.1.2.1 which requires
the server to attach the local state if it was provided by the client in
the original request. Also it is optional but generally a good practice
to include `error_description` in the error responses. In fact we follow
this pattern for the other types of error responses. So this is not a
one off.
- resolves#20323
<img width="1485" height="771" alt="Cancel_page_with_invalid_uri"
src="https://github.com/user-attachments/assets/5539d234-9ce3-4dda-b421-d023fc9aa99e"
/>
<img width="486" height="746" alt="Coder Toolbox handling the Cancel
button"
src="https://github.com/user-attachments/assets/acab71a6-d29c-4fa9-80ba-3c0095bbdc8f"
/>
<!--
If you have used AI to produce some or all of this PR, please ensure you
have read our [AI Contribution
guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING)
before submitting.
-->
This PR does three things:
- Exports derp expvars to the pprof endpoint
- Exports the expvar metrics as prometheus metrics in both coderd and
wsproxy
- Updates our tailscale to a fix I also had to make to avoid a data race
condition
I generated this with mux but I also manually tested that the metrics
were getting properly emitted
Add Prometheus metrics to the boundary log proxy for observability:
- batches_dropped_total (reason: buffer_full, forward_failed)
- logs_dropped_total (reason: buffer_full, forward_failed,
boundary_channel_full, boundary_batch_full)
- batches_forwarded_total
Also add BoundaryStatus to the BoundaryMessage envelope so boundary
can report dropped log counts as a separate wire message. The agent
records these as Prometheus metrics, making boundary-side data loss
visible. Backwards compatibility for older versions of boundary is maintained.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Description
- Updates `wsbuilder` to return a `BuildError` with
`http.StatusBadRequest` to signify a "validation error" on missing or
invalid parameters
- Adds a short-circuit in `prebuilds.StoreReconciler` to mark presets
for which creating a build returns a "validation error" as "validation
failed" and skip further attempts to reconcile.
- Adds a test to verify the above
- Introduces a new Prometheus metric
`coderd_prebuilt_workspaces_preset_validation_failed` to track the above
Closes: https://github.com/coder/coder/issues/21237
---------
Co-authored-by: Cian Johnston <cian@coder.com>
## Description
When multiple organizations have templates with the same name, the
Prometheus `/metrics` endpoint returns HTTP 500 because Prometheus
rejects duplicate label combinations. The three `coderd_insights_*`
metrics (`coderd_insights_templates_active_users`,
`coderd_insights_applications_usage_seconds`,
`coderd_insights_parameters`) used only `template_name` as a
distinguishing label, so two templates named e.g. `"openstack-v1"` in
different orgs would produce duplicate metric series.
This adds `organization_name` as a label to all three insight metric
descriptors to disambiguate templates across organizations.
## Changes
**`coderd/prometheusmetrics/insights/metricscollector.go`**:
- Added `organization_name` label to all three metric descriptors
- Added `organizationNames` field (template ID → org name) to the
`insightsData` struct
- In `doTick`: after fetching templates, collect unique org IDs, fetch
organizations via `GetOrganizations`, and build a
template-ID-to-org-name mapping
- In `Collect()`: pass the organization name as an additional label
value in every `MustNewConstMetric` call
**`coderd/prometheusmetrics/insights/testdata/insights-metrics.json`**:
Updated golden file to include `organization_name=coder` in all metric
label keys.
Fixes#21748
## Summary
Harden the OAuth2 provider with multiple security fixes addressing
`coder/security#121` (CSRF session takeover) and converge on OAuth 2.1
compliance.
### Security Fixes
| Fix | Description | Commits |
|-----|-------------|---------|
| **CSRF on `/oauth2/authorize`** | Enforce CSRF protection on the
authorize endpoint POST (consent form submission) | `ba7d646`, `b94a64e`
|
| **Clickjacking: `frame-ancestors` CSP** | Prevent consent page from
being iframed (`Content-Security-Policy: frame-ancestors 'none'` +
`X-Frame-Options: DENY`) | `597aeb2` |
| **Exact redirect URI matching** | Changed from prefix matching to full
string exact matching per OAuth 2.1 §4.1.2.1 | `73d64b1`, `93897f1` |
| **Store & verify `redirect_uri`** | Store redirect_uri with auth code
in DB, verify at token exchange matches exactly (RFC 6749 §4.1.3) |
`50569b9`, `d7ca315` |
| **Mandatory PKCE** | Require `code_challenge` at authorization (for
`response_type=code`) + unconditional `code_verifier` verification at
token exchange | `d7ca315`, `1cda1a9` |
| **Reject implicit grant** | `response_type=token` now returns
`unsupported_response_type` error page (OAuth 2.1 removes implicit flow)
| `d7ca315`, `91b8863` |
### Changes by File
**`coderd/httpmw/csrf.go`** — Extended the CSRF `ExemptFunc` to enforce
CSRF on `/oauth2/authorize` in addition to `/api` routes. The consent
form POST is now CSRF-protected to prevent cross-site authorization code
theft.
**`site/site.go`** — Added `Content-Security-Policy: frame-ancestors
'none'` and `X-Frame-Options: DENY` headers to `RenderOAuthAllowPage`
(consent page only — does not affect the SPA/global CSP used by AI
tasks).
**`coderd/httpapi/queryparams.go`** — Changed `RedirectURL` from prefix
matching (`strings.HasPrefix(v.Path, base.Path)`) to full URI exact
matching (`v.String() != base.String()`), comparing scheme, host, path,
and query.
**`coderd/oauth2provider/authorize.go`** — Added PKCE enforcement:
`code_challenge` is required when `response_type=code` (via a
conditional check, not `RequiredNotEmpty`, so `response_type=token` can
reach the explicit rejection path). `ShowAuthorizePage` (GET) validates
`response_type` before rendering and returns a 400 error page for
unsupported types. `ProcessAuthorize` (POST) stores the `redirect_uri`
with the auth code when explicitly provided.
**`coderd/oauth2provider/tokens.go`** — PKCE verification is now
unconditional (not gated on `code_challenge` being present in DB). If
the stored code has a `redirect_uri`, the token endpoint verifies it
matches exactly — mismatch returns `errBadCode` → `invalid_grant`.
Missing `code_verifier` returns `invalid_grant`.
**`codersdk/oauth2.go`** — `OAuth2ProviderResponseTypeToken` constant
and `Valid()` acceptance are **kept** so the authorize handler can parse
`response_type=token` and return the proper `unsupported_response_type`
error rather than failing at parameter validation.
**`coderd/database/migrations/000421_*`** — Added `redirect_uri text`
column to `oauth2_provider_app_codes`.
### Design Decisions
**`state` parameter remains optional** — The plan initially required
`state` via `RequiredNotEmpty`, but this was reverted in `376a753` to
avoid breaking existing clients. The `state` is still hashed and stored
when provided (via `state_hash` column), securing clients that opt in.
**`response_type=token` kept in `Valid()`** — Removing it from `Valid()`
would cause the parameter parser to reject the request before the
authorize handler can return the proper `unsupported_response_type`
error. The constant is kept for correct error handling flow.
**CSP scoped to consent page only** — `frame-ancestors 'none'` is set
only on the OAuth consent page renderer, not globally. The SPA/global
CSP was previously changed to allow framing for AI tasks
([#18102](https://github.com/coder/coder/pull/18102)); this change does
not regress that.
### Out of Scope (follow-up PRs)
- Bearer tokens in query strings (needs internal caller audit)
- Scope enforcement on OAuth2 tokens
- Rate limiting on dynamic client registration
---
<details>
<summary>📋 Implementation Plan</summary>
# Plan: Harden OAuth2 Provider — Security Fixes + OAuth 2.1 Compliance
## Context & Why
Security issue `coder/security#121` reports a critical session takeover
via CSRF on the OAuth2 provider. This plan covers all remaining security
fixes from that issue **plus** convergence on OAuth 2.1 requirements.
The goal is a single PR that closes all actionable gaps.
## Current State (already committed on branch `csrf-sjx1`)
| Fix | Status | Commits |
|-----|--------|---------|
| Fix 1: CSRF on `/oauth2/authorize` | ✅ Done | `ba7d646`, `b94a64e` |
| CSRF token in consent form HTML | ✅ Done | `b94a64e` |
| `state_hash` column + storage | ✅ Done (hash stored, but state still
optional) | `9167d83`, `b94a64e` |
| Tests for CSRF + state hash | ✅ Done | `e4119b5` |
## Remaining Work
### ~~Fix 2 — Require `state` parameter~~ (DROPPED)
> **Decision:** Do not enforce `state` as required. The `state`
parameter is still hashed and stored when provided (via
`hashOAuth2State` / `state_hash` column from prior commits), but clients
are not forced to supply it. This avoids breaking existing integrations
that omit state.
**Rollback:** Remove `"state"` from the `RequiredNotEmpty` call in
`coderd/oauth2provider/authorize.go:42`:
```go
// BEFORE (current on branch)
p.RequiredNotEmpty("response_type", "client_id", "state", "code_challenge")
// AFTER
p.RequiredNotEmpty("response_type", "client_id", "code_challenge")
```
No test changes needed — tests already pass `state` voluntarily.
### Fix 4 — Exact redirect URI matching
Currently `coderd/httpapi/queryparams.go:233` uses prefix matching:
```go
// CURRENT — prefix match
if v.Host != base.Host || !strings.HasPrefix(v.Path, base.Path) {
```
OAuth 2.1 requires **exact string matching**. Change to:
```go
// AFTER — exact match (OAuth 2.1 §4.1.2.1)
if v.Host != base.Host || v.Path != base.Path {
```
**File: `coderd/httpapi/queryparams.go` — `RedirectURL` method**
Also update the error message from "must be a subset of" to "must
exactly match".
**Additionally**, store `redirect_uri` with the auth code and verify at
the token endpoint (RFC 6749 §4.1.3):
1. **New migration** (same migration file or a new `000421`): Add
`redirect_uri text` column to `oauth2_provider_app_codes`
2. **Update INSERT query** in `coderd/database/queries/oauth2.sql` to
include `redirect_uri`
3. **`coderd/oauth2provider/authorize.go`**: Store
`params.redirectURL.String()` when inserting the code
4. **`coderd/oauth2provider/tokens.go`**: After retrieving the code from
DB, verify that `redirect_uri` from the token request matches the stored
value exactly. Currently `tokens.go:103` calls `p.RedirectURL(vals,
callbackURL, "redirect_uri")` for prefix validation only — it must
compare against the stored redirect_uri from the code, not just the
app's callback URL.
<details>
<summary>Why both exact match AND store+verify?</summary>
Exact matching at the authorize endpoint prevents open redirectors
(attacker can't use a sub-path).
Storing and verifying at the token endpoint prevents code injection — an
attacker who steals a code can't exchange it with a different
redirect_uri than was originally authorized. This is required by RFC
6749 §4.1.3 and OAuth 2.1.
</details>
### Fix 7 — `frame-ancestors` CSP on consent page
The consent page can be iframed by a workspace app (same-site), which is
the attack vector. Add a `Content-Security-Policy` header to prevent
framing.
**File: `site/site.go` — `RenderOAuthAllowPage` function (~line 731)**
Before writing the response, add:
```go
func RenderOAuthAllowPage(rw http.ResponseWriter, r *http.Request, data RenderOAuthAllowData) {
rw.Header().Set("Content-Type", "text/html; charset=utf-8")
// Prevent the consent page from being framed to mitigate
// clickjacking attacks (coder/security#121).
rw.Header().Set("Content-Security-Policy", "frame-ancestors 'none'")
rw.Header().Set("X-Frame-Options", "DENY")
...
```
Both headers for defense-in-depth (CSP for modern browsers,
X-Frame-Options for legacy).
### OAuth 2.1 — Mandatory PKCE
Currently PKCE is checked only when `code_challenge` was provided during
authorization (`tokens.go:258`):
```go
// CURRENT — conditional check
if dbCode.CodeChallenge.Valid && dbCode.CodeChallenge.String != "" {
// verify PKCE
}
```
OAuth 2.1 requires PKCE for ALL authorization code flows. Change to:
**File: `coderd/oauth2provider/authorize.go`** — Add `"code_challenge"`
to required params:
```go
p.RequiredNotEmpty("response_type", "client_id", "code_challenge")
```
**File: `coderd/oauth2provider/tokens.go:257-265`** — Make PKCE
verification unconditional:
```go
// AFTER — PKCE always required (OAuth 2.1)
if req.CodeVerifier == "" {
return codersdk.OAuth2TokenResponse{}, errInvalidPKCE
}
if !dbCode.CodeChallenge.Valid || dbCode.CodeChallenge.String == "" {
// Code was issued without a challenge — should not happen
// with the authorize endpoint enforcement, but defend in
// depth.
return codersdk.OAuth2TokenResponse{}, errInvalidPKCE
}
if !VerifyPKCE(dbCode.CodeChallenge.String, req.CodeVerifier) {
return codersdk.OAuth2TokenResponse{}, errInvalidPKCE
}
```
**File: `codersdk/oauth2.go`** — Remove
`OAuth2ProviderResponseTypeToken` from the enum or reject it explicitly
in the authorize handler. Currently it's defined at line 216 but the
handler ignores `response_type` and always issues a code. We should
either:
- (a) Remove the `"token"` variant from the enum and reject it with
`unsupported_response_type`, OR
- (b) Add an explicit check in `ProcessAuthorize` that rejects
`response_type=token`
Option (b) is simpler and more backwards-compatible:
```go
// In ProcessAuthorize, after extracting params:
if params.responseType != codersdk.OAuth2ProviderResponseTypeCode {
httpapi.WriteOAuth2Error(ctx, rw, http.StatusBadRequest,
codersdk.OAuth2ErrorCodeUnsupportedResponseType,
"Only response_type=code is supported")
return
}
```
### OAuth 2.1 — Bearer tokens in query strings
`coderd/httpmw/apikey.go:743` accepts `access_token` from URL query
parameters. OAuth 2.1 prohibits this. However, this may be used
internally (e.g., workspace apps, DERP). Need to audit callers before
removing.
**Approach:** This is a larger change with potential breakage. Mark as a
**separate follow-up issue** rather than including in this PR. Document
the finding.
### OAuth 2.1 — Removed flows
✅ **Already compliant.** `tokens.go` only supports `authorization_code`
and `refresh_token` grant types. The implicit grant
(`response_type=token`) will be explicitly rejected per the PKCE section
above.
### OAuth 2.1 — Refresh token rotation
✅ **Already compliant.** `tokens.go:442` deletes the old API key when a
refresh token is used.
## Migration Plan
All DB changes can go in a single new migration (or extend 000420 if the
branch is rebased before merge). Columns to add:
- `redirect_uri text` on `oauth2_provider_app_codes`
The `state_hash` column is already added by migration 000420.
## Implementation Order
1. **Fix 7** — CSP headers on consent page (isolated, no deps)
2. ~~**Fix 2** — Require `state` parameter~~ (DROPPED — state stays
optional)
3. **Fix 4** — Exact redirect URI matching + store/verify redirect_uri
4. **PKCE mandatory** — Require `code_challenge` + reject
`response_type=token`
5. **Rollback** — Remove `"state"` from `RequiredNotEmpty` in
`authorize.go`
6. **Tests** — Update/add tests for all changes
7. **`make gen`** after DB changes
## Out of Scope (separate PRs)
- Bearer tokens in query strings (needs internal caller audit)
- Scope enforcement on OAuth2 tokens
- Rate limiting / quota on dynamic client registration
</details>
---
_Generated with [`mux`](https://github.com/coder/mux) • Model:
`anthropic:claude-opus-4-6` • Thinking: `xhigh`_
## Description
This PR refactors `scripts/metricsdocgen/main.go` to support merging static and generated metrics files for documentation generation.
The static `metrics` file remains necessary for metrics not defined in the coder codebase (`go_*`, `process_*`, `promhttp_*`, `coder_aibridged_*`), as well as **edge cases** the scanner cannot handle (e.g., such as metrics with runtime-determined labels or function-local variable references for fields, ...). Handling these edge cases in the scanner would make it significantly more complex, so we keep this hybrid approach to accommodate them. This means that in such cases, developers need to update the `metrics` file directly, meaning there is still a risk of out-of-date information in the documentation. However, this solution should already encompass most cases.
Static metrics take priority over generated metrics when both files contain the same metric name, allowing manual overrides without modifying the scanner. Some of these edge cases could be easily fixed by updating the codebase to use one of the supported patterns.
## Changes
* Update `scripts/metricsdocgen/main.go` to read from two separate metrics files:
* `metrics`: static, manually maintained metrics (e.g., `go_*`, `process_*`, `promhttp_*`, `coder_aibridged_*`)
* `generated_metrics`: auto-generated by the AST scanner
* Update `metrics` file to contain only static and edge-case metrics
* Skip metrics with empty HELP descriptions in the scanner
* Update `generated_metrics` to reflect skipped metrics
* Update `docs/admin/integrations/prometheus.md` with merged metrics
Related to: https://github.com/coder/coder/issues/13223
**Disclosure:** This PR was mainly developed with Claude Sonnet 4, with iterative review and refinement by @ssncferreira
This PR adds some metrics to help identify job enqueue rates and
latencies. This work was initiated as a way to help reduce the cost of
the observation/measurement itself for autostart scaletests, which
impacts our ability to identify/reason about the load caused by
autostart. See: https://github.com/coder/internal/issues/1209
I've extended the metrics here to account for regular user initiated
builds, prebuilds, autostarts, etc. IMO there is still the question here
of whether we want to include or need the `transition` label, which is
only present on workspace builds. Including it does lead to an increase
in cardinality, and in the case of the histogram (when not using native
histograms) that's at least a few extra series for every bucket. We
could remove the transition label there but keep it on the counter.
Additionally, the histogram is currently observing latencies for other
jobs, such as template builds/version imports, those do not have a
transition type associated with them.
Tested briefly in a workspace, can see metric values like the following:
-
`coderd_workspace_builds_enqueued_total{build_reason="autostart",provisioner_type="terraform",status="success",transition="start"}
1`
-
`coderd_provisioner_job_queue_wait_seconds_bucket{build_reason="autostart",job_type="workspace_build",provisioner_type="terraform",transition="start",le="0.025"}
1`
---------
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Adds coderd_template_workspace_build_duration_seconds histogram that
tracks the full duration from workspace build creation to agent ready.
This captures the complete user-perceived build time including
provisioning and agent startup.
The metric is emitted when the agent reports ready/error/timeout via the
lifecycle API, ensuring each build is counted exactly once per replica.
## Description
Removes the following deprecated Prometheus metrics:
- `coderd_api_workspace_latest_build_total` → use
`coderd_api_workspace_latest_build` instead
- `coderd_oauth2_external_requests_rate_limit_total` → use
`coderd_oauth2_external_requests_rate_limit` instead
These metrics were deprecated in #12976 because gauge metrics should
avoid the `_total` suffix per [Prometheus naming
conventions](https://prometheus.io/docs/practices/naming/).
## Changes
- Removed deprecated metric `coderd_api_workspace_latest_build_total`
from `coderd/prometheusmetrics/prometheusmetrics.go`
- Removed deprecated metric
`coderd_oauth2_external_requests_rate_limit_total` from
`coderd/promoauth/oauth2.go`
- Updated tests to use the non-deprecated metric name
Fixes#12999
Fixes: coder/internal#767
Adds two new Prometheus metrics for license health monitoring:
- `coderd_license_warnings` - count of active license warnings
- `coderd_license_errors` - count of active license errors
Metrics endpoint after startup of a deployment with license enabled:
```
...
# HELP coderd_license_errors The number of active license errors.
# TYPE coderd_license_errors gauge
coderd_license_errors 0
...
# HELP coderd_license_warnings The number of active license warnings.
# TYPE coderd_license_warnings gauge
coderd_license_warnings 0
...
```
The user guide jumped straight into integration details without explaining
what dev containers are. Now it opens with a brief orientation linking to
the spec, then explains this guide covers the Docker-based approach.
Converted several NOTE callouts to prose where they were just cross-references
or stacked unnecessarily. The Envbuilder index note was reframed to lead with
its strengths rather than "we recommend the other thing."
Also updates platform support to Linux only per current status.
Refs #21157
Dev container admin docs were scattered across two locations: the Docker-based
integration under extending-templates/ and Envbuilder under managing-templates/.
There was no landing page explaining that two approaches exist or helping admins
choose between them.
This moves everything under admin/integrations/devcontainers/ with a decision
guide at the top. Dev containers are an integration with the dev container
specification, so integrations/ is a natural fit alongside JFrog, Vault, etc.
Stub pages remain at the original locations for discoverability.
New structure:
admin/integrations/devcontainers/
├── index.md # Landing page + decision guide
├── integration.md # Docker-based dev containers
└── envbuilder/
├── index.md
├── add-envbuilder.md
├── envbuilder-security-caching.md
└── envbuilder-releases-known-issues.md
Refs #21080
## Description
Add missing provisionerd metrics to Prometheus documentation:
* `coderd_provisionerd_num_daemons`: The number of provisioner daemons.
* `coderd_provisionerd_workspace_build_timings_seconds`: The time taken
for a workspace to build.
Related to internal thread:
https://codercom.slack.com/archives/C07GRNNRW03/p1760642020583019
## Description
This PR introduces one counter and two histograms related to workspace
creation and claiming. The goal is to provide clearer observability into
how workspaces are created (regular vs prebuild) and the time cost of
those operations.
### `coderd_workspace_creation_total`
* Metric type: Counter
* Name: `coderd_workspace_creation_total`
* Labels: `organization_name`, `template_name`, `preset_name`
This counter tracks whether a regular workspace (not created from a
prebuild pool) was created using a preset or not.
Currently, we already expose `coderd_prebuilt_workspaces_claimed_total`
for claimed prebuilt workspaces, but we lack a comparable metric for
regular workspace creations. This metric fills that gap, making it
possible to compare regular creations against claims.
Implementation notes:
* Exposed as a `coderd_` metric, consistent with other workspace-related
metrics (e.g. `coderd_api_workspace_latest_build`:
https://github.com/coder/coder/blob/main/coderd/prometheusmetrics/prometheusmetrics.go#L149).
* Every `defaultRefreshRate` (1 minute ), DB query
`GetRegularWorkspaceCreateMetrics` is executed to fetch all regular
workspaces (not created from a prebuild pool).
* The counter is updated with the total from all time (not just since
metric introduction). This differs from the histograms below, which only
accumulate from their introduction forward.
### `coderd_workspace_creation_duration_seconds` &
`coderd_prebuilt_workspace_claim_duration_seconds`
* Metric types: Histogram
* Names:
* `coderd_workspace_creation_duration_seconds`
* Labels: `organization_name`, `template_name`, `preset_name`, `type`
(`regular`, `prebuild`)
* `coderd_prebuilt_workspace_claim_duration_seconds`
* Labels: `organization_name`, `template_name`, `preset_name`
We already have `coderd_provisionerd_workspace_build_timings_seconds`,
which tracks build run times for all workspace builds handled by the
provisioner daemon.
However, in the context of this issue, we are only interested in
creation and claim build times, not all transitions; additionally, this
metric does not include `preset_name`, and adding it there would
significantly increase cardinality. Therefore, separate more focused
metrics are introduced here:
* `coderd_workspace_creation_duration_seconds`: Build time to create a
workspace (either a regular workspace or the build into a prebuild pool,
for prebuild initial provisioning build).
* `coderd_prebuilt_workspace_claim_duration_seconds`: Time to claim a
prebuilt workspace from the pool.
The reason for two separate histograms is that:
* Creation (regular or prebuild): provisioning builds with similar time
magnitude, generally expected to take longer than a claim operation.
* Claim: expected to be a much faster provisioning build.
#### Native histogram usage
Provisioning times vary widely between projects. Using static buckets
risks unbalanced or poorly informative histograms.
To address this, these metrics use [Prometheus native
histograms](https://prometheus.io/docs/specs/native_histograms/):
* First introduced in Prometheus v2.40.0
* Recommended stable usage from v2.45+
* Requires Go client `prometheus/client_golang` v1.15.0+
* Experimental and must be explicitly enabled on the server
(`--enable-feature=native-histograms`)
For compatibility, we also retain a classic bucket definition (aligned
with the existing provisioner metric:
https://github.com/coder/coder/blob/main/provisionerd/provisionerd.go#L182-L189).
* If native histograms are enabled, Prometheus ingests the
high-resolution histogram.
* If not, it falls back to the predefined buckets.
Implementation notes:
* Unlike the counter, these histograms are updated in real-time at
workspace build job completion.
* They reflect data only from the point of introduction forward (no
historical backfill).
## Relates to
Closes: https://github.com/coder/coder/issues/19528
Native histograms tested in observability stack:
https://github.com/coder/observability/pull/50
# Add OAuth2 Provider Documentation
This PR adds comprehensive documentation for the experimental OAuth2
Provider feature, which allows Coder to function as an OAuth2
authorization server. The documentation covers:
- Feature overview and experimental status warning
- Setup requirements and enabling the feature
- Methods for creating OAuth2 applications (UI and API)
- Integration patterns including standard OAuth2 and PKCE flows
- Discovery endpoints and token management
- Testing and development guidance
- Troubleshooting common issues
- Security considerations and current limitations
The documentation is marked as experimental and includes appropriate
warnings about production usage.
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Simplifies the title to reduce customer confusion as requested by
@kylejaggi.
The DX platform covers all products, not just Data Cloud. This change
makes the documentation clearer for customers who might get confused
about which DX product the integration refers to.
**Changes:**
- Updated page title from "DX Data Cloud" to "DX" in
`docs/admin/integrations/dx-data-cloud.md`
**Testing:**
- Verified the markdown renders correctly
- No functional changes, documentation-only update
---------
Co-authored-by: blink-so[bot] <211532188+blink-so[bot]@users.noreply.github.com>
Co-authored-by: bpmct <22407953+bpmct@users.noreply.github.com>
closes: #15385
- use consistent `prom-http` port (@johnstcn looks like this was
changed/added in #12214 - do we prefer `prom-http` over
`prometheus-http` or is it more important that they align?)
- add `namespaceSelector:` per @francisco-mata (thanks! - sorry it took
so long to get this in)
from issue:
> For some reason our target was not appearing on our prometheus
targets, we had to add a namespaceSelector key on the Service Monitor to
successfully appear
Co-authored-by: EdwardAngert <17991901+EdwardAngert@users.noreply.github.com>
Kyle Carberry