mirror of
https://github.com/coder/coder.git
synced 2026-10-16 14:48:56 +08:00
Nick Vigilante
149366a459
## Summary
The docs name specific experiment keys wherever an individual
experimental feature is documented, and nothing verifies those keys
still exist in the code.
When an experiment is renamed or graduated, the enablement example keeps
telling operators to set a key the server no longer accepts.
This adds `scripts/checkexperimentkeys`, which fails when a key named in
`docs/**` is absent from `codersdk.ExperimentsKnown`, and wires it into
`make lint` as `lint/check-experiment-keys`.
**The data flows one way.**
The check reads keys the docs already name and validates them against
code.
It does not generate a page, add a manifest route, or write any key into
`docs/`.
That is deliberate: #29113 retired the generated feature-stage lists
because the team decided not to advertise which features are currently
experimental or beta.
A guard against stale keys is compatible with that decision; an index is
not.
The five key references the check currently guards are the per-feature
ones that decision left in place: `chat-advisor`,
`chat-virtual-desktop`, `auto-fill-parameters`, `agent-lifecycle-hooks`,
and `mcp-server-http`.
### Recognition rules
Only explicit enablement forms are matched: `--experiments=<key-list>`
and `CODER_EXPERIMENTS=<key-list>`, including every key in a
comma-separated list.
Values may be single- or double-quoted.
Explicit shell line continuations are joined before matching, preserving
the first physical line for a finding.
Bare identifiers in prose are ignored on purpose.
A check that guesses at prose produces false positives, and a check with
false positives gets deleted by the first person it blocks.
`*`, `feature1`, `feature2`, and angle-bracketed values are recognized
placeholders.
The recognized forms are documented in the script's README.
Known gap: the space-separated `--experiments <key>` form is not
matched.
No prose uses it today, so this is latent rather than broken, and
closing it is a small follow-up if the form reappears.
### Also fixed
`ExperimentMCPToolSearch` had no case in `DisplayName()`, so it rendered
as a title-cased guess in the product UI as well as anywhere else
display names surface.
`TestExperimentDisplayNames` now pins the display name of every known
experiment against an expected map, with a length assertion, so a new
experiment cannot silently fall back to the title-cased key.
## Verification
- `go test ./scripts/checkexperimentkeys ./codersdk` passes.
Coverage includes known and unknown keys, quoted values, shell
continuations, placeholders, multi-key lists, bare-prose exclusion, and
failure-output formatting.
- `golangci-lint run ./scripts/checkexperimentkeys/... ./codersdk/...`
reports 0 issues.
- `make lint/check-experiment-keys` is clean against current `docs/`.
- Negative control: a fixture containing
`CODER_EXPERIMENTS=not-a-real-key` exits 1 with the expected finding.
- The full `pre-commit` hook passes, including
`lint/check-experiment-keys` inside aggregate `make lint`.
<details>
<summary>History: the deferred lint wiring, now resolved</summary>
This PR previously landed the tool and a standalone `make` target while
leaving the one-line addition to aggregate `make lint` for a follow-up,
and stayed draft, because three documented `oauth2` keys were stale and
owned by another PR:
- `docs/admin/integrations/oauth2-provider.md:34`
- `docs/ai-coder/mcp-server.md:108`
- `docs/ai-coder/mcp-server.md:114`
#29268 merged on 2026-09-15 and removed all three in favor of
`CODER_OAUTH2_PROVIDER_ENABLE`.
The branch is rebased onto that, the check is green against `main`, the
wiring is folded into this PR, and the merge-order constraint recorded
as CRF-5 no longer applies.
The rebase also collided with `DisplayName()`, which gained new cases on
`main`.
Resolved in favor of `main`'s switch rather than this branch's earlier
map refactor, keeping the production diff to the one missing case and
moving the exhaustiveness expectations into the test.
</details>
<details>
<summary>Why this replaced a generated reference page</summary>
DOCS-139 originally asked for a generated
`docs/reference/experiments.md` listing every key in `ExperimentsKnown`.
That was built and opened as #29311, then closed without merging.
Commit b23d73d5 (#29113, six days earlier) had removed both generated
feature-stage indexes from `docs/install/releases/feature-stages.md`,
recording that the decision, after discussion with David, was not to
advertise either list, and describing them as "drift surfaces that
promise a completeness the docs cannot maintain."
It explicitly rejected the superseded #28949, which generated the list
from `ExperimentsKnown` with descriptions parsed from Go constant
comments, for making docs track "an internal, unstable surface."
#29311 was that rejected design rebuilt.
The ticket predated the decision; the page should not have been built on
top of it.
This PR is the part of the original goal that survives: the drift
protection, without the publication.
</details>
<details>
<summary>Local pre-push note</summary>
The pre-push hook's test stage cannot run in this workspace: GNU Make
4.4.1 segfaults on the repo's `.SHELLFLAGS = $@ -ceu` configuration for
a target named `test`, tracked in DOCS-903 with a minimal reproduction.
Verified instead with the package tests, `golangci-lint`, and the full
pre-commit hook.
No `--no-verify`.
</details>
---
Generated with Coder Agents on behalf of @nickvigilante. Linear:
[DOCS-139](https://linear.app/codercom/issue/DOCS-139/validate-documented-experiment-keys-against-experimentsknown)
checkexperimentkeys
checkexperimentkeys validates experiment keys explicitly named in Markdown
under docs/ against codersdk.ExperimentsKnown. It runs as
make lint/check-experiment-keys, which is part of make lint.
Recognized forms
The check only recognizes explicit enablement examples:
--experiments=<key>CODER_EXPERIMENTS=<key>- comma-separated key lists in either form, such as
--experiments=key-one,key-two - single- or double-quoted values, such as
CODER_EXPERIMENTS="key-one" - shell line continuations that start with one of the forms above
It deliberately does not try to infer bare identifiers from prose. The
placeholder values *, the documented illustrative pair feature1 and
feature2, and any angle-bracketed value such as <experiment-key> or
<key> are ignored. Other featureN values are checked so a real stale key
is not treated as a placeholder.
Directory symlinks are not traversed. This keeps a directory scan contained to the explicitly supplied tree and prevents a documentation checkout from silently scanning Markdown outside it. A Markdown file supplied directly is still scanned according to normal file access rules.
Usage
$ go run ./scripts/checkexperimentkeys # scans docs/
$ go run ./scripts/checkexperimentkeys path/to/file.md path/to/dir