Thomas Kosiewski
12be685b59
Slice 4 of 4 for runtime experiment rules: an experimental management
API and a hidden CLI, so admins can target, widen, or kill a user-scoped
experiment at runtime without a redeploy. This PR also adds a
startup-default note on the Deployment Overview page and a docs section.
## What's in it
- **API** under `/api/experimental/experiments/rules` (behind
`apiKeyMiddleware` and `httpmw.NoStore`):
- `GET` returns, per user-scoped experiment, the startup default and the
stored rule (mode, condition, revision, actor, time). Stored rules for
other experiments follow, marked `ignored`.
- `PUT /{experiment}` takes `{mode, condition?, expected_revision}`.
- **Write path:**
- It authorizes read and update on `DeploymentConfig` before parsing
anything. A denied request gets 403 and is not audited.
- It validates next. An invalid condition returns 400 with the full CEL
diagnostics, is not audited, and leaves the rule unchanged.
- It then starts the audit and calls `experiments.WriteRule`. A stale
revision returns 409 and a transaction error returns 500; both are
audited with an empty diff.
- An identical write returns 200, creates no audit entry, and keeps the
revision.
- The audit export happens after commit. If it fails, that is logged and
the committed change stands (the existing audit contract).
- **SDK:** `ExperimentalClient.ExperimentRules` and
`ExperimentalClient.PutExperimentRule`.
- **CLI:** hidden `coder exp experiment-rules list|on|off|set|reset`.
- Write commands take `--expected-revision`. Without it they read the
current revision first, which only protects the gap between the two
calls; the help text says so.
- A 409 prints the current rule and exits non-zero, with no retry.
- `reset` warns when the startup default is on, because reset is not a
kill switch.
- **Deployment Overview:** a note says the Experiments values are
startup defaults that runtime rules can override per user.
- **Docs:** a new "Target experiments at runtime" section in
`docs/reference/feature-stages.md` covers the modes, roles, CEL fields
and limits, reset vs. off, revisions and audit, the kill-switch
guarantee (including the rolling-upgrade requirement), frontend
freshness, fail-closed behavior, and that rules are not an authorization
boundary.
## Testing
- **HTTP:** every mode applies on the next request with no restart;
rejected writes keep the rule; RBAC (members 403 on GET and PUT;
auditors can read but not write; an invalid body from an unauthorized
caller still gets 403); malformed and ignored stored rows.
- **HA:** two coderd instances share one database. A PUT through A is
seen by B's `/api/v2/experiments` and by B's evaluator, which is the one
chatd uses.
- **Audit (enterprise, licensed):** an effective change creates one
entry whose diff carries the condition, and the entry is found before
the diff is checked. No-ops, 403s and 400s create no entry. A 409 and an
injected 500 are audited with an empty diff. A failed export keeps the
change.
- **Redaction:** a sentinel condition appears only in the rules API,
audit entries, and the server log's audit lines (Enterprise's slog audit
backend writes every audit diff there). It is absent from every other
log line, `/deployment/config`, the support bundle, and telemetry. The
test wires the slog audit backend so this is checked as stated.
- **CLI:** write and list, a conflict that is not retried, and the reset
warning.
Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- the CLI;
- per-user results in the browser;
- 403 for members, including an invalid body;
- group targeting, including removing the user from the group;
- sentinel redaction across logs, `/deployment/config`, the support
bundle and the audit API (at that head the audit still masked the
condition; it is now tracked);
- the audit page showing the condition as changed but masked (same
note);
- the Overview note;
- the static default with `off` and `reset`;
- chatd enforcement toggled between turns.
That round tested `135aaaa41e`. The later commits are a CLI refactor and
a test addition, and a follow-up round will cover the final head.
## Follow-ups (low, not in this PR)
- A broken rule logs one error line per evaluation. Consider
rate-limiting it per experiment and revision.
- There's no API or CLI path to clear a stored rule for an experiment
that isn't user-scoped; those rules are inert and listed as `ignored`.
- A malformed stored rule is listed with zero `updated_by` and
`updated_at` values instead of omitting them.
- CLI wording: the 409 message, conflict output showing a UUID instead
of a username, a no-op write not saying nothing changed, and the parent
help lacking the `user.*` fields.
- `PutExperimentRuleRequest.mode` is not in the Swagger `required` list.
An omitted mode already returns 400.
- The dashboard's experiments query (from #29964) refreshes on mount,
window focus or reconnect once 60 seconds old, not on a timer, and its
comment says it bounds staleness. Correct the comment, or add a refetch
interval if the UI must pick up rule changes without a reload.
## Stack
1. #29869: evaluator library
2. #29898: storage and audit types
3. #29964: enforcement and discovery
4. **This PR:** control plane
> [!NOTE]
> This PR was written by an AI agent (Xum).
---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
---------
Signed-off-by: Thomas Kosiewski <tk@coder.com>
12 KiB
Feature stages
Some Coder features are released in feature stages before they are generally available.
If you encounter an issue with any Coder feature, please submit a GitHub issue or join the Coder Discord.
Feature stages
| Feature stage | Stable | Production-ready | Support | Description |
|---|---|---|---|---|
| Early Access | No | No | GitHub issues | For staging only. Not feature-complete or stable. Disabled by default. |
| Beta | No | Not fully | Docs, Discord, GitHub | Publicly available. In active development with minor bugs. Suitable for staging; optional for production. Not covered by SLA. |
| GA | Yes | Yes | License-based | Stable and tested. Enabled by default. Fully documented. Support based on license. |
Early access features
- Stable: No
- Production-ready: No
- Support: GitHub issues
Early access features are neither feature-complete nor stable. We do not recommend using early access features in production deployments.
Coder sometimes releases early access features that are available for use, but are disabled by default. You shouldn't use early access features in production because they might cause performance or stability issues. Early access features can be mostly feature-complete, but require further internal testing and remain in the early access stage for at least one month.
Coder may make significant changes or revert features to a feature flag at any time.
If you plan to activate an early access feature, we suggest that you use a staging deployment.
To enable early access features:
Use the Coder CLI --experiments flag to enable early
access features:
-
Enable all early access features:
coder server --experiments=* -
Enable multiple early access features:
coder server --experiments=feature1,feature2
You can also use the CODER_EXPERIMENTS
environment variable.
You can opt-out of a feature after you've enabled it.
Target experiments at runtime
Some early access features accept runtime rules.
A rule turns the experiment on or off for every user, or targets it to some users, without a restart.
Only user-scoped experiments accept rules.
Other experiments, such as no_nats_pubsub or workspace-capable-licensing, read only the startup --experiments list and need a restart to change.
Note
The rules API and the hidden
coder exp experiment-rulescommand are experimental. They can change or be removed without notice. There is no UI editor yet.
You need the Owner role to change a rule. The Auditor role can list rules.
Run coder exp experiment-rules list to show each user-scoped experiment, its startup default, and its rule.
The list also shows stored rules that target other experiments as ignored, because they have no effect.
Rule modes
Each user-scoped experiment has at most one rule:
| Rule | Effect | Command |
|---|---|---|
No rule, or inherit |
Uses the startup --experiments default |
coder exp experiment-rules reset <experiment> |
on |
On for every user | coder exp experiment-rules on <experiment> |
off |
Off for every user, even if the experiment is enabled at startup | coder exp experiment-rules off <experiment> |
condition |
On for users whose CEL condition is true, off otherwise | coder exp experiment-rules set <experiment> <condition> |
For example, to turn on an experiment for users in one group, widen it to every member of the coder organization, and then turn it off for everyone:
coder exp experiment-rules set mcp-tool-search '"coder/beta-testers" in user.groups'
coder exp experiment-rules set mcp-tool-search '"coder/Everyone" in user.groups'
coder exp experiment-rules off mcp-tool-search
Reset doesn't turn an experiment off.
It restores the startup default, which can enable the experiment for every user.
The command prints a warning when that happens.
To turn an experiment off for everyone, use off.
Each change increases the rule's revision.
Changing commands accept --expected-revision and fail when the stored rule has a different revision.
Without the flag, the command reads the current revision and writes against it, which protects only against a change made between that read and the write.
The command never retries a conflict: it prints the current rule and exits with an error.
If you use audit logs, each change creates an entry with the experiment_rule resource type.
The entry records who changed the rule, when, and the old and new condition.
Audit lines in the server log carry the same diff.
Condition variables
A condition is a CEL expression that returns a boolean.
It can read one variable, user, with these fields:
| Field | Type | Value |
|---|---|---|
user.id |
string | The user's ID |
user.username |
string | The user's username |
user.email |
string | The user's email address, compared case-sensitively |
user.roles |
list(string) | The user's explicit site roles, such as owner. The implied member role and organization roles are not included. |
user.organizations |
list(string) | Names of the organizations the user belongs to |
user.groups |
list(string) | Groups the user belongs to, as <organization>/<group>, including <organization>/Everyone |
A condition can be at most 4,096 characters long, and each evaluation has a CEL cost limit of 10,000. Coder rejects a condition that fails to compile and returns the error to the person who wrote it. Log lines about a failed condition record only the experiment, the rule revision, the error category, and the line and column, never the condition text.
When a change takes effect
On supporting replicas, a fresh authoritative database read after the update commits observes the new rule. In-flight evaluations and work already using an earlier decision are not cancelled. All serving replicas must run a supporting version before you rely on this control.
Replicas that run an earlier version ignore rules and use their startup --experiments list.
Every replica also falls back to its own startup list for experiments without a rule or with inherit, so give all replicas the same --experiments value.
The dashboard fetches the enabled experiments again when a page loads, when the browser window regains focus, and when the connection is restored, if its copy is more than 60 seconds old. It doesn't refresh on a timer, so a tab that stays focused can show an out-of-date view. The server applies the current rule to each request regardless of what the dashboard shows.
Failure behavior
If Coder can't read the rules, a stored rule is malformed, or a condition fails to compile or evaluate, the experiment is off for that decision, even if it is enabled at startup. You can't change this behavior.
Rules gate features; they are not an authorization boundary. Don't use a rule to restrict access to data or actions.
Beta
- Stable: No
- Production-ready: Not fully
- Support: Documentation, Discord, and GitHub issues
Beta features are open to the public and are tagged with a Beta label.
They’re in active development and subject to minor changes. They might contain minor bugs, but are generally ready for use.
Beta features are often ready for general availability within two-three releases. You should test beta features in staging environments. You can use beta features in production, but should set expectations and inform users that some features may be incomplete.
We keep documentation about beta features up-to-date with the latest information, including planned features, limitations, and workarounds. If you encounter an issue, please contact your Coder account team, reach out on Discord, or create a GitHub issues if there isn't one already. While we will do our best to provide support with beta features, most issues will be escalated to the product team. Beta features are not covered within service-level agreements (SLA).
Most beta features are enabled by default. Beta features are announced through the Coder Changelog, and more information is available in the documentation.
General Availability (GA)
- Stable: Yes
- Production-ready: Yes
- Support: Yes, based on license.
All features that are not explicitly tagged as Early access or Beta are considered generally available (GA).
They have been tested, are stable, and are enabled by default.
If your Coder license includes an SLA, please consult it for an outline of specific expectations.
For support, consult our knowledgeable and growing community on Discord, or create a GitHub issue if one doesn't exist already. Customers with a valid Coder license, can submit a support request or contact your account team.
We intend Coder documentation to be the single source of truth and all features should have some form of complete documentation that outlines how to use or implement a feature. If you discover an error or if you have a suggestion that could improve the documentation, please submit a GitHub issue.
Some GA features can be disabled for air-gapped deployments. Consult the feature's documentation or submit a support ticket for assistance.