This PR adds an opinionated harness-engineering layer for agent-driven
workflows: a small set of agent-readable docs, mechanical structure
checks, structured CI failure summaries, an architecture-lint umbrella,
and per-worktree dev-server isolation. The goal is to make local dev,
tests, and CI mechanically inspectable by agents without changing app
runtime behavior.
## What landed
**Agent docs and navigation**
- `.claude/docs/OBSERVABILITY.md`, `.claude/docs/DEV_ISOLATION.md`,
`.claude/docs/AGENT_FAILURES.md`: task-oriented guides for logs,
tracing, Prometheus, dev-server isolation, and a seeded failure catalog.
- `AGENTS.md`: added an `Agent navigation` block, then trimmed the file
from 375 to 229 lines by migrating duplicated detail into
`WORKFLOWS.md`, `GO.md`, `TESTING.md`, and `DATABASE.md`. The
user-managed custom-instructions block is preserved.
- `.agents/docs`: symlink mirror of `.claude/docs` for agent runtimes
that look under `.agents`.
**Mechanical checks**
- `scripts/check_agents_structure.sh`: validates `@...` references in
tracked `AGENTS.md` files and warns when root grows past 600 lines.
Wired as `make lint/agents` and into `make lint`.
- `scripts/audit-agent-readiness.sh`: report-first audit of harness
readiness. Currently `10 ok, 0 warn, 0 fail`.
- `scripts/check_architecture.sh` / `make lint/architecture`: umbrella
architecture-lint target. Consolidates the existing
`check_enterprise_imports.sh` and `check_codersdk_imports.sh` so they
run exactly once via the umbrella. Slot is open for new high-confidence
rules.
**Structured CI failure summaries**
- `scripts/playwright-failure-summary.sh`: parses
`site/test-results/results.json` and writes Markdown to
`$GITHUB_STEP_SUMMARY` on failure. Wired into the `test-e2e` matrix job.
- `scripts/go-test-failure-summary.sh`: parses `go test -json`
line-delimited output the same way. Wired into `test-go-pg`,
`test-go-pg-17`, and `test-go-race-pg` by injecting `gotestsum
--jsonfile` in the workflow without touching `Makefile`. JSON also
uploaded as a CI artifact on failure.
- `site/e2e/playwright.config.ts`: enables `screenshot:
only-on-failure`, `trace: retain-on-failure`, JSON reporter, and HTML
reporter alongside existing reporters.
- `.github/workflows/ci.yaml`: failure artifact uploads for Playwright
now use `if: failure()` and predictable names
(`playwright-artifacts-<variant>-<sha>`).
**Per-worktree dev-server isolation** (`scripts/develop/main.go`)
- Deterministic FNV-64a hash of the worktree path produces a port offset
in `[0, 1000)` (50 buckets, step 20 to avoid API/proxy overlap across
adjacent buckets).
- Offset is applied only to defaults; both env vars (`CODER_DEV_PORT`,
`CODER_DEV_WEB_PORT`, `CODER_DEV_PROXY_PORT`,
`CODER_DEV_PROMETHEUS_PORT`) and CLI flags retain priority.
- Hardcoded ports `9090` (embedded Prometheus UI) and `12345` (Delve)
are unchanged by design.
- Startup banner shows each port's source: `default`, `offset`, or
`explicit`.
- Unit tests in `scripts/develop/main_test.go` cover determinism,
bounds, no-overlap across the four ports, and explicit-skip behavior.
- State (`.coderv2/`) was already worktree-isolated via `os.Getwd()`, so
no state-dir changes were needed.
## Validation
`make lint/agents`, `make lint/architecture`, `make lint/emdash`, `bash
scripts/audit-agent-readiness.sh` (10 ok, 0 warn, 0 fail), `shellcheck`
on all 5 new scripts, `go test ./scripts/develop/...`, and `js-yaml`
parse of `ci.yaml` all pass. Synthetic fixtures verify both
failure-summary scripts handle empty/missing input (silent exit 0),
ANSI-stripped output, and parent/subtest formatting.
## Known follow-ups (deferred)
- Frontend Storybook/Vitest failure summary: lowest-leverage slice of
the failure-summary work. Skipping until observed pain.
- Architecture lint currently only delegates to existing import checks;
new rules (`InTx` outer-store detection, swagger-annotation lint) plug
in as needed.
- 50 port-offset buckets means two worktree paths can occasionally
collide. The DEV_ISOLATION doc tells users to set the relevant env var
when this happens.
> Mux opened this PR on Mike's behalf.
The Playwright e2e `webServer` starts the Coder server via
`go run -tags embed`, which must compile before serving. The default 60s
timeout leaves no margin when the CI runner is slow.
Failed run:
https://github.com/coder/coder/actions/runs/22782592241/job/66091950715
Successful run:
https://github.com/coder/coder/actions/runs/22782107623/job/66090828826
The server started and printed its banner, but with only ~4s left on the
clock the health check (`/api/v2/deployment/config`) could not complete
before the timeout fired. The same ~2x slowdown shows in the
`make site/e2e/bin/coder` step (45s vs 67s), confirming this is runner
performance variability.
Increase timeout to 120s.
Refs #22727
This PR introduces failing test retries in CI for e2e tests, Go tests
with the in-memory database, Go tests with Postgres, and the CLI tests.
Retries are not enabled for race tests.
The goal is to reduce how often flakes disrupt developers' workflows.
Using `go run` inside of a test is fragile, because it means we have to
wait for `go` to compile the binary while also constrained on resources
by the fact that Playwright and coderd are already running. We should
instead compile a coder binary for the current platform before the tests
and use it directly.
Closescoder/internal#168
Gets rid of the "global state" authentication, and adds a `login` helper
which should be called at the beginning of each test. This means that
not every test needs to authenticated as admin, and we can even have
tests that encompass multiple permission levels.
We also now create more than just the single admin user during setup, so
that we can have a set of users to pick from as appropriate.
* chore: allow terraform & echo built-in provisioners
Built-in provisioners serve all specified types. This allows running terraform, echo, or both in built in.
The cli flag to control the types is hidden by default, to be used primarily for testing purposes.
This PR solves #10478 by auto-filling previously used template values in create and update workspace flows.
I decided against explicit user values in settings for these reasons:
* Autofill is far easier to implement
* Users benefit from autofill _by default_ — we don't need to teach them new concepts
* If we decide that autofill creates more harm than good, we can remove it without breaking compatibility
* feat: support configurable web terminal rendering
- Added a deployment option for configuring web terminal rendering.
Valid values are 'webgl', 'canvas', and 'dom'.
* feat: allow external services to be authable
* Refactor external auth config structure for defaults
* Add support for new config properties
* Change the name of external auth
* Move externalauth -> external-auth
* Run gen
* Fix tests
* Fix MW tests
* Fix git auth redirect
* Fix lint
* Fix name
* Allow any ID
* Fix invalid type test
* Fix e2e tests
* Fix comments
* Fix colors
* Allow accepting any type as string
* Run gen
* Fix href
* feat: Add tunnel by default
If an access URL is not specified, we will always tunnel.
This is from community-member feedback who exclaimed that
it's confusing having the default for `coder server` display
a warning message, and I agree.
There is very little (maybe none) in running `coder server`
without tunnel and without an access URL, so this seems like
overall a much better UX.
* Update install.sh
Co-authored-by: Ben Potter <ben@coder.com>
* Update docs/install/packages.md
Co-authored-by: Ben Potter <ben@coder.com>
* Fix reset pass test
* Fix e2e test
Co-authored-by: Ben Potter <ben@coder.com>
Michael Suchacz