feat: add harness engineering layer for agent workflows (#24791)

This PR adds an opinionated harness-engineering layer for agent-driven workflows: a small set of agent-readable docs, mechanical structure checks, structured CI failure summaries, an architecture-lint umbrella, and per-worktree dev-server isolation. The goal is to make local dev, tests, and CI mechanically inspectable by agents without changing app runtime behavior. ## What landed **Agent docs and navigation** - `.claude/docs/OBSERVABILITY.md`, `.claude/docs/DEV_ISOLATION.md`, `.claude/docs/AGENT_FAILURES.md`: task-oriented guides for logs, tracing, Prometheus, dev-server isolation, and a seeded failure catalog. - `AGENTS.md`: added an `Agent navigation` block, then trimmed the file from 375 to 229 lines by migrating duplicated detail into `WORKFLOWS.md`, `GO.md`, `TESTING.md`, and `DATABASE.md`. The user-managed custom-instructions block is preserved. - `.agents/docs`: symlink mirror of `.claude/docs` for agent runtimes that look under `.agents`. **Mechanical checks** - `scripts/check_agents_structure.sh`: validates `@...` references in tracked `AGENTS.md` files and warns when root grows past 600 lines. Wired as `make lint/agents` and into `make lint`. - `scripts/audit-agent-readiness.sh`: report-first audit of harness readiness. Currently `10 ok, 0 warn, 0 fail`. - `scripts/check_architecture.sh` / `make lint/architecture`: umbrella architecture-lint target. Consolidates the existing `check_enterprise_imports.sh` and `check_codersdk_imports.sh` so they run exactly once via the umbrella. Slot is open for new high-confidence rules. **Structured CI failure summaries** - `scripts/playwright-failure-summary.sh`: parses `site/test-results/results.json` and writes Markdown to `$GITHUB_STEP_SUMMARY` on failure. Wired into the `test-e2e` matrix job. - `scripts/go-test-failure-summary.sh`: parses `go test -json` line-delimited output the same way. Wired into `test-go-pg`, `test-go-pg-17`, and `test-go-race-pg` by injecting `gotestsum --jsonfile` in the workflow without touching `Makefile`. JSON also uploaded as a CI artifact on failure. - `site/e2e/playwright.config.ts`: enables `screenshot: only-on-failure`, `trace: retain-on-failure`, JSON reporter, and HTML reporter alongside existing reporters. - `.github/workflows/ci.yaml`: failure artifact uploads for Playwright now use `if: failure()` and predictable names (`playwright-artifacts-<variant>-<sha>`). **Per-worktree dev-server isolation** (`scripts/develop/main.go`) - Deterministic FNV-64a hash of the worktree path produces a port offset in `[0, 1000)` (50 buckets, step 20 to avoid API/proxy overlap across adjacent buckets). - Offset is applied only to defaults; both env vars (`CODER_DEV_PORT`, `CODER_DEV_WEB_PORT`, `CODER_DEV_PROXY_PORT`, `CODER_DEV_PROMETHEUS_PORT`) and CLI flags retain priority. - Hardcoded ports `9090` (embedded Prometheus UI) and `12345` (Delve) are unchanged by design. - Startup banner shows each port's source: `default`, `offset`, or `explicit`. - Unit tests in `scripts/develop/main_test.go` cover determinism, bounds, no-overlap across the four ports, and explicit-skip behavior. - State (`.coderv2/`) was already worktree-isolated via `os.Getwd()`, so no state-dir changes were needed. ## Validation `make lint/agents`, `make lint/architecture`, `make lint/emdash`, `bash scripts/audit-agent-readiness.sh` (10 ok, 0 warn, 0 fail), `shellcheck` on all 5 new scripts, `go test ./scripts/develop/...`, and `js-yaml` parse of `ci.yaml` all pass. Synthetic fixtures verify both failure-summary scripts handle empty/missing input (silent exit 0), ANSI-stripped output, and parent/subtest formatting. ## Known follow-ups (deferred) - Frontend Storybook/Vitest failure summary: lowest-leverage slice of the failure-summary work. Skipping until observed pain. - Architecture lint currently only delegates to existing import checks; new rules (`InTx` outer-store detection, swagger-annotation lint) plug in as needed. - 50 port-offset buckets means two worktree paths can occasionally collide. The DEV_ISOLATION doc tells users to set the relevant env var when this happens. > Mux opened this PR on Mike's behalf.
2026-09-24 15:04:27 +08:00 · 2026-05-11 17:27:29 +02:00
parent 915956460a
commit 85792d08bc
20 changed files with 1561 additions and 237 deletions
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# shellcheck source=scripts/lib.sh
+# shellcheck disable=SC1091
+source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
+cdroot
+
+usage() {
+	cat <<'USAGE'
+Usage: scripts/audit-agent-readiness.sh [--help]
+
+Print a report-first audit of agent harness readiness. Warnings identify
+aspirational checks and do not fail the script. Missing required harness docs
+fail the script. Run manually with:
+
+  bash scripts/audit-agent-readiness.sh
+USAGE
+}
+
+if [[ "${1:-}" == "--help" ]]; then
+	usage
+	exit 0
+fi
+
+ok_count=0
+warn_count=0
+fail_count=0
+
+ok() {
+	printf '[ok] %s\n' "$1"
+	((ok_count++)) || true
+}
+
+warn() {
+	printf '[warn] %s\n' "$1"
+	((warn_count++)) || true
+}
+
+fail() {
+	printf '[fail] %s\n' "$1"
+	((fail_count++)) || true
+}
+
+contains() {
+	local file="$1"
+	local pattern="$2"
+	grep -qiE "$pattern" "$file"
+}
+
+echo "Agent harness readiness audit"
+echo
+echo "Required harness docs"
+
+for doc in \
+	".claude/docs/OBSERVABILITY.md" \
+	".claude/docs/DEV_ISOLATION.md" \
+	".claude/docs/AGENT_FAILURES.md"; do
+	if [[ -f "$doc" ]]; then
+		ok "$doc exists."
+	else
+		fail "$doc is missing."
+	fi
+done
+
+if [[ -L ".agents/docs" ]]; then
+	agents_docs_target="$(readlink ".agents/docs")"
+	if [[ "$agents_docs_target" == "../.claude/docs" ]]; then
+		ok ".agents/docs points to .claude/docs."
+	else
+		fail ".agents/docs points to $agents_docs_target, expected ../.claude/docs."
+	fi
+else
+	fail ".agents/docs compatibility symlink is missing."
+fi
+
+echo
+echo "Navigation and report-first checks"
+
+if contains AGENTS.md '^##[[:space:]].*(Agent navigation|Where to look)' ||
+	{ grep -qF ".claude/docs/OBSERVABILITY.md" AGENTS.md &&
+		grep -qF ".claude/docs/DEV_ISOLATION.md" AGENTS.md &&
+		grep -qF ".claude/docs/AGENT_FAILURES.md" AGENTS.md; }; then
+	ok "Root AGENTS.md appears to include agent navigation."
+else
+	warn "Root AGENTS.md may be missing agent navigation."
+fi
+
+if contains site/e2e/playwright.config.ts 'screenshot' &&
+	contains site/e2e/playwright.config.ts 'video' &&
+	contains site/e2e/playwright.config.ts 'trace' &&
+	contains site/e2e/playwright.config.ts 'failure'; then
+	ok "Playwright failure artifact settings appear configured."
+else
+	warn "Playwright failure artifact settings were not all detected."
+fi
+
+if grep -qi "playwright" .github/workflows/ci.yaml &&
+	grep -q "upload-artifact" .github/workflows/ci.yaml &&
+	grep -qF "failure()" .github/workflows/ci.yaml; then
+	ok "E2E CI failure artifact upload appears configured."
+else
+	warn "E2E CI failure artifact upload was not detected."
+fi
+
+if contains .claude/docs/OBSERVABILITY.md 'Prometheus' &&
+	contains .claude/docs/OBSERVABILITY.md 'log'; then
+	ok "Observability doc mentions logs and Prometheus."
+else
+	warn "Observability doc may be missing logs or Prometheus coverage."
+fi
+
+if contains .claude/docs/DEV_ISOLATION.md 'port' &&
+	contains .claude/docs/DEV_ISOLATION.md 'CODER_DEV|override'; then
+	ok "Development isolation doc mentions ports and overrides."
+else
+	warn "Development isolation doc may be missing ports or override coverage."
+fi
+
+if grep -q 'lint/architecture' Makefile; then
+	ok "Architecture lint target exists."
+else
+	warn "Architecture lint target is not present yet."
+fi
+
+echo
+printf 'Summary: %d ok, %d warn, %d fail.\n' "$ok_count" "$warn_count" "$fail_count"
+
+if ((fail_count > 0)); then
+	exit 1
+fi
@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# shellcheck source=scripts/lib.sh
+# shellcheck disable=SC1091
+source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
+cdroot
+
+echo "--- check agent docs structure"
+
+required_docs=(
+	".claude/docs/OBSERVABILITY.md"
+	".claude/docs/DEV_ISOLATION.md"
+	".claude/docs/AGENT_FAILURES.md"
+)
+
+fail=0
+
+for doc in "${required_docs[@]}"; do
+	if [[ ! -f "$doc" ]]; then
+		echo "error: required harness doc is missing: $doc"
+		fail=1
+	fi
+done
+
+if [[ ! -L ".agents/docs" ]]; then
+	echo "error: agent docs compatibility symlink is missing: .agents/docs -> ../.claude/docs"
+	fail=1
+elif [[ "$(readlink ".agents/docs")" != "../.claude/docs" ]]; then
+	echo "error: agent docs compatibility symlink points to $(readlink ".agents/docs"), expected ../.claude/docs"
+	fail=1
+fi
+
+is_reference_path() {
+	local ref="$1"
+	case "$ref" in
+	*/* | package.json | AGENTS.local.md)
+		return 0
+		;;
+	*)
+		return 1
+		;;
+	esac
+}
+
+# TODO: Add circular AGENTS.md include detection if nested agent docs begin
+# referencing each other. Current checks validate file existence only.
+mapfile -t agent_files < <(git ls-files '*AGENTS.md' | sort)
+
+for agent_file in "${agent_files[@]}"; do
+	agent_dir="$(dirname "$agent_file")"
+	while IFS=$'\t' read -r line_number ref; do
+		if [[ -z "${line_number:-}" || -z "${ref:-}" ]]; then
+			continue
+		fi
+		if ! is_reference_path "$ref"; then
+			continue
+		fi
+
+		candidate="$agent_dir/$ref"
+		candidate="${candidate#./}"
+		if [[ -e "$candidate" ]]; then
+			continue
+		fi
+
+		if [[ "$(basename "$ref")" == "AGENTS.local.md" ]]; then
+			echo "warning: $agent_file:$line_number: optional local agent file is not present: $ref"
+			continue
+		fi
+
+		echo "error: $agent_file:$line_number: referenced file does not exist: $ref"
+		fail=1
+	done < <(
+		awk '
+			/^[[:space:]]*(-[[:space:]]+)?@/ {
+				ref = $0
+				sub(/^[[:space:]]*(-[[:space:]]+)?@/, "", ref)
+				sub(/[[:space:]`)>].*$/, "", ref)
+				sub(/[,:;)]+$/, "", ref)
+				print FNR "\t" ref
+			}
+		' "$agent_file"
+	)
+done
+
+if [[ -f AGENTS.md ]]; then
+	root_agent_lines=$(wc -l <AGENTS.md)
+	if ((root_agent_lines > 600)); then
+		echo "warning: AGENTS.md is $root_agent_lines lines, consider keeping the root guide concise."
+	fi
+fi
+
+if [[ "$fail" -ne 0 ]]; then
+	exit 1
+fi
+
+echo "OK: agent docs structure looks valid."
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# Umbrella architecture-boundary check.
+#
+# Delegates to existing import-boundary scripts. New architecture rules can be
+# added here as needed.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+echo "--- check architecture (import boundaries)"
+
+"$SCRIPT_DIR/check_enterprise_imports.sh"
+"$SCRIPT_DIR/check_codersdk_imports.sh"
+
+echo "OK: architecture checks passed."
@@ -10,6 +10,7 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"hash/fnv"
 	"net"
 	"net/http"
 	"net/url"
@@ -52,7 +53,13 @@ const (
 	prometheusContainerName = "coder-prometheus"
 	// defaultPrometheusPort avoids 2112 (agent prometheus) and
 	// 2113 (agent debug) already bound inside Coder workspaces.
-	defaultPrometheusPort  = "2114"
+	defaultPrometheusPort = "2114"
+	// portOffsetBuckets keeps the offset below 1000 while leaving
+	// enough hash buckets for common multi-worktree use.
+	portOffsetBuckets = 50
+	// portOffsetStep avoids overlap between the default API and proxy
+	// ports when two worktrees land in adjacent buckets.
+	portOffsetStep         = 20
 	prometheusImage        = "prom/prometheus:v3.11.2"
 	defaultAccessURL       = "http://127.0.0.1:%d"
 	defaultPassword        = "SomeSecurePassword!"
@@ -96,6 +103,13 @@ func main() {
 				Description: "Prometheus metrics port. Set to 0 to disable.",
 				Value:       serpent.Int64Of(&cfg.coderMetricsPort),
 			},
+			{
+				Flag:        "port-offset",
+				Env:         "CODER_DEV_PORT_OFFSET",
+				Default:     "false",
+				Description: "Apply a deterministic per-worktree offset to default API, web, proxy, and Coder metrics ports. Useful when running multiple worktrees in parallel.",
+				Value:       serpent.BoolOf(&cfg.portOffsetEnabled),
+			},
 			{
 				Flag:        "prometheus-server",
 				Env:         "CODER_DEV_PROMETHEUS_SERVER",
@@ -171,12 +185,13 @@ func main() {
 		},
 		Handler: func(inv *serpent.Invocation) error {
 			cfg.serverExtraArgs = inv.Args
+			cfg.portExplicit = portExplicitFromInvocation(inv)

 			logger := slog.Make(sloghuman.Sink(inv.Stderr))
-			if err := cfg.validate(); err != nil {
+			if err := cfg.resolveEnv(); err != nil {
 				return err
 			}
-			if err := cfg.resolveEnv(); err != nil {
+			if err := cfg.validate(); err != nil {
 				return err
 			}
 			return develop(inv.Context(), logger, &cfg)
@@ -191,30 +206,123 @@ func main() {
 }

 type devConfig struct {
-	apiPort          int64
-	webPort          int64
-	proxyPort        int64
-	coderMetricsPort int64
-	prometheusServer bool
-	agpl             bool
-	accessURL        string
-	password         string
-	useProxy         bool
-	debug            bool
-	skipSetup        bool
-	multiOrg         bool
-	starterTemplate  string
-	dbRollback       bool
-	dbReset          bool
-	dbContinue       bool
-	projectRoot      string
-	binaryPath       string
-	configDir        string
-	childEnv         []string
+	apiPort           int64
+	webPort           int64
+	proxyPort         int64
+	coderMetricsPort  int64
+	portOffsetEnabled bool
+	prometheusServer  bool
+	agpl              bool
+	accessURL         string
+	password          string
+	useProxy          bool
+	debug             bool
+	skipSetup         bool
+	multiOrg          bool
+	starterTemplate   string
+	dbRollback        bool
+	dbReset           bool
+	dbContinue        bool
+	projectRoot       string
+	binaryPath        string
+	configDir         string
+	childEnv          []string
+	portExplicit      portExplicit
+	portOffset        int
+	apiPortSource     portSource
+	webPortSource     portSource
+	proxyPortSource   portSource
+	metricsPortSource portSource
 	// Extra args after flags forwarded to "coder server".
 	serverExtraArgs []string
 }

+type portExplicit struct {
+	api     bool
+	web     bool
+	proxy   bool
+	metrics bool
+}
+
+type portSource string
+
+const (
+	portSourceDefault  portSource = "default"
+	portSourceExplicit portSource = "explicit"
+	portSourceOffset   portSource = "offset"
+)
+
+func portExplicitFromInvocation(inv *serpent.Invocation) portExplicit {
+	return portExplicit{
+		api:     isPortExplicit(inv, "port", "CODER_DEV_PORT"),
+		web:     isPortExplicit(inv, "web-port", "CODER_DEV_WEB_PORT"),
+		proxy:   isPortExplicit(inv, "proxy-port", "CODER_DEV_PROXY_PORT"),
+		metrics: isPortExplicit(inv, "prometheus-port", "CODER_DEV_PROMETHEUS_PORT"),
+	}
+}
+
+func isPortExplicit(inv *serpent.Invocation, flagName, envName string) bool {
+	if flag := inv.ParsedFlags().Lookup(flagName); flag != nil && flag.Changed {
+		return true
+	}
+	if val, ok := inv.Environ.Lookup(envName); ok && val != "" {
+		return true
+	}
+	for _, opt := range inv.Command.Options {
+		if opt.Flag == flagName {
+			return opt.ValueSource == serpent.ValueSourceFlag ||
+				opt.ValueSource == serpent.ValueSourceEnv
+		}
+	}
+	return false
+}
+
+// portOffset returns a deterministic offset in [0, 1000) derived from the
+// worktree path. Successive callers with the same projectRoot get the same
+// offset; different projectRoots get different offsets with high probability.
+func portOffset(projectRoot string) int {
+	h := fnv.New64a()
+	_, _ = h.Write([]byte(projectRoot))
+	bucket := h.Sum64() % uint64(portOffsetBuckets)
+	return int(bucket) * portOffsetStep //nolint:gosec // Bucket is less than portOffsetBuckets.
+}
+
+func (c *devConfig) applyPortOffset() {
+	c.portOffset = 0
+	if !c.portOffsetEnabled {
+		return
+	}
+	c.portOffset = portOffset(c.projectRoot)
+	if c.portExplicit.api {
+		c.apiPortSource = portSourceExplicit
+	} else {
+		c.apiPortSource = c.applyDefaultPortOffset(&c.apiPort)
+	}
+	if c.portExplicit.web {
+		c.webPortSource = portSourceExplicit
+	} else {
+		c.webPortSource = c.applyDefaultPortOffset(&c.webPort)
+	}
+	if c.portExplicit.proxy {
+		c.proxyPortSource = portSourceExplicit
+	} else {
+		c.proxyPortSource = c.applyDefaultPortOffset(&c.proxyPort)
+	}
+	if c.portExplicit.metrics {
+		c.metricsPortSource = portSourceExplicit
+	} else {
+		c.metricsPortSource = c.applyDefaultPortOffset(&c.coderMetricsPort)
+	}
+}
+
+func (c *devConfig) applyDefaultPortOffset(port *int64) portSource {
+	if c.portOffset == 0 {
+		return portSourceDefault
+	}
+	*port += int64(c.portOffset)
+	return portSourceOffset
+}
+
 func (c *devConfig) validate() error {
 	if c.agpl && c.useProxy {
 		return xerrors.New("cannot use both --agpl and --use-proxy")
@@ -293,10 +401,6 @@ func (c *devConfig) validate() error {
 // resolveEnv sets defaults, unsets leaked credentials, resolves
 // filesystem paths, and computes the child process environment.
 func (c *devConfig) resolveEnv() error {
-	if strings.Contains(c.accessURL, "%d") {
-		c.accessURL = fmt.Sprintf(c.accessURL, c.apiPort)
-	}
-
 	// Prevent inherited credentials from leaking into child
 	// processes or being picked up by config reads.
 	_ = os.Unsetenv("CODER_SESSION_TOKEN")
@@ -311,6 +415,11 @@ func (c *devConfig) resolveEnv() error {
 		fmt.Sprintf("coder_%s_%s", runtime.GOOS, runtime.GOARCH))
 	c.configDir = filepath.Join(c.projectRoot, ".coderv2")

+	c.applyPortOffset()
+	if strings.Contains(c.accessURL, "%d") {
+		c.accessURL = fmt.Sprintf(c.accessURL, c.apiPort)
+	}
+
 	// Compute once, reused by cmd().
 	c.childEnv = filterEnv(os.Environ(), "CODER_SESSION_TOKEN", "CODER_URL")

@@ -1120,6 +1229,28 @@ func prometheusBannerEntry(cfg *devConfig, prometheusServerStarted bool) (label
 	}
 }

+func portBannerLine(label string, port int64, source portSource, offset int) string {
+	portValue := strconv.FormatInt(port, 10)
+	if port == 0 {
+		portValue = "disabled"
+	}
+	if source == "" {
+		return fmt.Sprintf("%s: %s", label, portValue)
+	}
+	return fmt.Sprintf("%s: %s (%s)", label, portValue, portSourceLabel(source, offset))
+}
+
+func portSourceLabel(source portSource, offset int) string {
+	switch source {
+	case portSourceExplicit:
+		return fmt.Sprintf("explicit, offset +%d skipped", offset)
+	case portSourceOffset:
+		return fmt.Sprintf("offset +%d", offset)
+	default:
+		return fmt.Sprintf("default, offset +%d", offset)
+	}
+}
+
 func printBanner(ctx context.Context, logger slog.Logger, cfg *devConfig, prometheusServerStarted bool) {
 	ifaces := []string{"localhost"}
 	if addrs, err := net.InterfaceAddrs(); err == nil {
@@ -1153,6 +1284,12 @@ func printBanner(ctx context.Context, logger slog.Logger, cfg *devConfig, promet
 		"",
 		indent("Coder is now running in development mode."),
 		"",
+		"Effective ports:",
+		indent(portBannerLine("API", cfg.apiPort, cfg.apiPortSource, cfg.portOffset)),
+		indent(portBannerLine("Web UI", cfg.webPort, cfg.webPortSource, cfg.portOffset)),
+		indent(portBannerLine("Proxy", cfg.proxyPort, cfg.proxyPortSource, cfg.portOffset)),
+		indent(portBannerLine("Coder metrics", cfg.coderMetricsPort, cfg.metricsPortSource, cfg.portOffset)),
+		"",
 		"API:",
 	)

@@ -146,6 +146,127 @@ func TestShellBool(t *testing.T) {
 	assert.Equal(t, "0", shellBool(false))
 }

+func TestPortOffset(t *testing.T) {
+	t.Parallel()
+
+	root := "/tmp/coder/worktree-a"
+	offset := portOffset(root)
+	assert.Equal(t, offset, portOffset(root))
+	assert.GreaterOrEqual(t, offset, 0)
+	assert.Less(t, offset, 1000)
+	assert.Equal(t, 0, offset%10)
+
+	var foundDifferent bool
+	for _, otherRoot := range []string{
+		"/tmp/coder/worktree-b",
+		"/tmp/coder/worktree-c",
+		"/tmp/coder/worktree-d",
+	} {
+		if portOffset(otherRoot) != offset {
+			foundDifferent = true
+			break
+		}
+	}
+	assert.True(t, foundDifferent, "expected typical worktree paths to use different offsets")
+}
+
+func TestApplyPortOffsetSkipsExplicitPorts(t *testing.T) {
+	t.Parallel()
+
+	projectRoot := "/tmp/coder/worktree-offset"
+	for i := range 100 {
+		candidate := fmt.Sprintf("/tmp/coder/worktree-offset-%d", i)
+		if portOffset(candidate) != 0 {
+			projectRoot = candidate
+			break
+		}
+	}
+	offset := portOffset(projectRoot)
+	require.NotZero(t, offset)
+
+	cfg := &devConfig{
+		apiPort:           3000,
+		webPort:           8080,
+		proxyPort:         3010,
+		coderMetricsPort:  2114,
+		portOffsetEnabled: true,
+		projectRoot:       projectRoot,
+		portExplicit: portExplicit{
+			web:     true,
+			metrics: true,
+		},
+	}
+	cfg.applyPortOffset()
+
+	assert.Equal(t, int64(3000+offset), cfg.apiPort)
+	assert.Equal(t, int64(8080), cfg.webPort)
+	assert.Equal(t, int64(3010+offset), cfg.proxyPort)
+	assert.Equal(t, int64(2114), cfg.coderMetricsPort)
+	assert.Equal(t, portSourceOffset, cfg.apiPortSource)
+	assert.Equal(t, portSourceExplicit, cfg.webPortSource)
+	assert.Equal(t, portSourceOffset, cfg.proxyPortSource)
+	assert.Equal(t, portSourceExplicit, cfg.metricsPortSource)
+}
+
+func TestApplyPortOffsetDisabledUsesDefaultPorts(t *testing.T) {
+	t.Parallel()
+
+	projectRoot := "/tmp/coder/worktree-offset"
+	for i := range 100 {
+		candidate := fmt.Sprintf("/tmp/coder/worktree-offset-disabled-%d", i)
+		if portOffset(candidate) != 0 {
+			projectRoot = candidate
+			break
+		}
+	}
+	require.NotZero(t, portOffset(projectRoot))
+
+	cfg := &devConfig{
+		apiPort:          3000,
+		webPort:          8080,
+		proxyPort:        3010,
+		coderMetricsPort: 2114,
+		projectRoot:      projectRoot,
+	}
+	cfg.applyPortOffset()
+
+	assert.Equal(t, int64(3000), cfg.apiPort)
+	assert.Equal(t, int64(8080), cfg.webPort)
+	assert.Equal(t, int64(3010), cfg.proxyPort)
+	assert.Equal(t, int64(2114), cfg.coderMetricsPort)
+	assert.Zero(t, cfg.portOffset)
+	assert.Empty(t, cfg.apiPortSource)
+	assert.Empty(t, cfg.webPortSource)
+	assert.Empty(t, cfg.proxyPortSource)
+	assert.Empty(t, cfg.metricsPortSource)
+	assert.Equal(t, "API: 3000", portBannerLine("API", cfg.apiPort, cfg.apiPortSource, cfg.portOffset))
+}
+
+func TestPortOffsetDefaultPortsDoNotOverlap(t *testing.T) {
+	t.Parallel()
+
+	ports := []struct {
+		name string
+		base int
+	}{
+		{name: "API", base: 3000},
+		{name: "Web UI", base: 8080},
+		{name: "Proxy", base: 3010},
+		{name: "Coder metrics", base: 2114},
+	}
+	seen := make(map[int]string)
+	for bucket := range portOffsetBuckets {
+		offset := bucket * portOffsetStep
+		for _, port := range ports {
+			effective := port.base + offset
+			if other, ok := seen[effective]; ok {
+				t.Fatalf("%s collides with %s on port %d", port.name, other, effective)
+			}
+			seen[effective] = fmt.Sprintf("%s with offset %d", port.name, offset)
+		}
+	}
+}
+
 func TestDevelopInCoder(t *testing.T) {
 	t.Run("DEVELOP_IN_CODER", func(t *testing.T) {
 		t.Setenv("DEVELOP_IN_CODER", "1")
@@ -431,15 +552,17 @@ func TestDevConfigResolveEnv(t *testing.T) {
 	t.Setenv("CODER_SESSION_TOKEN", "leaked")
 	t.Setenv("CODER_URL", "https://leaked.example.com")

+	wd, _ := os.Getwd()
 	cfg := &devConfig{apiPort: 3000, accessURL: defaultAccessURL}
 	require.NoError(t, cfg.resolveEnv())

-	wd, _ := os.Getwd()
 	assert.Equal(t, wd, cfg.projectRoot)
 	assert.Equal(t, filepath.Join(wd, "build",
 		fmt.Sprintf("coder_%s_%s", runtime.GOOS, runtime.GOARCH)), cfg.binaryPath)
 	assert.Equal(t, filepath.Join(wd, ".coderv2"), cfg.configDir)
 	assert.Equal(t, "http://127.0.0.1:3000", cfg.accessURL)
+	assert.Equal(t, int64(3000), cfg.apiPort)
+	assert.Zero(t, cfg.portOffset)

 	// Should have unset leaked env vars.
 	assert.Empty(t, os.Getenv("CODER_SESSION_TOKEN"))
@@ -454,11 +577,92 @@ func TestDevConfigResolveEnv(t *testing.T) {
 	}
 }

+func TestDevConfigResolveEnvUsesDefaultPortsWithoutPortOffset(t *testing.T) {
+	t.Setenv("CODER_SESSION_TOKEN", "")
+	t.Setenv("CODER_URL", "")
+
+	baseRoot := t.TempDir()
+	projectRoot := filepath.Join(baseRoot, "worktree")
+	for i := range 100 {
+		candidate := filepath.Join(baseRoot, fmt.Sprintf("worktree-default-%d", i))
+		if portOffset(candidate) != 0 {
+			projectRoot = candidate
+			break
+		}
+	}
+	require.NotZero(t, portOffset(projectRoot))
+	require.NoError(t, os.MkdirAll(projectRoot, 0o755))
+	t.Chdir(projectRoot)
+
+	cfg := &devConfig{
+		apiPort:          3000,
+		webPort:          8080,
+		proxyPort:        3010,
+		coderMetricsPort: 2114,
+		accessURL:        defaultAccessURL,
+	}
+	require.NoError(t, cfg.resolveEnv())
+
+	assert.Equal(t, projectRoot, cfg.projectRoot)
+	assert.Equal(t, int64(3000), cfg.apiPort)
+	assert.Equal(t, int64(8080), cfg.webPort)
+	assert.Equal(t, int64(3010), cfg.proxyPort)
+	assert.Equal(t, int64(2114), cfg.coderMetricsPort)
+	assert.Zero(t, cfg.portOffset)
+	assert.Empty(t, cfg.apiPortSource)
+	assert.Empty(t, cfg.webPortSource)
+	assert.Empty(t, cfg.proxyPortSource)
+	assert.Empty(t, cfg.metricsPortSource)
+	assert.Equal(t, "http://127.0.0.1:3000", cfg.accessURL)
+}
+
+func TestDevConfigResolveEnvAppliesPortOffsetWhenEnabled(t *testing.T) {
+	t.Setenv("CODER_SESSION_TOKEN", "")
+	t.Setenv("CODER_URL", "")
+
+	baseRoot := t.TempDir()
+	projectRoot := filepath.Join(baseRoot, "worktree")
+	for i := range 100 {
+		candidate := filepath.Join(baseRoot, fmt.Sprintf("worktree-%d", i))
+		if portOffset(candidate) != 0 {
+			projectRoot = candidate
+			break
+		}
+	}
+	require.NotZero(t, portOffset(projectRoot))
+	require.NoError(t, os.MkdirAll(projectRoot, 0o755))
+	t.Chdir(projectRoot)
+
+	cfg := &devConfig{
+		apiPort:           3000,
+		webPort:           8080,
+		proxyPort:         3010,
+		coderMetricsPort:  2114,
+		portOffsetEnabled: true,
+		accessURL:         defaultAccessURL,
+	}
+	require.NoError(t, cfg.resolveEnv())
+
+	offset := portOffset(projectRoot)
+	assert.Equal(t, projectRoot, cfg.projectRoot)
+	assert.Equal(t, int64(3000+offset), cfg.apiPort)
+	assert.Equal(t, int64(8080+offset), cfg.webPort)
+	assert.Equal(t, int64(3010+offset), cfg.proxyPort)
+	assert.Equal(t, int64(2114+offset), cfg.coderMetricsPort)
+	assert.Equal(t, offset, cfg.portOffset)
+	assert.Equal(t, portSourceOffset, cfg.apiPortSource)
+	assert.Equal(t, fmt.Sprintf("http://127.0.0.1:%d", 3000+offset), cfg.accessURL)
+}
+
 func TestDevConfigResolveEnvExplicitAccessURL(t *testing.T) {
 	t.Setenv("CODER_SESSION_TOKEN", "")
 	t.Setenv("CODER_URL", "")

-	cfg := &devConfig{apiPort: 5000, accessURL: "http://myhost:5000"}
+	cfg := &devConfig{
+		apiPort:      5000,
+		accessURL:    "http://myhost:5000",
+		portExplicit: portExplicit{api: true},
+	}
 	require.NoError(t, cfg.resolveEnv())
 	assert.Equal(t, "http://myhost:5000", cfg.accessURL)
 }
@@ -0,0 +1,100 @@
+#!/usr/bin/env bash
+
+# Summarize failed Go tests from go test JSON output.
+
+set -euo pipefail
+# shellcheck source=scripts/lib.sh
+# shellcheck disable=SC1091
+source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
+cdroot
+
+if [[ $# -ne 1 ]]; then
+	error "Usage: go-test-failure-summary.sh <go-test.json>"
+fi
+
+results_file=$1
+if [[ ! -s "$results_file" ]]; then
+	exit 0
+fi
+
+if ! command -v jq >/dev/null; then
+	error "jq is required to summarize Go test failures."
+fi
+
+jq -sr '
+	def clean_block:
+		tostring
+		| gsub("\u001b\\[[0-9;?]*[ -/]*[@-~]"; "")
+		| gsub("```"; "``");
+	def clean_inline:
+		tostring | gsub("`"; "") | gsub("[\r\n]"; " ");
+	def truncate($max):
+		if length > $max then .[0:$max] + "..." else . end;
+	def terminal_action:
+		.Action == "pass" or .Action == "fail" or .Action == "skip";
+	def test_key:
+		(.Package // "") + "\u0000" + (.Test // "");
+	def output_for($events; $package; $test):
+		[
+			$events[]
+			| select(.Action == "output")
+			| select((.Package // "") == $package)
+			| select((.Test // "") == $test)
+			| .Output // ""
+		]
+		| join("")
+		| clean_block
+		| if . == "" then "No output recorded." else . end
+		| truncate(600);
+
+	map(select(type == "object")) as $events
+	| [
+		$events
+		| to_entries[]
+		| .value + {idx: .key}
+		| select((.Test // "") != "")
+		| select(terminal_action)
+	] as $terminal_tests
+	| [
+		$terminal_tests
+		| group_by(test_key)
+		| .[]
+		| max_by(.idx)
+		| select(.Action == "fail")
+		| {
+			package: ((.Package // "unknown") | clean_inline),
+			test: ((.Test // "unknown") | clean_inline),
+			elapsed: (.Elapsed // 0),
+			output: output_for($events; (.Package // ""); (.Test // ""))
+		}
+	] as $failures
+	| if ($failures | length) == 0 then
+		empty
+	else
+		($failures | length) as $failed
+		| ($failures | map(.package) | unique | length) as $packages
+		| ([
+			$events[]
+			| select((.Test // "") == "")
+			| select(.Action == "pass" or .Action == "fail")
+			| .Elapsed // 0
+		] | add // 0) as $duration
+		| ([
+			$events[]
+			| select((.Test // "") == "")
+			| select(.Action == "fail")
+			| .Package // empty
+		] | unique | length) as $package_failures
+		| [
+			"## Go test failures (\($failed) in \($packages))",
+			"- Duration: \($duration)s",
+			"- Package failures: \($package_failures)",
+			"",
+			($failures[]
+				| "### \(.package) :: \(.test)\n"
+				+ "- Elapsed: \(.elapsed)s\n\n"
+				+ "```\n\(.output)\n```\n")
+		]
+		| join("\n")
+	end
+' "$results_file"
@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+
+# Summarize failed Playwright tests from the JSON reporter output.
+
+set -euo pipefail
+# shellcheck source=scripts/lib.sh
+# shellcheck disable=SC1091
+source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
+cdroot
+
+if [[ $# -ne 1 ]]; then
+	error "Usage: playwright-failure-summary.sh <results.json>"
+fi
+
+results_file=$1
+if [[ ! -f "$results_file" ]]; then
+	exit 0
+fi
+
+if ! command -v jq >/dev/null; then
+	error "jq is required to summarize Playwright failures."
+fi
+
+artifact="playwright-artifacts-${MATRIX_VARIANT:-unknown}-${GITHUB_SHA_SHORT:-unknown}"
+
+jq -r --arg artifact "$artifact" --arg root "$PROJECT_ROOT" '
+	def clean_block:
+		tostring
+		| gsub("\u001b\\[[0-9;]*[A-Za-z]"; "")
+		| gsub("```"; "``");
+	def clean_inline:
+		tostring | gsub("`"; "");
+	def truncate($max):
+		if length > $max then .[0:$max] + "..." else . end;
+	def failure_status:
+		. == "failed" or . == "timedOut" or . == "interrupted";
+	def relpath($root):
+		if startswith($root + "/") then .[($root | length) + 1:]
+		elif startswith("site/") then .
+		elif startswith("e2e/") then "site/" + .
+		else "site/e2e/" + .
+		end;
+	def all_specs($titles):
+		([$titles[], (.title // empty)] | map(select(. != ""))) as $next_titles
+		| (
+			.specs[]?
+			| . + {
+				titlePath: ($next_titles + ([.title // ""] | map(select(. != ""))))
+			}
+		),
+		(.suites[]? | all_specs($next_titles));
+	def failure_entries:
+		[
+			.suites[]?
+			| all_specs([]) as $spec
+			| $spec.tests[]? as $test
+			| select(($test.status // "") != "flaky")
+			| select(
+				(($test.status // "") == "unexpected")
+				or any($test.results[]?; .status | failure_status)
+			)
+			| ([ $test.results[]? | select(.status | failure_status) ][0]
+				// ($test.results[0] // {})) as $result
+			| ((($result.error.message // "") | clean_block) as $message
+				| (($result.error.stack // "") | clean_block) as $stack
+				| {
+					file: (($spec.file // "") | relpath($root)),
+					line: ($spec.line // 0),
+					title: (($spec.titlePath // [$spec.title // ""]) | join(" > ") | clean_inline),
+					project: (($test.projectName // "unknown") | clean_inline),
+					message: (if $message != "" then $message else $stack end | if . != "" then . else "No error message recorded." end | truncate(600)),
+					attachments: ([ $result.attachments[]? | .name // empty | clean_inline ] | unique)
+				})
+		];
+	failure_entries as $entries
+	| if ($entries | length) == 0 then
+		empty
+	else
+		(.stats // {}) as $stats
+		| ($stats.unexpected // 0) as $stats_failed
+		| ([($stats_failed | tonumber), ($entries | length)] | max) as $failed
+		| (($stats.expected // 0) + ($stats.unexpected // 0) + ($stats.flaky // 0) + ($stats.skipped // 0)) as $computed_total
+		| ($stats.total // $computed_total) as $total
+		| [
+			"## Playwright failures (\($failed) of \($total))",
+			"- Duration: \($stats.duration // 0)ms",
+			"- Skipped: \($stats.skipped // 0), Flaky: \($stats.flaky // 0)",
+			"- Artifact: `\($artifact)` (download from the run summary)",
+			"",
+			($entries[]
+				| "### \(.file):\(.line)\n"
+				+ "- Test: `\(.title)`\n"
+				+ "- Project: `\(.project)`\n"
+				+ "- Attachments:\n"
+				+ (if (.attachments | length) == 0 then
+					"  - None recorded in artifact `\($artifact)`"
+				else
+					(.attachments | map("  - `\(.)` in artifact `\($artifact)`") | join("\n"))
+				end)
+				+ "\n\n```\n\(.message)\n```\n")
+		]
+		| join("\n")
+	end
+' "$results_file" | sed -E $'s/\x1b\[[0-9;]*m//g'