feat: add harness engineering layer for agent workflows (#24791)

This PR adds an opinionated harness-engineering layer for agent-driven workflows: a small set of agent-readable docs, mechanical structure checks, structured CI failure summaries, an architecture-lint umbrella, and per-worktree dev-server isolation. The goal is to make local dev, tests, and CI mechanically inspectable by agents without changing app runtime behavior. ## What landed **Agent docs and navigation** - `.claude/docs/OBSERVABILITY.md`, `.claude/docs/DEV_ISOLATION.md`, `.claude/docs/AGENT_FAILURES.md`: task-oriented guides for logs, tracing, Prometheus, dev-server isolation, and a seeded failure catalog. - `AGENTS.md`: added an `Agent navigation` block, then trimmed the file from 375 to 229 lines by migrating duplicated detail into `WORKFLOWS.md`, `GO.md`, `TESTING.md`, and `DATABASE.md`. The user-managed custom-instructions block is preserved. - `.agents/docs`: symlink mirror of `.claude/docs` for agent runtimes that look under `.agents`. **Mechanical checks** - `scripts/check_agents_structure.sh`: validates `@...` references in tracked `AGENTS.md` files and warns when root grows past 600 lines. Wired as `make lint/agents` and into `make lint`. - `scripts/audit-agent-readiness.sh`: report-first audit of harness readiness. Currently `10 ok, 0 warn, 0 fail`. - `scripts/check_architecture.sh` / `make lint/architecture`: umbrella architecture-lint target. Consolidates the existing `check_enterprise_imports.sh` and `check_codersdk_imports.sh` so they run exactly once via the umbrella. Slot is open for new high-confidence rules. **Structured CI failure summaries** - `scripts/playwright-failure-summary.sh`: parses `site/test-results/results.json` and writes Markdown to `$GITHUB_STEP_SUMMARY` on failure. Wired into the `test-e2e` matrix job. - `scripts/go-test-failure-summary.sh`: parses `go test -json` line-delimited output the same way. Wired into `test-go-pg`, `test-go-pg-17`, and `test-go-race-pg` by injecting `gotestsum --jsonfile` in the workflow without touching `Makefile`. JSON also uploaded as a CI artifact on failure. - `site/e2e/playwright.config.ts`: enables `screenshot: only-on-failure`, `trace: retain-on-failure`, JSON reporter, and HTML reporter alongside existing reporters. - `.github/workflows/ci.yaml`: failure artifact uploads for Playwright now use `if: failure()` and predictable names (`playwright-artifacts-<variant>-<sha>`). **Per-worktree dev-server isolation** (`scripts/develop/main.go`) - Deterministic FNV-64a hash of the worktree path produces a port offset in `[0, 1000)` (50 buckets, step 20 to avoid API/proxy overlap across adjacent buckets). - Offset is applied only to defaults; both env vars (`CODER_DEV_PORT`, `CODER_DEV_WEB_PORT`, `CODER_DEV_PROXY_PORT`, `CODER_DEV_PROMETHEUS_PORT`) and CLI flags retain priority. - Hardcoded ports `9090` (embedded Prometheus UI) and `12345` (Delve) are unchanged by design. - Startup banner shows each port's source: `default`, `offset`, or `explicit`. - Unit tests in `scripts/develop/main_test.go` cover determinism, bounds, no-overlap across the four ports, and explicit-skip behavior. - State (`.coderv2/`) was already worktree-isolated via `os.Getwd()`, so no state-dir changes were needed. ## Validation `make lint/agents`, `make lint/architecture`, `make lint/emdash`, `bash scripts/audit-agent-readiness.sh` (10 ok, 0 warn, 0 fail), `shellcheck` on all 5 new scripts, `go test ./scripts/develop/...`, and `js-yaml` parse of `ci.yaml` all pass. Synthetic fixtures verify both failure-summary scripts handle empty/missing input (silent exit 0), ANSI-stripped output, and parent/subtest formatting. ## Known follow-ups (deferred) - Frontend Storybook/Vitest failure summary: lowest-leverage slice of the failure-summary work. Skipping until observed pain. - Architecture lint currently only delegates to existing import checks; new rules (`InTx` outer-store detection, swagger-annotation lint) plug in as needed. - 50 port-offset buckets means two worktree paths can occasionally collide. The DEV_ISOLATION doc tells users to set the relevant env var when this happens. > Mux opened this PR on Mike's behalf.
2026-09-21 20:51:01 +08:00 · 2026-05-11 17:27:29 +02:00
parent 915956460a
commit 85792d08bc
20 changed files with 1561 additions and 237 deletions
@@ -464,6 +464,24 @@ jobs:
          source scripts/normalize_path.sh
          normalize_path_with_symlinks "$RUNNER_TEMP/sym" "$(dirname "$(which terraform)")"

+      - name: Configure Go test JSON capture
+        if: runner.os == 'Linux'
+        shell: bash
+        run: |
+          set -euo pipefail
+          bin_dir="${RUNNER_TEMP}/go-test-json-bin"
+          mkdir -p "$bin_dir"
+
+          real_gotestsum="$(command -v gotestsum)"
+          real_gotestsum_quoted="$(printf '%q' "$real_gotestsum")"
+          printf '%s\n' \
+            '#!/usr/bin/env bash' \
+            'set -euo pipefail' \
+            "exec ${real_gotestsum_quoted} --jsonfile \"\${RUNNER_TEMP}/go-test.json\" \"\$@\"" \
+            > "${bin_dir}/gotestsum"
+          chmod +x "${bin_dir}/gotestsum"
+          echo "$bin_dir" >> "$GITHUB_PATH"
+
      - name: Setup RAM disk for Embedded Postgres (Windows)
        if: runner.os == 'Windows'
        shell: bash
@@ -540,6 +558,18 @@ jobs:
          embedded-pg-path: "R:/temp/embedded-pg"
          embedded-pg-cache: ${{ steps.embedded-pg-cache.outputs.embedded-pg-cache }}

+      - name: Publish Go test failure summary
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        run: bash scripts/go-test-failure-summary.sh "${RUNNER_TEMP}/go-test.json" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload Go test JSON
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: go-test-json-${{ github.job }}-${{ github.sha }}
+          path: ${{ runner.temp }}/go-test.json
+          retention-days: 7
+
      - name: Upload failed test db dumps
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
@@ -610,6 +640,24 @@ jobs:
          source scripts/normalize_path.sh
          normalize_path_with_symlinks "$RUNNER_TEMP/sym" "$(dirname "$(which terraform)")"

+      - name: Configure Go test JSON capture
+        if: runner.os == 'Linux'
+        shell: bash
+        run: |
+          set -euo pipefail
+          bin_dir="${RUNNER_TEMP}/go-test-json-bin"
+          mkdir -p "$bin_dir"
+
+          real_gotestsum="$(command -v gotestsum)"
+          real_gotestsum_quoted="$(printf '%q' "$real_gotestsum")"
+          printf '%s\n' \
+            '#!/usr/bin/env bash' \
+            'set -euo pipefail' \
+            "exec ${real_gotestsum_quoted} --jsonfile \"\${RUNNER_TEMP}/go-test.json\" \"\$@\"" \
+            > "${bin_dir}/gotestsum"
+          chmod +x "${bin_dir}/gotestsum"
+          echo "$bin_dir" >> "$GITHUB_PATH"
+
      - name: Test with PostgreSQL Database
        uses: ./.github/actions/test-go-pg
        with:
@@ -621,6 +669,18 @@ jobs:
          # On main, run tests without cache for the inverse.
          test-count: ${{ github.ref == 'refs/heads/main' && '1' || '' }}

+      - name: Publish Go test failure summary
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        run: bash scripts/go-test-failure-summary.sh "${RUNNER_TEMP}/go-test.json" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload Go test JSON
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: go-test-json-${{ github.job }}-${{ github.sha }}
+          path: ${{ runner.temp }}/go-test.json
+          retention-days: 7
+
      - name: Upload Test Cache
        uses: ./.github/actions/test-cache/upload
        with:
@@ -678,6 +738,24 @@ jobs:
      # c.f. discussion on https://github.com/coder/coder/pull/15106
      # Our Linux runners have 16 cores, but we reduce parallelism since race detection adds a lot of overhead.
      # We aim to have parallelism match CPU count (4*4=16) to avoid making flakes worse.
+      - name: Configure Go test JSON capture
+        if: runner.os == 'Linux'
+        shell: bash
+        run: |
+          set -euo pipefail
+          bin_dir="${RUNNER_TEMP}/go-test-json-bin"
+          mkdir -p "$bin_dir"
+
+          real_gotestsum="$(command -v gotestsum)"
+          real_gotestsum_quoted="$(printf '%q' "$real_gotestsum")"
+          printf '%s\n' \
+            '#!/usr/bin/env bash' \
+            'set -euo pipefail' \
+            "exec ${real_gotestsum_quoted} --jsonfile \"\${RUNNER_TEMP}/go-test.json\" \"\$@\"" \
+            > "${bin_dir}/gotestsum"
+          chmod +x "${bin_dir}/gotestsum"
+          echo "$bin_dir" >> "$GITHUB_PATH"
+
      - name: Run Tests
        uses: ./.github/actions/test-go-pg
        with:
@@ -686,6 +764,18 @@ jobs:
          test-parallelism-tests: "4"
          race-detection: "true"

+      - name: Publish Go test failure summary
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        run: bash scripts/go-test-failure-summary.sh "${RUNNER_TEMP}/go-test.json" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload Go test JSON
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: go-test-json-${{ github.job }}-${{ github.sha }}
+          path: ${{ runner.temp }}/go-test.json
+          retention-days: 7
+
      - name: Upload Test Cache
        uses: ./.github/actions/test-cache/upload
        with:
@@ -820,27 +910,36 @@ jobs:
          CODER_E2E_REQUIRE_PREMIUM_TESTS: "1"
        working-directory: site

-      - name: Upload Playwright Failed Tests
-        if: always() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+      - name: Upload Playwright failure artifacts
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
-          name: failed-test-videos${{ matrix.variant.premium && '-premium' || '' }}
-          path: ./site/test-results/**/*.webm
+          name: playwright-artifacts-${{ matrix.variant.name }}-${{ github.sha }}
+          path: |
+            ./site/test-results/**
+            ./site/playwright-report/**
          retention-days: 7

+      - name: Publish Playwright failure summary
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        env:
+          MATRIX_VARIANT: ${{ matrix.variant.name }}
+          GITHUB_SHA_SHORT: ${{ github.sha }}
+        run: bash scripts/playwright-failure-summary.sh site/test-results/results.json >> "$GITHUB_STEP_SUMMARY"
+
      - name: Upload debug log
-        if: always() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
-          name: coderd-debug-logs${{ matrix.variant.premium && '-premium' || ''  }}
+          name: coderd-debug-logs-${{ matrix.variant.name }}-${{ github.sha }}
          path: ./site/e2e/test-results/debug.log
          retention-days: 7

      - name: Upload pprof dumps
-        if: always() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
+        if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
-          name: debug-pprof-dumps${{ matrix.variant.premium && '-premium' || ''  }}
+          name: debug-pprof-dumps-${{ matrix.variant.name }}-${{ github.sha }}
          path: ./site/test-results/**/debug-pprof-*.txt
          retention-days: 7