mirror of
https://github.com/coder/coder.git
synced 2026-11-01 04:47:17 +08:00
Andrew Aquino
5699f1cdfb
closes DEVEX-651 ## Summary Fixes coder/internal#218 (`flake: e2e-test / test ssh`). Despite the title, the `ssh with client v2.8.0` / `ssh with agent v2.12.1` cases (`site/e2e/tests/outdatedCLI.spec.ts`, `outdatedAgent.spec.ts`) are not failing because of a bug in SSH. They fail during **setup**, in `downloadCoderVersion()`, which runs `install.sh` to fetch an old Coder release from GitHub. Transient GitHub errors (HTTP 403/503, surfacing as nonzero `curl` exit codes such as 22 or 1) make `install.sh` fail and take the whole ssh test down with it. This is an external-download flake, confirmed by the recurring `install.sh failed with code {22,1}` evidence in the issue thread and Ethan's note ("Networking issues again"). ## Changes 1. **Retry-with-backoff** (`site/e2e/helpers.ts`): `downloadCoderVersion()` now retries `install.sh` up to 5 times with exponential backoff and jitter (~1s, 2s, 4s, 8s). A single transient download failure no longer fails the test. `install.sh` already reuses completed binaries and resumes partial downloads (`curl -C -`), so retries are cheap. 2. **Cross-run cache** (`.github/workflows/ci.yaml`): the `test-e2e` job now persists `/tmp/coder-e2e-cache` with `actions/cache`, so most runs skip the GitHub download entirely. The key is derived from the spec files that pin the downloaded versions, so it invalidates when those versions change. Saves are restricted to `main` (`restore` runs everywhere), matching the existing cache-poisoning convention used for the Vale and golangci-lint caches. Before this change, neither retry, mirror, nor cross-run caching protected this path; the only caching was within a single run. ## Testing - `biome check e2e/helpers.ts` passes. - `tsc --noEmit` introduces no new errors. - CI `test-e2e` exercises the changed path. <details> <summary>Investigation notes</summary> - The failure always originates in `downloadCoderVersion` -> `install.sh` -> `fetch()` (`curl -#fL ... https://github.com/coder/coder/releases/download/vX.Y.Z/...`). - `curl` exit 22 = server returned an HTTP error (403 seen in logs); exit 1 = other transient failure. GitHub also returned 503s across the workflow in some occurrences. - `/tmp/coder-e2e-cache` was not persisted by any `actions/cache` step in `ci.yaml`, so every fresh job re-downloaded from GitHub and was exposed to the flake. - Retry addresses transient failures; the cache removes the dependency on GitHub for most runs. Combined, they target the root cause at two layers. </details> --- This PR was generated by Coder Agents on behalf of @aqandrew.
e2e
The structure of the end-to-end tests is optimized for speed and reliability. Not all tests require setting up a new PostgreSQL instance or using the Terraform provisioner. Deciding when to trade time for robustness rests with the developers; the framework's role is to facilitate this process.
Take a look at prior art in tests/ for inspiration. To run a test:
cd site
# Build the frontend assets. If you are actively changing
# the site to debug an issue, add `--watch`.
pnpm build
# Alternatively, build with debug info and source maps:
NODE_ENV=development pnpm vite build --mode=development
# Install the browsers to `~/.cache/ms-playwright`.
pnpm playwright:install
# Run E2E tests. You can see the configuration of the server
# in `playwright.config.ts`. This builds and runs `site/e2e/bin/coder`.
pnpm playwright:test
# Run a specific test (`-g` stands for grep. It accepts regex).
pnpm playwright:test -g '<your test here>'
Using nix
If this breaks, it is likely because the flake chromium version and playwright are no longer compatible. To fix this, update the flake to get the latest chromium version, and adjust the playwright version in the package.json.
You can see the playwright version here: https://search.nixos.org/packages?channel=unstable&show=playwright-driver&from=0&size=50&sort=relevance&type=packages&query=playwright-driver
# Optionally add '--command zsh' to choose your shell.
nix develop
cd site
pnpm install
pnpm build
pnpm playwright:test
To run the playwright debugger from VSCode, just launch VSCode from the nix environment and have the extension installed.
# Optionally add '--command zsh' to choose your shell.
nix develop
code .
Enterprise tests
Enterprise tests require a license key to run.
export CODER_E2E_LICENSE=<license key>
Debugging tests
To debug a test, it is more helpful to run it in ui mode.
pnpm playwright:test-ui