Jake Howell
b341ce63fb
> 🤖 This PR was written by Coder Agents on behalf of Jake Howell. Stack: 1. #26575 `fix(site/e2e): close mock external-auth servers in teardown` ← this PR 2. #26793 `fix(site/e2e): accept 404 from external auth reset hook` 3. #26795 `fix(site/src): refresh provider state after device-flow exchange` 4. #26798 `fix(site/e2e): reset both providers in external auth hook` 5. #26648 `chore(site/e2e): re-enable externalAuth suite` The externalAuth e2e suite has been skipped since #17235 because `createServer` in `site/e2e/helpers.ts` started an express server but never gave callers a way to close it. On retries or repeated runs, the listener from the previous invocation was still bound to the hardcoded port and the next `beforeAll` failed with `EADDRINUSE`, eventually timing out in `waitForPort`. `createServer` now returns a `{ app, close }` pair. The web flow closes in `afterAll`; the device flow uses `try/finally`. `closeAllConnections()` is called before `close()` so teardown stays bounded if keep-alive connections linger. The suite remains `test.describe.skip` here; #26648 flips the skip off once the rest of the stack is in. Refs https://linear.app/codercom/issue/DEVEX-413 Refs https://github.com/coder/internal/issues/356 <details> <summary>Decision log</summary> Discussed the full options list with @jakehwll before drafting. Picked option A (minimal teardown) because: - Two prior PRs (#15537, #16528) attacked symptoms (port probing, longer timeout) without addressing the leaked listener. - Kayla's diagnosis on coder/internal#356 pointed at exactly this case: nothing else in CI is grabbing the port, the listener from the previous run is still bound. - A is mechanical and orthogonal: it adds a real teardown without changing port allocation, fixture wiring, or what gets mocked. If the flake persists after A, we know to escalate to a worker-scoped fixture or dynamically allocated ports. Returning `close` rather than the raw `http.Server` encapsulates the `closeAllConnections` + `close` choreography so callers don't repeat it. `closeAllConnections` is optional-chained because it landed in Node 18.2; coder/coder runs newer, but the chain costs nothing. The device test uses `try/finally` rather than a shared `afterEach` to keep per-test state local. The web flow's `afterAll` mirrors its `beforeAll`. </details>
e2e
The structure of the end-to-end tests is optimized for speed and reliability. Not all tests require setting up a new PostgreSQL instance or using the Terraform provisioner. Deciding when to trade time for robustness rests with the developers; the framework's role is to facilitate this process.
Take a look at prior art in tests/ for inspiration. To run a test:
cd site
# Build the frontend assets. If you are actively changing
# the site to debug an issue, add `--watch`.
pnpm build
# Alternatively, build with debug info and source maps:
NODE_ENV=development pnpm vite build --mode=development
# Install the browsers to `~/.cache/ms-playwright`.
pnpm playwright:install
# Run E2E tests. You can see the configuration of the server
# in `playwright.config.ts`. This uses `go run -tags embed ...`.
pnpm playwright:test
# Run a specific test (`-g` stands for grep. It accepts regex).
pnpm playwright:test -g '<your test here>'
Using nix
If this breaks, it is likely because the flake chromium version and playwright are no longer compatible. To fix this, update the flake to get the latest chromium version, and adjust the playwright version in the package.json.
You can see the playwright version here: https://search.nixos.org/packages?channel=unstable&show=playwright-driver&from=0&size=50&sort=relevance&type=packages&query=playwright-driver
# Optionally add '--command zsh' to choose your shell.
nix develop
cd site
pnpm install
pnpm build
pnpm playwright:test
To run the playwright debugger from VSCode, just launch VSCode from the nix environment and have the extension installed.
# Optionally add '--command zsh' to choose your shell.
nix develop
code .
Enterprise tests
Enterprise tests require a license key to run.
export CODER_E2E_LICENSE=<license key>
Debugging tests
To debug a test, it is more helpful to run it in ui mode.
pnpm playwright:test-ui