Ethan
5112ab7da9
_PR generated by Mux but reviewed by a human_
## Problem
The e2e test `template update with new name redirects on successful
submit` is flaky.
After saving template settings, the app navigates to
`/templates/<name>`, which immediately redirects to
`/templates/<name>/docs` via the router's index route (`<Navigate
to="docs" replace />`). The assertion used `expect.poll()` with
`toHavePathNameEndingWith(`/${name}`)`, which matches only the
**transient intermediate URL** — it only exists while `TemplateLayout`'s
async data fetch is pending. Once the fetch resolves and the `<Outlet
/>` renders, the index route fires the `/docs` redirect and the URL no
longer matches.
## Why it's flaky (not deterministic)
The flakiness depends on whether the template query cache is warm:
- **Cache miss → PASSES**: The mutation's `onSuccess` handler
invalidates the query cache. If `TemplateLayout` needs to re-fetch, it
shows a `<Loader />`, which delays rendering the `<Outlet />` that
contains the `<Navigate to="docs">`. This gives `expect.poll()` time to
see the transient `/new-name` URL → **pass**.
- **Cache hit → FAILS**: If the template data is still in the query
client, `TemplateLayout` renders immediately and the `<Navigate
to="docs" replace />` fires nearly instantly. By the time the first poll
runs, the URL is already `/new-name/docs` → **fail**.
## Fix
Assert the **final stable URL** (`/${name}/docs`) instead of the
transient one.
This is safe because `expect.poll()` is retry-based: it keeps sampling
until a match is found (or timeout). Seeing the transient `/new-name`
URL just causes harmless retries — once the redirect completes and the
URL settles on `/new-name/docs`, the poll matches and the test passes.
| Poll | URL | Ends with `/new-name/docs`? | Action |
|---|---|---|---|
| 1st | `/templates/new-name` | No | Retry |
| 2nd | `/templates/new-name` | No | Retry |
| 3rd | `/templates/new-name/docs` | Yes | **Pass** ✅ |
Closes https://github.com/coder/internal/issues/1403
e2e
The structure of the end-to-end tests is optimized for speed and reliability. Not all tests require setting up a new PostgreSQL instance or using the Terraform provisioner. Deciding when to trade time for robustness rests with the developers; the framework's role is to facilitate this process.
Take a look at prior art in tests/ for inspiration. To run a test:
cd site
# Build the frontend assets. If you are actively changing
# the site to debug an issue, add `--watch`.
pnpm build
# Alternatively, build with debug info and source maps:
NODE_ENV=development pnpm vite build --mode=development
# Install the browsers to `~/.cache/ms-playwright`.
pnpm playwright:install
# Run E2E tests. You can see the configuration of the server
# in `playwright.config.ts`. This uses `go run -tags embed ...`.
pnpm playwright:test
# Run a specific test (`-g` stands for grep. It accepts regex).
pnpm playwright:test -g '<your test here>'
Using nix
If this breaks, it is likely because the flake chromium version and playwright are no longer compatible. To fix this, update the flake to get the latest chromium version, and adjust the playwright version in the package.json.
You can see the playwright version here: https://search.nixos.org/packages?channel=unstable&show=playwright-driver&from=0&size=50&sort=relevance&type=packages&query=playwright-driver
# Optionally add '--command zsh' to choose your shell.
nix develop
cd site
pnpm install
pnpm build
pnpm playwright:test
To run the playwright debugger from VSCode, just launch VSCode from the nix environment and have the extension installed.
# Optionally add '--command zsh' to choose your shell.
nix develop
code .
Enterprise tests
Enterprise tests require a license key to run.
export CODER_E2E_LICENSE=<license key>
Debugging tests
To debug a test, it is more helpful to run it in ui mode.
pnpm playwright:test-ui