Faur Ioan-Aurel
0caaac5d27
The test-e2e job could flake while Playwright waited for its webServer to become ready. CI already built site/e2e/bin/coder, but that binary was built before pnpm build and without the embed tag, so Playwright ignored it and started coderd with go run -tags embed instead. That put Go compilation, including frontend asset embedding, inside Playwright's webServer timeout. Local runs had a similar ordering trap. pnpm playwright:test invoked Playwright directly, and Playwright starts webServer before globalSetup, so the existing preflight build could not guarantee a binary existed before webServer startup. Build the E2E coder binary with the embed tag and make it depend on site/out/index.html. In CI, build the frontend first, then build the E2E binary, then run Playwright. For local runs, have pnpm playwright:test and pnpm playwright:test-ui run make site/e2e/bin/coder before invoking Playwright. Playwright now starts the prebuilt site/e2e/bin/coder binary and waits on the public /healthz endpoint instead of an authenticated API endpoint. This keeps compilation outside the webServer readiness timeout and makes the readiness probe quieter and more direct. I moved to /healtz because the existing endpoint was returning 401, which still means “server is up,” but it is a strange readiness signal. /healthz is explicitly public and returns 200 OK once the HTTP server is accepting requests. Caveat: running pnpm exec playwright test --config=e2e/playwright.config.ts directly bypasses the package script and therefore bypasses the prebuild. Use pnpm playwright:test or make test-e2e for the supported local flow. Because the site package scripts now call make, add make to knip's ignored system binaries. Tests: - pnpm lint:types - pnpm exec playwright test --config=e2e/playwright.config.ts --list - PATH="/opt/homebrew/opt/make/libexec/gnubin:/opt/homebrew/opt/gnu-getopt/bin:$PATH" make gen/mark-fresh - PATH="/opt/homebrew/opt/make/libexec/gnubin:/opt/homebrew/opt/gnu-getopt/bin:$PATH" pnpm playwright:test --list <!-- If you have used AI to produce some or all of this PR, please ensure you have read our [AI Contribution guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING) before submitting. -->
1.9 KiB
e2e
The structure of the end-to-end tests is optimized for speed and reliability. Not all tests require setting up a new PostgreSQL instance or using the Terraform provisioner. Deciding when to trade time for robustness rests with the developers; the framework's role is to facilitate this process.
Take a look at prior art in tests/ for inspiration. To run a test:
cd site
# Build the frontend assets. If you are actively changing
# the site to debug an issue, add `--watch`.
pnpm build
# Alternatively, build with debug info and source maps:
NODE_ENV=development pnpm vite build --mode=development
# Install the browsers to `~/.cache/ms-playwright`.
pnpm playwright:install
# Run E2E tests. You can see the configuration of the server
# in `playwright.config.ts`. This builds and runs `site/e2e/bin/coder`.
pnpm playwright:test
# Run a specific test (`-g` stands for grep. It accepts regex).
pnpm playwright:test -g '<your test here>'
Using nix
If this breaks, it is likely because the flake chromium version and playwright are no longer compatible. To fix this, update the flake to get the latest chromium version, and adjust the playwright version in the package.json.
You can see the playwright version here: https://search.nixos.org/packages?channel=unstable&show=playwright-driver&from=0&size=50&sort=relevance&type=packages&query=playwright-driver
# Optionally add '--command zsh' to choose your shell.
nix develop
cd site
pnpm install
pnpm build
pnpm playwright:test
To run the playwright debugger from VSCode, just launch VSCode from the nix environment and have the extension installed.
# Optionally add '--command zsh' to choose your shell.
nix develop
code .
Enterprise tests
Enterprise tests require a license key to run.
export CODER_E2E_LICENSE=<license key>
Debugging tests
To debug a test, it is more helpful to run it in ui mode.
pnpm playwright:test-ui