This PR adds a fake SMTP server for scale testing. It collects emails sent during tests, which you can then check using the HTTP API.
#### Changes
- Added mock SMTP server
- Added `coder scaletest smtp` CLI command
- Implemented HTTP API endpoints to retrieve messages by email
- Added auto-purge to prevent memory issues
#### HTTP API Endpoints
- `GET /messages?email=<email>` – Get messages sent to an email address
- `POST /purge` – Clear all messages from memory
The HTTP API parses raw email messages to extract the **date**, **subject**, and **notification ID**.
Notification IDs are sent in emails like this:
```html
<p>
<a href="http://127.0.0.1:3000/settings/notifications?disabled=4e19c0ac-94e1-4532-9515-d1801aa283b2"
style="color: #2563eb; text-decoration: none;">
Stop receiving emails like this
</a>
</p>
```
#### CLI
```bash
coder scaletest smtp --host localhost --port 33199 --api-port 8080 --purge-at-count 1000
```
**Flags:**
- `--host`: Host for the mock SMTP and API server (default: localhost)
- `--port`: Port for the mock SMTP server (random if not specified)
- `--api-port`: Port for the HTTP API server (random if not specified)
- `--purge-at-count`: Max number of messages before auto-purging (default: 100000)
fixes https://github.com/coder/internal/issues/1050
Extends the timeout on the affected test. It uses a postgres database, and so 15s timeout isn't enough to not flake with our test infra these days.
part of https://github.com/coder/internal/issues/912
Adds CLI command `coder exp scaletest dynamic-parameters`
I've left out the configuration of tracing and timeouts for now. I think I want to do some refactoring of the scaletest CLI to make handling those flags take up less boiler plate.
I will add tracing and timeout flags in a follow up PR.
Relates to https://github.com/coder/internal/issues/910
This PR adds a scaletest runner that simulates users receiving notifications through WebSocket connections.
An instance of this notification runner does the following:
1. Creates a user (optionally with specific roles like owner).
2. Connects to /api/v2/notifications/inbox/watch via WebSocket to receive notifications in real-time.
3. Waits for all other concurrently executing runners (per the DialBarrier WaitGroup) to also connect their websockets.
4. For receiving users: Watches the WebSocket for expected notifications and records delivery latency for each notification type.
5. For regular users: Maintains WebSocket connections to simulate concurrent load while receiving users wait for notifications.
6. Waits on the ReceivingWatchBarrier to coordinate between receiving and regular users.
7. Cleans up the created user after the test completes.
Exposes three prometheus metrics:
1. notification_delivery_latency_seconds - HistogramVec. Labels = {username, notification_type}
2. notification_delivery_errors_total - CounterVec. Labels = {username, action}
3. notification_delivery_missed_total - CounterVec. Labels = {username}
The runner measures end-to-end notification latency from when a notification-triggering event occurs (e.g., user creation/deletion) to when the notification is received by a WebSocket client.
Relates to https://github.com/coder/internal/issues/911
This PR adds a scaletest runner that simulates a user with a workspace configured to autostart at a specific time.
An instance of this autostart runner does the following:
1. Creates a user.
2. Creates a workspace.
3. Listens on `/api/v2/workspaces/<ws-id>/watch` to wait for build completions throughout the run.
4. Stops the workspace, waits for the build to finish.
4. Waits for all other concurrently executing runners (per the `SetupBarrier` `WaitGroup`) to also have a stopped workspace.
5. Sets the workspace autostart time to `time.Now().Add(cfg.AutoStartDelay)`
6. Waits for the autostart workspace build to begin, enter the `pending` status, and then `running`.
7. Waits for the autostart workspace build to end.
Exposes four prometheus metrics:
- `autostart_job_creation_latency_seconds` - HistogramVec. Labels = {username, workspace_name}
- `autostart_job_acquired_latency_seconds` - HistogramVec. Labels = {username, workspace_name}
- `autostart_total_latency_seconds` - `HistogramVec`. Labels = `{username, workspace_name}`.
- `autostart_errors_total` - `CounterVec`. Labels = `{username, action}`.
Similar idea as in #19811, this runner doesn't need to conform to `Runnable`, so we have it return the workspace from the `RunReturningWorkspace` function, instead of the more fragile `Run`, followed by a `.WorkspaceID()`.
Relates to https://github.com/coder/internal/issues/889.
This PR adds a scaletest runner that simulates a single Coder Connect client receiving workspace updates.
An instance of a workspace updates runner does the following:
- Creates a user, if a session token is not supplied.
- Attempts to repeatedly dial the Coder Connect endpoint, with a configurable (two minutes by default) timeout.
- Once dialed successfully, waits for any other concurrently executing runners to also dial successfully, or timeout (using the barrier).
- Starts a configurable number of workspace builds.
- Waits for that many workspaces to be seen over the workspace updates stream (with a configurable timeout).
Exposes two prometheus metrics:
- `workspace_updates_latency_seconds` - `HistogramVec`. Labels = `{username, num_owned_workspaces, workspace_name}`
- This is the time between starting a workspace build, and receiving both the corresponding workspace update.
- `workspace_updates_errors_total` - `NewCounterVec`. Labels = `{username, num_owned_workspaces, action}`
- The number of times a specific action of the runner has failed, per user/client.
relates to https://github.com/coder/internal/issues/912
Adds a new scaletest Runner to generate dynamic parameters load.
A later PR will add the CLI command, including creating the template & version.
Relates to https://github.com/coder/internal/issues/889
The existing implementation for exposing read and written bytes was a little awkward - we're going to be adding a bunch of scaletest runners / load generators that *don't* transfer any bytes. This PR has the scaletest reports expose a map of arbitrary string-keyed metrics instead.
FWIW, the latest iteration of the scaletesting infrastructure doesn't parse these reports right now - they're just logged to stdout, so we're good to break the json schema here.
Closes https://github.com/coder/internal/issues/985
Simple refactor of the user creation logic into it's own test runner. This lets us create users independently of workspaces, for use in a bunch of load generators, including the Coder Connect load generator.
This PR creates the new runner, and has the existing `createworkspaces` runner use it.
Relates to https://github.com/coder/internal/issues/985.
Some scaletest runners would autogenerate names if they weren't supplied on the config, while others required a name be supplied, and a name was autogenerated in the CLI command handler. This PR unifies the runners to make names and emails optional on each config, and generate them in the scaletest runner if omitted.
The create user runner in the PR above in the stack will do this too.
This PR improves the ruleguard rule for detecting `t.Fail` calls in goroutines. It picks up additional violations, of which are fixed in this PR.
See self-review for details.
The motivation for fixing this comes from a flake I fixed in https://github.com/coder/coder/pull/19599, where tests would fail from a `require` in an `Eventually`.
Refactors Agent instance identity to be a SessionTokenProvider.
Refactors the CLI to create Agent clients via a centralized function, rather than add-hoc via individual command handlers and their flags.
This allows commands besides `coder agent`, but which still use the agent identity, to support instance identity authentication.
Fixes#19111 by unifying all API requests to go thru the SessionTokenProvider for auth credentials.
Refactors `codersdk.Client`'s use of session tokens to use a `SessionTokenProvider`, which abstracts the obtaining and storing of the session token.
The main motiviation is to unify Agent authentication an an upstack PR, which can use cloud instance identity via token exchange, rather than a fixed session token.
However, the abstraction could also allow functionality like obtaining the session token from other external sources like the OS credential manager, or an external secret/key management system like Vault.
Relates to https://github.com/coder/internal/issues/888
As part of our renewed connection scaletesting efforts, we want to
scaletest coder in a scenario where direct connections aren't available
(relatively common for our customers), and all concurrent connections
are relayed via DERP.
This PR adds a flag, `--disable-direct` that can be included on the
existing`coder exp scaletest workspace-traffic -ssh` to disable direct
connections.
We've successfully migrated the latest iteration of our scaletest
infrastructure (`scaletest/terraform/action`) to
https://github.com/coder/scaletest (private repo). This PR removes the
older iterations, and the scriptsfor spinning up & running the load
generators against that infrastructure (`scaletest.sh`). The tooling for
generating load against a Coder deployment remains untouched, as does
the public documentation for that tooling (i.e. `coder exp scaletest`).
If we ever need that old scaletest Terraform code, it's always in the
git history!
This PR refactors the scaletest infrastructure to use a dedicated VPC for each deployment (i.e. alpha, bravo, charlie). It then peers that VPC with the observability VPC, and the Cloud SQL database. It also sets up subnetting for and within each deployment.
With this deployed, I was able to get the scaletest running with metrics flowing into `scaletest.cdr.dev`.
Co-authored-by: Dean Sheather <dean@deansheather.com>
Closes https://github.com/coder/internal/issues/850
This PR has the scaletest infrastructure retrieve and use TLS certificates from the persistent observability cluster.
To support creating multiple instances of the infrastructure simultaneously, `var.name` can be set to `alpha`, `bravo` or `charlie`, which retrieves the corresponding certificates.
Also:
- Adds support for wildcard apps.
- Retrieves the Cloudflare token from GCP secrets.
Removes the requirement to obtain a Cloudflare DNS token from our scaletest/terraform/action builds. Instead, by default, we pull the token from Google Secrets Manager and use the `scaletest.dev` DNS domain.
Removes cloudflare_email as this was unneeded.
Removes the cloudflare_zone_id and instead pulls it from a data source via the Cloudflare API.
closes https://github.com/coder/internal/issues/839
Fixes https://github.com/coder/internal/issues/907
We convert `workspacesdk.AgentConn` to an interface and generate a mock
for it. This allows writing `coderd` tests that rely on the agent's HTTP
api to not have to set up an entire tailnet networking stack.
Relaxes the `terraform` version constraint to be at least 1.9, since
1.12 is installed in our Dogfood image
Adds a `small` scenario to keep costs down while we continue to develop
capabilities.
Closes#17791
This PR adds ability to cancel workspace builds that are in "pending"
status.
Breaking changes:
- CancelWorkspaceBuild method in codersdk now accepts an optional
request parameter
API:
- Added `expect_status` query parameter to the cancel workspace build
endpoint
- This parameter ensures the job hasn't changed state before canceling
- API returns `412 Precondition Failed` if the job is not in the
expected status
- Valid values: `running` or `pending`
- Wrapped the entire cancel method in a database transaction
UI:
- Added confirmation dialog to the `Cancel` button, since it's a
destructive operation
- Enabled cancel action for pending workspaces (`expect_status=pending`
is sent if workspace is in pending status)
---------
Co-authored-by: Dean Sheather <dean@deansheather.com>
- Update go.mod to use Go 1.24.1
- Update GitHub Actions setup-go action to use Go 1.24.1
- Fix linting issues with golangci-lint by:
- Updating to golangci-lint v1.57.1 (more compatible with Go 1.24.1)
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <claude@anthropic.com>
The experimental functions in `golang.org/x/exp/slices` are now
available in the standard library since Go 1.21.
Reference: https://go.dev/doc/go1.21#slices
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Fixes https://github.com/coder/internal/issues/324
We had been using a `testutil.Context` in combination with a separate
`context.WithTimeout()` that smelled iffy to me.
Also reworked part of the second `require.Eventually` loop to pull a job
ID from the first one, and added some more logging to aid future
debugging.
Part of https://github.com/coder/internal/issues/150
- Stop creating the VPC since we rely on an existing VPC in the
scaletest GCP project now
- Add prometheus remote writer to the new prometheus deployment
Migrates us to `coder/websocket` v1.8.12 rather than `nhooyr/websocket` on an older version.
Works around https://github.com/coder/websocket/issues/504 by adding an explicit test for `xerrors.Is(err, io.EOF)` where we were previously getting `io.EOF` from the netConn.
Closes https://github.com/coder/internal/issues/148
This has been validated to have working proxies with the `small`
scenario.
- [x] multi-region gcp infrastructure
- [x] use cloudflare provider to automate dns entries
- [x] automate proxy registration
- [x] multi-regional proxies
- [x] move scenarios into locals
This excludes the infrastructure for `cert-manager` and `otel-collector`
and those will be included in followup PRs.
Run logs are currently dropped on the floor when workspacetraffic tests fail.
e.g. https://github.com/coder/coder/runs/32640144785
This prints the logs when we get a failure, like
```
run_test.go:341:
Error Trace: /Users/spike/repos/coder/scaletest/workspacetraffic/run_test.go:341
/Users/spike/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.8.darwin-arm64/src/runtime/asm_arm64.s:1222
Error: Received unexpected error:
badness:
github.com/coder/coder/v2/scaletest/workspacetraffic.(*Runner).Run
/Users/spike/repos/coder/scaletest/workspacetraffic/run.go:174
Test: TestRun/App
Messages: RUN LOGS:
2024-11-07 13:04:30.274 [debu] config agent_id=00000000-0000-0000-0000-000000000000 reconnecting_pty_id=95bb480f-cd26-4633-8332-328d44ffa997 height=25 width=80 tick_interval=1s bytes_per_tick=1024
2024-11-07 13:04:30.274 [debu] connect to workspace agent agent_id=00000000-0000-0000-0000-000000000000
2024-11-07 13:04:30.274 [info] sending traffic to workspace app agent_id=00000000-0000-0000-0000-000000000000 app=echo
2024-11-07 13:04:30.276 [debu] reading from agent agent_id=00000000-0000-0000-0000-000000000000
2024-11-07 13:04:30.276 [debu] writing to agent agent_id=00000000-0000-0000-0000-000000000000
2024-11-07 13:04:31.526 [debu] done reading from agent agent_id=00000000-0000-0000-0000-000000000000
2024-11-07 13:04:32.276 [debu] done writing to agent agent_id=00000000-0000-0000-0000-000000000000
END RUN LOGS
```
So, hopefully we can diagnose flakes.
* chore: allow CreateUser to accept multiple organizations
In a multi-org deployment, it makes more sense to allow for multiple
org memberships to be assigned at create. The legacy param will still
be honored.
* Handle sdk deprecation better by maintaining cli functions
Kacper Sawicki