Fixes: https://github.com/coder/internal/issues/1451
## Problem
Flaky test
`TestStreamAgentReinitEvents/doesn't_transmit_events_if_the_transmitter_context_is_canceled`
(coder/internal#1451):
```
agentsdk_test.go:84:
Error: Received unexpected error:
Get "http://127.0.0.1:XXXXX": net/http: HTTP/1.x transport connection broken: http: CloseIdleConnections called
```
## Root cause
The subtests used `client := &http.Client{}`. A client with a nil
`Transport` uses the process-global `http.DefaultTransport`, which is
shared by every parallel test in the test binary.
`httptest.Server.Close()` calls
`http.DefaultTransport.CloseIdleConnections()`. When any other parallel
test closes its `httptest.Server` while this test's request is in
flight, the shared transport tears the connection down and
`client.Do(req)` fails with `http: CloseIdleConnections called`. This is
the same class of flake already documented/fixed in `testutil/oauth2.go`
and the `mcphttpclient` helpers, and related to coder/internal#1020.
## Fix
Give each client a dedicated `*http.Transport` (`&http.Client{Transport:
&http.Transport{}}`) so cross-test `CloseIdleConnections` calls cannot
break its requests. The construction is extracted into a small
`newReinitTestClient()` helper used by all three subtests, with a
comment documenting the reason.
## Verification
`go test ./codersdk/agentsdk -run TestStreamAgentReinitEvents -count=20`
passes.
The flake was reproduced against the exact failing subtest logic (real
`NewSSEAgentReinitTransmitter` with a pre-canceled transmit context,
same client pattern) under a `CloseIdleConnections` stress loop:
- Fix reverted to `&http.Client{}`: reliably FAILs (e.g. 15 broken
requests in 10s).
- Fix present: 0 broken requests across repeated runs.
<details>
<summary>Optional stress harness to reproduce/verify locally (not
committed)</summary>
Drop this into `codersdk/agentsdk/` as a throwaway `*_test.go` file. It
runs the verbatim body of the failing subtest in a loop while parallel
goroutines call `CloseIdleConnections` (exactly what
`httptest.Server.Close()` does). With the fix present it reports
`closeIdleErrs=0`; revert `newReinitTestClient` to `&http.Client{}` to
reproduce.
```go
package agentsdk_test
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/google/uuid"
"cdr.dev/slog/v3/sloggers/slogtest"
"github.com/coder/coder/v2/codersdk/agentsdk"
)
func TestFlakeReproRealSubtest(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var wg sync.WaitGroup
var closeIdleErrs int64
var sample atomic.Value
defaultTransport := http.DefaultTransport.(*http.Transport)
for range 8 {
wg.Add(1)
go func() {
defer wg.Done()
for ctx.Err() == nil {
defaultTransport.CloseIdleConnections()
}
}()
}
for range 32 {
wg.Add(1)
go func() {
defer wg.Done()
for ctx.Err() == nil {
// Verbatim body of the failing subtest.
eventToSend := agentsdk.ReinitializationEvent{
WorkspaceID: uuid.New(),
Reason: agentsdk.ReinitializeReasonPrebuildClaimed,
}
events := make(chan agentsdk.ReinitializationEvent, 1)
events <- eventToSend
transmitCtx, cancelTransmit := context.WithCancel(context.Background())
cancelTransmit()
transmitErrCh := make(chan error, 1)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
transmitter := agentsdk.NewSSEAgentReinitTransmitter(slogtest.Make(t, nil), w, r)
transmitErrCh <- transmitter.Transmit(transmitCtx, events)
}))
req, err := http.NewRequestWithContext(ctx, "GET", srv.URL, nil)
if err != nil {
srv.Close()
continue
}
client := newReinitTestClient() // revert to &http.Client{} to reproduce
resp, err := client.Do(req)
if err != nil {
if strings.Contains(err.Error(), "CloseIdleConnections called") {
atomic.AddInt64(&closeIdleErrs, 1)
sample.CompareAndSwap(nil, err.Error())
}
srv.Close()
continue
}
resp.Body.Close()
srv.Close()
}
}()
}
wg.Wait()
t.Logf("closeIdleErrs=%d", atomic.LoadInt64(&closeIdleErrs))
if n := atomic.LoadInt64(&closeIdleErrs); n > 0 {
t.Fatalf("reproduced coder/internal#1451 on the real subtest: %d requests broken (e.g. %v)", n, sample.Load())
}
}
```
Example output with the fix reverted to `&http.Client{}`:
```
flakerepro_test.go:88: closeIdleErrs=15
flakerepro_test.go:90: reproduced coder/internal#1451 on the real subtest: 15 requests broken (e.g. Get "http://127.0.0.1:43755": net/http: HTTP/1.x transport connection broken: http: CloseIdleConnections called)
--- FAIL: TestFlakeReproRealSubtest (10.14s)
```
With the fix present: `closeIdleErrs=0` and PASS.
</details>
---
Generated by Coder Agents on behalf of @mtojek.
## Problem
When a prebuilt workspace is claimed, the agent reinitializes via a
single fire-and-forget pubsub event over SSE. If the agent's SSE
connection is interrupted at claim time, the event is permanently lost —
the workspace is stuck with no self-healing path.
Additionally, regular (non-prebuild) workspaces had no way to opt out of
the `/reinit` polling loop — agents would reconnect indefinitely to an
endpoint that would never send them anything useful.
## Root Cause
`workspaceAgentReinit` fetches the workspace (with its current
`owner_id`) via `GetWorkspaceByAgentID`, but never checked whether a
claim already happened. It only subscribed to pubsub for future events.
The database already has durable claim state (`owner_id` changes from
`PrebuildsSystemUserID` to the real user), but no layer ever consulted
it on reconnection.
## Solution
### Server-side durable check with first-build-initiator gating
**TOCTOU-safe ordering**: Subscribe to pubsub claim events *before* any
durable checks, so a claim that fires during the check is buffered in
the channel rather than lost.
**First-build-initiator gating**: When `!workspace.IsPrebuild()` (owner
is no longer the system user), look up the first build's `InitiatorID`.
The prebuild reconciler always uses `PrebuildsSystemUserID` as the
initiator. This distinguishes claimed prebuilds from regular workspaces
without any SQL schema changes.
- **Regular workspace** (first build initiator ≠ system user) → **409
Conflict**, agent stops reconnecting
- **Claimed prebuild, build completed** → pre-seed channel with reinit
event and close it, transmitter delivers one-shot then exits
- **Claimed prebuild, build in-progress** → fall through to pubsub
subscription, agent waits for completion event
- **Unclaimed prebuild** → pubsub subscription (existing happy path)
### Declarative reinit events (defense-in-depth)
- Added `UserID` field to `ReinitializationEvent` with JSON tags
- Switched pubsub serialization from raw string to JSON (with
backward-compat fallback for rolling upgrades)
- Populated `UserID` at both the publish site and the durable check
### Agent SDK: 409 handling
`WaitForReinitLoop` detects 409 Conflict from the server and closes the
`reinitEvents` channel, cleanly exiting the retry goroutine.
### Agent CLI: fixed two bugs + added reinitCtx
- **Closed channel (`!ok`)**: now blocks on `<-ctx.Done()` instead of
`continue`, keeping the current agent running. Previously this would
leak agents by skipping `agnt.Close()` and re-entering the loop.
- **Duplicate owner reinit**: cancels `reinitCtx` (stops the reinit
goroutine), then blocks on `<-ctx.Done()`. Previously `continue` would
skip cleanup and create a new agent on the next loop iteration.
- **`reinitCtx`**: a cancellable child of `ctx` passed to
`WaitForReinitLoop`, allowing the agent to stop the reinit HTTP polling
after reinit completes.
### Agent-side idempotency
Tracks `lastOwnerID` in the agent reinit loop — duplicate events for the
same owner are skipped.
## Testing
- **"unclaimed prebuild receives reinit via pubsub"**: prebuild owned by
system user, pubsub event triggers reinit
- **"claimed prebuild receives one-shot reinit on reconnect"**: first
build by system user, owner changed, build completed → immediate reinit
(no pubsub needed)
- **"claimed prebuild waits during in-progress claim build"**: claimed
but build still running → no reinit until build completes
- **"regular workspace gets 409"**: first build by real user → 409
Conflict, agent stops polling
- Updated claim publisher/listener tests: verify `UserID` survives JSON
round-trip + backward compat with raw string payloads
- Updated SSE round-trip test: verify `UserID` survives transmit →
receive cycle
Fixes#22359
## Rolling upgrade note
During a rolling deploy where old coderd instances coexist with new
ones, the pubsub `ReinitializationEvent` has a new `workspace_id` field
(JSON key `workspace_id`). Old publishers send a raw reason string
instead of JSON; the new listener gracefully falls back by treating the
entire payload as the reason and filling in `WorkspaceID` from context.
The only visible effect during the upgrade window is that `WorkspaceID`
may be the zero UUID in agent-side logs — this is cosmetic and resolves
once all instances are updated.
## Problem
When the git askpass flow triggered diff status refreshes, it updated
**every chat** connected to the workspace. This was wasteful and could
cause confusing status updates on unrelated chats.
## Solution
Thread the chat ID through the entire git askpass flow so only the chat
that initiated the git operation gets updated:
1. **`coderd/chatd/chattool/execute.go`** — Sets `CODER_CHAT_ID` env var
on spawned processes (alongside the existing `CODER_CHAT_AGENT`)
2. **`cli/gitaskpass.go`** — Reads `CODER_CHAT_ID` from the environment
and sends it as a `chat_id` query parameter in the `ExternalAuthRequest`
3. **`codersdk/agentsdk/agentsdk.go`** — Adds `ChatID` field to
`ExternalAuthRequest` and encodes it as a query param
4. **`coderd/workspaceagents.go`** — Parses `chat_id` query param and
passes it through to `storeChatGitRef` and
`triggerWorkspaceChatDiffStatusRefresh`
5. **`coderd/chats.go`** — `storeChatGitRef` and
`refreshWorkspaceChatDiffStatuses` now scope updates to just the
initiating chat when a chat ID is provided, falling back to
all-workspace-chats behavior for backwards compatibility (non-chat git
operations)
Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
This PR makes the initial steps at removing usage of the global Go HTTP
client, which was seen to have impacts on test flakiness in
https://github.com/coder/internal/issues/1020. The first commit removes
uses from tests, with the exception of one test that is tightly coupled
to the default client. The second commit makes easy/low-risk removals
from application code. This should have some impact to reduce test flakiness.
Refactors Agent instance identity to be a SessionTokenProvider.
Refactors the CLI to create Agent clients via a centralized function, rather than add-hoc via individual command handlers and their flags.
This allows commands besides `coder agent`, but which still use the agent identity, to support instance identity authentication.
Fixes#19111 by unifying all API requests to go thru the SessionTokenProvider for auth credentials.
The agentsdk currently does a remap of the DERP map to change the
EmbeddedRelay node's URL to match the agent's access URL.
This PR makes changes to the `workspacesdk` (used by clients like the
CLI) and `vpn` (used by Coder Desktop) to match this behavior.
This enables us the ability to try Coder clients in dogfood over a VPN
without changing the global access URL.
This pull request allows coder workspace agents to be reinitialized when
a prebuilt workspace is claimed by a user. This facilitates the transfer
of ownership between the anonymous prebuilds system user and the new
owner of the workspace.
Only a single agent per prebuilt workspace is supported for now, but
plumbing has already been done to facilitate the seamless transition to
multi-agent support.
---------
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
Co-authored-by: Danny Kopping <dannykopping@gmail.com>
Marcin Tojek