Files
coder/codersdk/workspacesdk/agentconn_test.go
T
Ethan eeb2624549 fix: pin workspace agent API client to intended agent (#26600)
## Summary

The control-plane HTTP client used to talk to workspace agents followed
HTTP redirects and trusted the redirected host, letting a malicious
workspace agent bounce a coderd request onto a different agent on the
shared tailnet. Because the agent HTTP API on port 4 is unauthenticated
(it relies on tailnet reachability plus control-plane authorization),
this allowed cross-tenant file read/write and remote code execution.
This PR refuses redirects and pins every dial to the intended agent.

Closes CODAGT-668.

## Problem

`agentConn.apiClient` in `codersdk/workspacesdk/agentconn.go`
constructed an `http.Client` with no `CheckRedirect`, so Go's default
policy followed up to 10 redirects. Its custom `Transport.DialContext`
parsed the host from the (post-redirect) request URL and dialed that IP
over the shared tailnet, validating only that the port was
`AgentHTTPAPIServerPort` (4). It never pinned the connection to the
intended `AgentID` / `agentAddress()`.

A workspace owner (any regular org member, not just admins) controls
their own agent and can make its port-4 handler return a `3xx`
`Location` pointing at a victim agent's tailnet IP. When a control-plane
action (for example a chat tool or the HTTP MCP server) sends an agent
API request to the attacker's agent, coderd acts as a confused deputy
and replays the request against the victim:

- `301/302/303` rewrite POST to GET, but `307/308` preserve method and
body when the body is replayable. The real callers pass replayable
bodies, so a redirected `POST /api/v0/write-file` writes
attacker-controlled content into the victim workspace and a redirected
`POST /api/v0/processes/start` executes it, giving RCE on the victim
agent.

The dangerous callers run server-side on coderd's single deployment-wide
`ServerTailnet`, which is authorized to tunnel to any agent, so the
blast radius is cross-tenant / cross-organization (limited in practice
to victim agents coderd currently has a live tunnel to).

## Fix

In `agentConn.apiClient`:

- Set `CheckRedirect: http.ErrUseLastResponse` so the client never
follows a redirect. A `3xx` is surfaced to the caller as the response
(which the existing `ReadBodyAsError` path turns into an error) instead
of being replayed against another host.
- Capture the intended agent address once from `AgentID` (`agentAddr :=
netip.AddrPortFrom(c.agentAddress(), AgentHTTPAPIServerPort)`), reject
any dial whose host or port does not match it, and always dial that
pinned address rather than the URL-derived host.

In `coderd/aitasks.go`, the task app proxy client (`taskAppHTTPClient`)
also now sets `CheckRedirect: http.ErrUseLastResponse`. This client
dials through `agentConn.DialContext`, which already pins the host to
the originating workspace's agent (it takes only the port from the dial
address), so it was never cross-agent. The change is hardening for
parity so a malicious app cannot bounce the request to a different port
on the same agent.

## Hardening and defense in depth

The two layers are independent. `CheckRedirect` removes the
redirect-following behavior entirely, and the dial pinning guarantees
that even a request constructed with a foreign host can only ever reach
the intended agent. Removing either one in the future cannot, on its
own, reintroduce the cross-agent vector.

## Tests

- `codersdk/workspacesdk/agentconn_redirect_test.go` builds a three-peer
tailnet (client, attacker, victim). The attacker agent redirects to the
victim's port-4 URL, and the test asserts that `GET` `302`, `POST`
`307`, and `POST` `308` all return an error and that the victim is never
contacted.
- `coderd/aitasks_internal_test.go` adds
`TestTaskAppHTTPClient_RejectsRedirect`, which verifies the task app
client surfaces a `307` instead of following it to a stand-in victim.

## Why this closes the whole vulnerability class

`apiClient` is the only HTTP chokepoint to the agent port-4 API, so
fixing it covers every server-side caller:

- Every agent HTTP API method in `agentConn` funnels through
`apiClient`, either via `apiRequest`, a direct `apiClient(ctx).Do(...)`
(`ExecuteDesktopAction`), or as the websocket `HTTPClient`
(`WatchContainers`, `WatchGit`, `ConnectDesktopVNC`). The websocket
handshake matters here: `coder/websocket` follows `3xx` during the
handshake by default and only requires `101` on the final hop, but it
honors the underlying client's `CheckRedirect`, so reusing `apiClient`
closes the websocket paths too.
- The HTTP MCP server coderd hosts at `/api/experimental/mcp/http`
registers tools (`coder_workspace_bash`, `_write_file`, `_read_file`,
`_edit_files`, etc.) that reach the agent through
`workspacesdk.AgentConn` methods, so they go through `apiClient` and are
covered. The same is true for agent-hosted MCP, which coderd reaches
only via `agentConn.CallMCPTool` / `ListMCPTools`. coderd never opens an
MCP client connection directly to an agent over the tailnet.
- Raw-TCP agent services (reconnecting PTY, SSH, speedtest, generic
`DialContext`) speak non-HTTP protocols and have no redirect surface.
The workspace apps reverse proxy targets user app ports, not port 4,
forwards `3xx` to the browser rather than following them, and pins its
transport to the request's agent.
- `provisionerd` does not talk to the agent HTTP API at all.

No other server-side client follows redirects to an agent-controllable
tailnet host, so no further redirect changes are required for this
class.
2026-06-23 23:06:44 +10:00

265 lines
7.9 KiB
Go

package workspacesdk_test
import (
"context"
"errors"
"fmt"
"net"
"net/http"
"net/netip"
"strings"
"sync/atomic"
"testing"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/goleak"
"tailscale.com/tailcfg"
"github.com/coder/coder/v2/codersdk/workspacesdk"
"github.com/coder/coder/v2/tailnet"
"github.com/coder/coder/v2/tailnet/proto"
"github.com/coder/coder/v2/tailnet/tailnettest"
"github.com/coder/coder/v2/testutil"
)
// TestAgentConn_DialBoundedByRequestContext verifies that the
// transport dial behind the agent HTTP API stops when the request
// context ends. http.Transport detaches dial contexts from the
// request context so a pending dial can outlive its request and
// serve future ones, but the agent API client is request-scoped
// with keep-alives disabled, so a detached dial can never be
// reused. If the transport does not re-link cancellation, the dial
// goroutine stays blocked in AwaitReachable pinging an unreachable
// agent forever, even after the tailnet conn is closed, and leaks.
//
//nolint:paralleltest // goleak.IgnoreCurrent requires this test to run non-parallel.
func TestAgentConn_DialBoundedByRequestContext(t *testing.T) {
// goleak.IgnoreCurrent snapshots running goroutines, so this
// test must not run in parallel with other tests.
logger := testutil.Logger(t)
// Snapshot before the tailnet conn exists so everything spawned
// below, including the transport dial goroutine, is verified.
ignoreCurrent := goleak.IgnoreCurrent()
tailnetConn, err := tailnet.NewConn(&tailnet.Options{
Addresses: []netip.Prefix{tailnet.TailscaleServicePrefix.RandomPrefix()},
Logger: logger.Named("client"),
})
require.NoError(t, err)
t.Cleanup(func() {
_ = tailnetConn.Close()
})
conn := workspacesdk.NewAgentConn(tailnetConn, workspacesdk.AgentConnOptions{
AgentID: uuid.New(),
})
// No agent exists, so the transport dial blocks in
// AwaitReachable until the request context expires. The timeout
// only needs to be long enough for the dial goroutine to start;
// its expiry is the behavior under test.
ctx, cancel := context.WithTimeout(context.Background(), testutil.IntervalSlow)
defer cancel()
_, err = conn.ListeningPorts(ctx)
require.Error(t, err)
// Close the conn like test teardown would. The conn's own
// goroutines exit on close; the dial goroutine must have already
// exited when the request context expired.
err = tailnetConn.Close()
require.NoError(t, err)
goleak.VerifyNone(t, ignoreCurrent)
}
func TestAgentConnRejectsCrossAgentRedirects(t *testing.T) {
t.Parallel()
derpMap, _ := tailnettest.RunDERPAndSTUN(t)
cases := []struct {
name string
status int
invoke func(context.Context, workspacesdk.AgentConn) error
}{
{
name: "get 302",
status: http.StatusFound,
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
_, err := conn.ListeningPorts(ctx)
return err
},
},
{
name: "post 307",
status: http.StatusTemporaryRedirect,
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
return conn.WriteFile(ctx, "/tmp/attacker", strings.NewReader("redirect-body"))
},
},
{
name: "post 308",
status: http.StatusPermanentRedirect,
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
return conn.WriteFile(ctx, "/tmp/attacker", strings.NewReader("redirect-body"))
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
ctx := testutil.Context(t, testutil.WaitMedium)
clientID := uuid.New()
attackerID := uuid.New()
victimID := uuid.New()
clientConn, _ := newTailnetConn(t, derpMap, clientID, "client")
attackerConn, attackerIP := newTailnetConn(t, derpMap, attackerID, "attacker")
victimConn, victimIP := newTailnetConn(t, derpMap, victimID, "victim")
stitchTailnet(t, map[uuid.UUID]*tailnet.Conn{
clientID: clientConn,
attackerID: attackerConn,
victimID: victimConn,
})
var victimHit atomic.Bool
victimRouter := http.NewServeMux()
victimRouter.HandleFunc("/api/v0/listening-ports", func(rw http.ResponseWriter, _ *http.Request) {
victimHit.Store(true)
rw.Header().Set("Content-Type", "application/json")
_, _ = rw.Write([]byte(`{"ports":[]}`))
})
victimRouter.HandleFunc("/api/v0/write-file", func(rw http.ResponseWriter, _ *http.Request) {
victimHit.Store(true)
rw.WriteHeader(http.StatusOK)
})
serveTailnetHTTP(t, victimConn, victimRouter)
victimBaseURL := fmt.Sprintf("http://[%s]:%d", victimIP, workspacesdk.AgentHTTPAPIServerPort)
attackerRouter := http.NewServeMux()
attackerRouter.HandleFunc("/", func(rw http.ResponseWriter, r *http.Request) {
http.Redirect(rw, r, victimBaseURL+r.URL.RequestURI(), tc.status)
})
serveTailnetHTTP(t, attackerConn, attackerRouter)
require.True(t, clientConn.AwaitReachable(ctx, attackerIP))
require.True(t, clientConn.AwaitReachable(ctx, victimIP))
conn := workspacesdk.NewAgentConn(clientConn, workspacesdk.AgentConnOptions{
AgentID: attackerID,
})
err := tc.invoke(ctx, conn)
require.Error(t, err)
require.False(t, victimHit.Load())
})
}
}
// TestAgentConnAppHTTPClientRefusesRedirects verifies the app HTTP client does
// not follow redirects.
func TestAgentConnAppHTTPClientRefusesRedirects(t *testing.T) {
t.Parallel()
tailnetConn, err := tailnet.NewConn(&tailnet.Options{
Addresses: []netip.Prefix{tailnet.TailscaleServicePrefix.RandomPrefix()},
Logger: testutil.Logger(t),
})
require.NoError(t, err)
t.Cleanup(func() {
_ = tailnetConn.Close()
})
conn := workspacesdk.NewAgentConn(tailnetConn, workspacesdk.AgentConnOptions{
AgentID: uuid.New(),
})
client := conn.AppHTTPClient()
require.NotNil(t, client.CheckRedirect)
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "http://example.invalid/", nil)
require.NoError(t, err)
require.ErrorIs(t, client.CheckRedirect(req, nil), http.ErrUseLastResponse)
}
func newTailnetConn(t *testing.T, derpMap *tailcfg.DERPMap, id uuid.UUID, name string) (*tailnet.Conn, netip.Addr) {
t.Helper()
addr := tailnet.TailscaleServicePrefix.AddrFromUUID(id)
conn, err := tailnet.NewConn(&tailnet.Options{
ID: id,
Addresses: []netip.Prefix{netip.PrefixFrom(addr, 128)},
Logger: testutil.Logger(t).Named(name),
DERPMap: derpMap,
})
require.NoError(t, err)
t.Cleanup(func() {
assert.NoError(t, conn.Close())
})
return conn, addr
}
func serveTailnetHTTP(t *testing.T, conn *tailnet.Conn, handler http.Handler) {
t.Helper()
ln, err := conn.Listen("tcp", fmt.Sprintf(":%d", workspacesdk.AgentHTTPAPIServerPort))
require.NoError(t, err)
server := &http.Server{Handler: handler, ReadHeaderTimeout: testutil.WaitShort}
t.Cleanup(func() {
assert.NoError(t, server.Close())
assert.NoError(t, ln.Close())
})
go func() {
err := server.Serve(ln)
if err != nil && !errors.Is(err, net.ErrClosed) && !errors.Is(err, http.ErrServerClosed) {
assert.NoError(t, err)
}
}()
}
// stitchTailnet cross-programs every conn's node into every other conn, the
// N-peer analog of tailnet's stitch test helper, so the peers can reach each
// other without a coordinator.
func stitchTailnet(t *testing.T, conns map[uuid.UUID]*tailnet.Conn) {
t.Helper()
sendNode := func(srcID uuid.UUID, node *tailnet.Node) {
protoNode, err := tailnet.NodeToProto(node)
if !assert.NoError(t, err) {
return
}
for dstID, dst := range conns {
if dstID == srcID {
continue
}
err = dst.UpdatePeers([]*proto.CoordinateResponse_PeerUpdate{{
Id: srcID[:],
Node: protoNode,
Kind: proto.CoordinateResponse_PeerUpdate_NODE,
}})
assert.NoError(t, err)
}
}
for srcID, src := range conns {
src.SetNodeCallback(func(node *tailnet.Node) {
sendNode(srcID, node)
})
if node := src.Node(); node != nil {
sendNode(srcID, node)
}
}
t.Cleanup(func() {
for _, conn := range conns {
conn.SetNodeCallback(nil)
}
})
}