mirror of
https://github.com/coder/coder.git
synced 2026-09-22 21:22:17 +08:00
## Summary The control-plane HTTP client used to talk to workspace agents followed HTTP redirects and trusted the redirected host, letting a malicious workspace agent bounce a coderd request onto a different agent on the shared tailnet. Because the agent HTTP API on port 4 is unauthenticated (it relies on tailnet reachability plus control-plane authorization), this allowed cross-tenant file read/write and remote code execution. This PR refuses redirects and pins every dial to the intended agent. Closes CODAGT-668. ## Problem `agentConn.apiClient` in `codersdk/workspacesdk/agentconn.go` constructed an `http.Client` with no `CheckRedirect`, so Go's default policy followed up to 10 redirects. Its custom `Transport.DialContext` parsed the host from the (post-redirect) request URL and dialed that IP over the shared tailnet, validating only that the port was `AgentHTTPAPIServerPort` (4). It never pinned the connection to the intended `AgentID` / `agentAddress()`. A workspace owner (any regular org member, not just admins) controls their own agent and can make its port-4 handler return a `3xx` `Location` pointing at a victim agent's tailnet IP. When a control-plane action (for example a chat tool or the HTTP MCP server) sends an agent API request to the attacker's agent, coderd acts as a confused deputy and replays the request against the victim: - `301/302/303` rewrite POST to GET, but `307/308` preserve method and body when the body is replayable. The real callers pass replayable bodies, so a redirected `POST /api/v0/write-file` writes attacker-controlled content into the victim workspace and a redirected `POST /api/v0/processes/start` executes it, giving RCE on the victim agent. The dangerous callers run server-side on coderd's single deployment-wide `ServerTailnet`, which is authorized to tunnel to any agent, so the blast radius is cross-tenant / cross-organization (limited in practice to victim agents coderd currently has a live tunnel to). ## Fix In `agentConn.apiClient`: - Set `CheckRedirect: http.ErrUseLastResponse` so the client never follows a redirect. A `3xx` is surfaced to the caller as the response (which the existing `ReadBodyAsError` path turns into an error) instead of being replayed against another host. - Capture the intended agent address once from `AgentID` (`agentAddr := netip.AddrPortFrom(c.agentAddress(), AgentHTTPAPIServerPort)`), reject any dial whose host or port does not match it, and always dial that pinned address rather than the URL-derived host. In `coderd/aitasks.go`, the task app proxy client (`taskAppHTTPClient`) also now sets `CheckRedirect: http.ErrUseLastResponse`. This client dials through `agentConn.DialContext`, which already pins the host to the originating workspace's agent (it takes only the port from the dial address), so it was never cross-agent. The change is hardening for parity so a malicious app cannot bounce the request to a different port on the same agent. ## Hardening and defense in depth The two layers are independent. `CheckRedirect` removes the redirect-following behavior entirely, and the dial pinning guarantees that even a request constructed with a foreign host can only ever reach the intended agent. Removing either one in the future cannot, on its own, reintroduce the cross-agent vector. ## Tests - `codersdk/workspacesdk/agentconn_redirect_test.go` builds a three-peer tailnet (client, attacker, victim). The attacker agent redirects to the victim's port-4 URL, and the test asserts that `GET` `302`, `POST` `307`, and `POST` `308` all return an error and that the victim is never contacted. - `coderd/aitasks_internal_test.go` adds `TestTaskAppHTTPClient_RejectsRedirect`, which verifies the task app client surfaces a `307` instead of following it to a stand-in victim. ## Why this closes the whole vulnerability class `apiClient` is the only HTTP chokepoint to the agent port-4 API, so fixing it covers every server-side caller: - Every agent HTTP API method in `agentConn` funnels through `apiClient`, either via `apiRequest`, a direct `apiClient(ctx).Do(...)` (`ExecuteDesktopAction`), or as the websocket `HTTPClient` (`WatchContainers`, `WatchGit`, `ConnectDesktopVNC`). The websocket handshake matters here: `coder/websocket` follows `3xx` during the handshake by default and only requires `101` on the final hop, but it honors the underlying client's `CheckRedirect`, so reusing `apiClient` closes the websocket paths too. - The HTTP MCP server coderd hosts at `/api/experimental/mcp/http` registers tools (`coder_workspace_bash`, `_write_file`, `_read_file`, `_edit_files`, etc.) that reach the agent through `workspacesdk.AgentConn` methods, so they go through `apiClient` and are covered. The same is true for agent-hosted MCP, which coderd reaches only via `agentConn.CallMCPTool` / `ListMCPTools`. coderd never opens an MCP client connection directly to an agent over the tailnet. - Raw-TCP agent services (reconnecting PTY, SSH, speedtest, generic `DialContext`) speak non-HTTP protocols and have no redirect surface. The workspace apps reverse proxy targets user app ports, not port 4, forwards `3xx` to the browser rather than following them, and pins its transport to the request's agent. - `provisionerd` does not talk to the agent HTTP API at all. No other server-side client follows redirects to an agent-controllable tailnet host, so no further redirect changes are required for this class.
265 lines
7.9 KiB
Go
265 lines
7.9 KiB
Go
package workspacesdk_test
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"net"
|
|
"net/http"
|
|
"net/netip"
|
|
"strings"
|
|
"sync/atomic"
|
|
"testing"
|
|
|
|
"github.com/google/uuid"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
"go.uber.org/goleak"
|
|
"tailscale.com/tailcfg"
|
|
|
|
"github.com/coder/coder/v2/codersdk/workspacesdk"
|
|
"github.com/coder/coder/v2/tailnet"
|
|
"github.com/coder/coder/v2/tailnet/proto"
|
|
"github.com/coder/coder/v2/tailnet/tailnettest"
|
|
"github.com/coder/coder/v2/testutil"
|
|
)
|
|
|
|
// TestAgentConn_DialBoundedByRequestContext verifies that the
|
|
// transport dial behind the agent HTTP API stops when the request
|
|
// context ends. http.Transport detaches dial contexts from the
|
|
// request context so a pending dial can outlive its request and
|
|
// serve future ones, but the agent API client is request-scoped
|
|
// with keep-alives disabled, so a detached dial can never be
|
|
// reused. If the transport does not re-link cancellation, the dial
|
|
// goroutine stays blocked in AwaitReachable pinging an unreachable
|
|
// agent forever, even after the tailnet conn is closed, and leaks.
|
|
//
|
|
//nolint:paralleltest // goleak.IgnoreCurrent requires this test to run non-parallel.
|
|
func TestAgentConn_DialBoundedByRequestContext(t *testing.T) {
|
|
// goleak.IgnoreCurrent snapshots running goroutines, so this
|
|
// test must not run in parallel with other tests.
|
|
logger := testutil.Logger(t)
|
|
|
|
// Snapshot before the tailnet conn exists so everything spawned
|
|
// below, including the transport dial goroutine, is verified.
|
|
ignoreCurrent := goleak.IgnoreCurrent()
|
|
|
|
tailnetConn, err := tailnet.NewConn(&tailnet.Options{
|
|
Addresses: []netip.Prefix{tailnet.TailscaleServicePrefix.RandomPrefix()},
|
|
Logger: logger.Named("client"),
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() {
|
|
_ = tailnetConn.Close()
|
|
})
|
|
|
|
conn := workspacesdk.NewAgentConn(tailnetConn, workspacesdk.AgentConnOptions{
|
|
AgentID: uuid.New(),
|
|
})
|
|
|
|
// No agent exists, so the transport dial blocks in
|
|
// AwaitReachable until the request context expires. The timeout
|
|
// only needs to be long enough for the dial goroutine to start;
|
|
// its expiry is the behavior under test.
|
|
ctx, cancel := context.WithTimeout(context.Background(), testutil.IntervalSlow)
|
|
defer cancel()
|
|
_, err = conn.ListeningPorts(ctx)
|
|
require.Error(t, err)
|
|
|
|
// Close the conn like test teardown would. The conn's own
|
|
// goroutines exit on close; the dial goroutine must have already
|
|
// exited when the request context expired.
|
|
err = tailnetConn.Close()
|
|
require.NoError(t, err)
|
|
|
|
goleak.VerifyNone(t, ignoreCurrent)
|
|
}
|
|
|
|
func TestAgentConnRejectsCrossAgentRedirects(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
derpMap, _ := tailnettest.RunDERPAndSTUN(t)
|
|
cases := []struct {
|
|
name string
|
|
status int
|
|
invoke func(context.Context, workspacesdk.AgentConn) error
|
|
}{
|
|
{
|
|
name: "get 302",
|
|
status: http.StatusFound,
|
|
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
|
|
_, err := conn.ListeningPorts(ctx)
|
|
return err
|
|
},
|
|
},
|
|
{
|
|
name: "post 307",
|
|
status: http.StatusTemporaryRedirect,
|
|
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
|
|
return conn.WriteFile(ctx, "/tmp/attacker", strings.NewReader("redirect-body"))
|
|
},
|
|
},
|
|
{
|
|
name: "post 308",
|
|
status: http.StatusPermanentRedirect,
|
|
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
|
|
return conn.WriteFile(ctx, "/tmp/attacker", strings.NewReader("redirect-body"))
|
|
},
|
|
},
|
|
}
|
|
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
|
|
clientID := uuid.New()
|
|
attackerID := uuid.New()
|
|
victimID := uuid.New()
|
|
clientConn, _ := newTailnetConn(t, derpMap, clientID, "client")
|
|
attackerConn, attackerIP := newTailnetConn(t, derpMap, attackerID, "attacker")
|
|
victimConn, victimIP := newTailnetConn(t, derpMap, victimID, "victim")
|
|
stitchTailnet(t, map[uuid.UUID]*tailnet.Conn{
|
|
clientID: clientConn,
|
|
attackerID: attackerConn,
|
|
victimID: victimConn,
|
|
})
|
|
|
|
var victimHit atomic.Bool
|
|
victimRouter := http.NewServeMux()
|
|
victimRouter.HandleFunc("/api/v0/listening-ports", func(rw http.ResponseWriter, _ *http.Request) {
|
|
victimHit.Store(true)
|
|
rw.Header().Set("Content-Type", "application/json")
|
|
_, _ = rw.Write([]byte(`{"ports":[]}`))
|
|
})
|
|
victimRouter.HandleFunc("/api/v0/write-file", func(rw http.ResponseWriter, _ *http.Request) {
|
|
victimHit.Store(true)
|
|
rw.WriteHeader(http.StatusOK)
|
|
})
|
|
serveTailnetHTTP(t, victimConn, victimRouter)
|
|
|
|
victimBaseURL := fmt.Sprintf("http://[%s]:%d", victimIP, workspacesdk.AgentHTTPAPIServerPort)
|
|
attackerRouter := http.NewServeMux()
|
|
attackerRouter.HandleFunc("/", func(rw http.ResponseWriter, r *http.Request) {
|
|
http.Redirect(rw, r, victimBaseURL+r.URL.RequestURI(), tc.status)
|
|
})
|
|
serveTailnetHTTP(t, attackerConn, attackerRouter)
|
|
|
|
require.True(t, clientConn.AwaitReachable(ctx, attackerIP))
|
|
require.True(t, clientConn.AwaitReachable(ctx, victimIP))
|
|
|
|
conn := workspacesdk.NewAgentConn(clientConn, workspacesdk.AgentConnOptions{
|
|
AgentID: attackerID,
|
|
})
|
|
|
|
err := tc.invoke(ctx, conn)
|
|
require.Error(t, err)
|
|
require.False(t, victimHit.Load())
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestAgentConnAppHTTPClientRefusesRedirects verifies the app HTTP client does
|
|
// not follow redirects.
|
|
func TestAgentConnAppHTTPClientRefusesRedirects(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tailnetConn, err := tailnet.NewConn(&tailnet.Options{
|
|
Addresses: []netip.Prefix{tailnet.TailscaleServicePrefix.RandomPrefix()},
|
|
Logger: testutil.Logger(t),
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() {
|
|
_ = tailnetConn.Close()
|
|
})
|
|
|
|
conn := workspacesdk.NewAgentConn(tailnetConn, workspacesdk.AgentConnOptions{
|
|
AgentID: uuid.New(),
|
|
})
|
|
|
|
client := conn.AppHTTPClient()
|
|
require.NotNil(t, client.CheckRedirect)
|
|
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "http://example.invalid/", nil)
|
|
require.NoError(t, err)
|
|
require.ErrorIs(t, client.CheckRedirect(req, nil), http.ErrUseLastResponse)
|
|
}
|
|
|
|
func newTailnetConn(t *testing.T, derpMap *tailcfg.DERPMap, id uuid.UUID, name string) (*tailnet.Conn, netip.Addr) {
|
|
t.Helper()
|
|
|
|
addr := tailnet.TailscaleServicePrefix.AddrFromUUID(id)
|
|
conn, err := tailnet.NewConn(&tailnet.Options{
|
|
ID: id,
|
|
Addresses: []netip.Prefix{netip.PrefixFrom(addr, 128)},
|
|
Logger: testutil.Logger(t).Named(name),
|
|
DERPMap: derpMap,
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() {
|
|
assert.NoError(t, conn.Close())
|
|
})
|
|
|
|
return conn, addr
|
|
}
|
|
|
|
func serveTailnetHTTP(t *testing.T, conn *tailnet.Conn, handler http.Handler) {
|
|
t.Helper()
|
|
|
|
ln, err := conn.Listen("tcp", fmt.Sprintf(":%d", workspacesdk.AgentHTTPAPIServerPort))
|
|
require.NoError(t, err)
|
|
|
|
server := &http.Server{Handler: handler, ReadHeaderTimeout: testutil.WaitShort}
|
|
t.Cleanup(func() {
|
|
assert.NoError(t, server.Close())
|
|
assert.NoError(t, ln.Close())
|
|
})
|
|
|
|
go func() {
|
|
err := server.Serve(ln)
|
|
if err != nil && !errors.Is(err, net.ErrClosed) && !errors.Is(err, http.ErrServerClosed) {
|
|
assert.NoError(t, err)
|
|
}
|
|
}()
|
|
}
|
|
|
|
// stitchTailnet cross-programs every conn's node into every other conn, the
|
|
// N-peer analog of tailnet's stitch test helper, so the peers can reach each
|
|
// other without a coordinator.
|
|
func stitchTailnet(t *testing.T, conns map[uuid.UUID]*tailnet.Conn) {
|
|
t.Helper()
|
|
|
|
sendNode := func(srcID uuid.UUID, node *tailnet.Node) {
|
|
protoNode, err := tailnet.NodeToProto(node)
|
|
if !assert.NoError(t, err) {
|
|
return
|
|
}
|
|
for dstID, dst := range conns {
|
|
if dstID == srcID {
|
|
continue
|
|
}
|
|
err = dst.UpdatePeers([]*proto.CoordinateResponse_PeerUpdate{{
|
|
Id: srcID[:],
|
|
Node: protoNode,
|
|
Kind: proto.CoordinateResponse_PeerUpdate_NODE,
|
|
}})
|
|
assert.NoError(t, err)
|
|
}
|
|
}
|
|
|
|
for srcID, src := range conns {
|
|
src.SetNodeCallback(func(node *tailnet.Node) {
|
|
sendNode(srcID, node)
|
|
})
|
|
if node := src.Node(); node != nil {
|
|
sendNode(srcID, node)
|
|
}
|
|
}
|
|
|
|
t.Cleanup(func() {
|
|
for _, conn := range conns {
|
|
conn.SetNodeCallback(nil)
|
|
}
|
|
})
|
|
}
|