Follow-ups from #27627
- Memoizes `Config.Git()` with a mutex so the provider's ETag response
cache survives across calls. Only successful construction is cached;
errors are retried.
- Moves the HTTP client onto `Config.HTTPClient`, wired through
`ConvertConfig`, so `Git()` no longer takes a per-call argument that
would be silently ignored after memoization.
- `newGitHub` and `newGitLab` now return `(Provider, error)`,
eliminating the typed-nil-interface class in `gitprovider.New` rather
than the single instance.
- Gates the 304 branch on a `haveCached` flag instead of a nil body
check.
- Only caches bodies that decode successfully, preventing poisoned
entries.
- Keys the response cache on the full token digest rather than a
truncated prefix.
- Tests added: `TestConfigGitMemoizesProvider`,
`TestConfigGitRetriesOnConstructorError`,
`TestGitLabConstructorErrorReturnsNilInterface`,
`TestResponseCacheStore`,
`TestConditionalRequestReuse/MalformedResponseNotCached`;
`TestConvertYAML/CustomScopesAndEndpoint` now asserts
`Config.HTTPClient` wiring.
Follow-ups tracked in #28139, #28140, #28141, #28142.
> 🤖 Generated by Coder Agents on behalf of @johnstcn.
When `ValidateToken` keeps a token because the external auth validation
endpoint was rate-limited (a `403` with rate-limit headers or a `429`),
it returns `valid=true` without provider confirmation. Previously this
happened silently, so operators couldn't tell a provider-confirmed token
from one kept optimistically during a rate limit.
This adds a `Logger` to `externalauth.Config` and emits a `Warn` (with
`provider_id`, `provider_type`, `status_code`, and `reason`) on those
rate-limit branches. It also adds a
`coderd_oauth2_external_requests_rate_limited_total{name, source,
status_code}` counter, incremented in the instrumented round tripper
whenever a provider returns a rate-limited response. The rate-limit
detection is the shared `xhttp.IsRateLimited` (in `coderd/util/xhttp`),
used by both the tripper and `ValidateToken` so the metric and the
validation decision share one definition; no extra wiring is needed
since `ValidateToken` already routes through the instrumented client
with `source="ValidateToken"`.
One deliberate behavioral change rides along: rate-limit detection now
also recognizes the unprefixed `RateLimit-Remaining` header (GitLab, and
the IETF draft rate-limit headers), so a `403` with
`RateLimit-Remaining: 0` is treated as optimistically valid where it was
previously treated as revoked. All other valid/invalid decisions are
unchanged. `TestValidateToken` asserts the warning's fields on the
rate-limited cases and no warning for revocations, `401`, and confirmed
responses; `promoauth` and `xhttp` tests cover the detector and the new
counter.
<details>
<summary>Manual testing</summary>
The signals fire on the external-auth status check (`GET
/api/v2/external-auth/{id}`), which calls `ValidateToken`. To force a
rate-limited response, point a provider's `validate_url` at a mock that
returns the rate-limit shape:
1. Run a mock returning `429` on one path and `403` +
`X-RateLimit-Remaining: 0` on another.
2. Start `coder server` with `--prometheus-enable` and external auth
providers whose `validate_url` point at those mock paths (e.g.
`CODER_EXTERNAL_AUTH_0_VALIDATE_URL=http://127.0.0.1:5599/429`).
3. Create a stored link, either complete the OAuth flow, or insert a row
into `external_auth_links` with a future `oauth_expiry` (token contents
are irrelevant; the mock rejects regardless).
4. `curl` the status endpoint with a session token, then check:
- coderd logs for the `Warn` (`reason=status_code` for `429`,
`reason=rate_limit_headers` for `403`),
- the metrics endpoint for
`coderd_oauth2_external_requests_rate_limited_total{...,status_code="429"|"403"}`.
Notes: `scripts/testidp -429` only rate-limits `/oauth2/userinfo`, not
the `/external-auth-validate/...` path, so it does not exercise this;
use a mock `validate_url`. The default Prometheus port `2112` may
already be taken on dogfood workspaces, set `CODER_PROMETHEUS_ADDRESS`
to a free port.
</details>
🤖 Generated with the help of Coder Agents on behalf of @jscottmiller.
Fixes#27627.
## What
The chat diff-status gitsync worker polls open pull requests on a fixed
10s interval and re-downloads the full JSON body every tick, even when
nothing changed, because the GitHub client never sends `If-None-Match` /
ETag.
This adds a small, concurrency-safe, bounded in-memory ETag+body cache
(`coderd/externalauth/gitprovider/conditional.go`) and wires it into
`githubProvider.decodeJSON`. When an ETag is cached for a request, we
send `If-None-Match`; on `304 Not Modified` we decode the cached body;
on `200` we cache `{etag, body}` when an ETag is present and the body is
under a size cap.
## Why
`304` responses do not count against GitHub's primary rate limit, but
full `200`s do. Today every unchanged poll burns quota that the same
token also needs for interactive Git and API operations, so busy
instances can hit rate-limit errors and stalls elsewhere. Unchanged PRs
now revalidate for free with no behavior change; only genuine changes
transfer a body.
## Details
- Cache key = request URL + a hash of the token, so one token's response
is never served under another; raw tokens are not retained.
- Bounded by entry count (LRU eviction, default 2048) and per-body size
(1 MiB) to cap memory.
- Scope limited to the JSON reads through `decodeJSON`; the raw-diff
path (`fetchDiff`, up to `MaxDiffSize`) is intentionally left out to
avoid caching large bodies.
## Tests
`TestConditionalRequestReuse` in `github_test.go` covers:
- `NotModifiedReusesCachedBody` — a warm poll sends `If-None-Match` with
the prior ETag and reuses the cached body on `304`, yielding the same
result with exactly two upstream requests.
- `DifferentTokenDoesNotShareCache` — a different token never sends
another token's cached ETag.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4959e1f9-f8e6-4e97-a487-f395a0123c79
This can cause bad refresh token errors, since it can only be used once.
Looks like there was an attempt to fix this by checking the database
after a failed refresh, but of course this depends on the first request
having updated the database in time, so both that and this fix are
required to fully solve.
Under CI load the request's 10ms revoke timeout could expire before
the request reached the FakeIDP revoke handler. The handler never
ran, so the test's wait for it to finish blocked until the 25s test
context expired instead of passing quickly.
Raise `RevokeTimeout` to 100ms so the request has ~10x more headroom to
reach the handler under load. After RevokeToken returns, check a
`handlerStarted` signal before asserting: this anchors the
`DeadlineExceeded` assertion to a request that was actually in flight,
and turns any residual scheduling race into a fast, labeled failure
instead of a hang.
Unblock the handler on the early-exit path with a `t.Cleanup`. It must
be registered after the FakeIDP setup so LIFO runs it before the
server's `Close()`; otherwise a handler that dispatched late would
block `Close()` and hang teardown until the test timeout. Drop the
previous `time.Sleep` watchdog and the handler-done channel, since the
FakeIDP server's `Close()` already joins the in-flight handler.
Refs: https://linear.app/codercom/issue/PLAT-317
`TestRefreshToken/RefreshRetries` flakes on Windows. The subtest
disables transient-failure refresh retries by setting
`RefreshRetryTimeout = time.Nanosecond`, but a near-zero timeout cannot
deterministically prevent a retry: on coarse-clock platforms the 1ns
deadline may not register as expired until after the first refresh
attempt completes, and `retry.Wait`'s first delay is zero, so an extra
IDP refresh attempt slips through and the attempt-count assertion fails
with `refreshCount = totalRefreshes + 1`.
A negative `RefreshRetryTimeout` now disables transient-failure retries
explicitly so exactly one refresh attempt is made, and the test sets
`-1` instead of `time.Nanosecond`. The retry config fields are only set
from tests, so default refresh behavior is unchanged.
Closes https://github.com/coder/internal/issues/1550 (PLAT-293)
<details>
<summary>Root cause analysis</summary>
1. The test sets `RefreshRetryTimeout = time.Nanosecond` intending "no
retries".
2. `refreshTokenWithRetry` creates `context.WithTimeout(ctx, 1ns)`. On
Linux this context is canceled synchronously at creation: consecutive
`time.Now()` reads differ by more than 1ns, so `context.WithDeadline`
observes `time.Until(deadline) <= 0`. The `retryCtx.Err() != nil` guard
then deterministically stops after one attempt.
3. On Windows, `time.Now()` is coarse, so both clock reads inside
`WithTimeout` can return the same instant, and a real 1ns timer is
scheduled instead of synchronous cancellation.
4. The fake IDP is served in-process, so the first refresh attempt can
complete before that timer fires. `retryCtx.Err()` is still nil and
`retry.Wait`'s first delay is zero, so a second refresh attempt happens.
5. `require.Equal(t, refreshCount, totalRefreshes)` then fails with
`expected: 2, actual: 1` (or `4 vs 3` when the race hits a later loop
iteration), matching all CI occurrences.
Timing-based test-side mitigations cannot close this race, so the fix
adds explicit retry-disable semantics instead. `RefreshRetries` passed
100 consecutive local runs with the change.
</details>
*This PR was generated by Coder Agents on behalf of @jscottmiller.*
## Summary
Wraps external auth token refresh in an exponential-backoff retry so a
brief upstream hiccup (5xx, network timeout, rate-limited 429) no longer
surfaces as an `InvalidTokenError` and forces users to re-authenticate.
GitHub in particular has been flaky enough lately that this is hitting
real users.
## Behavior
- `(*Config).RefreshToken` now calls a helper that retries the
`TokenSource.Token()` exchange with exponential backoff (250ms → 2s),
bounded by a 10s total budget.
- Errors classified as permanent by `isFailedRefresh` (e.g.
`bad_refresh_token`, `invalid_grant`, `unauthorized_client`, ...) skip
the retry loop. Retrying a permanent failure wastes the refresh quota
and, on providers with single-use refresh tokens, can mask a legitimate
concurrent winner with repeated `bad_refresh_token` responses.
- Refreshes with an empty refresh token still short-circuit without
making an API call.
- The existing concurrent-refresh-race detection and optimistic-lock
paths are unchanged.
## Tunables
Three new `time.Duration` fields on `externalauth.Config`
(`RefreshRetryInitialBackoff`, `RefreshRetryMaxBackoff`,
`RefreshRetryTimeout`) let callers override the defaults. They default
to zero, which falls back to the package defaults, so existing call
sites are unaffected. The fields exist primarily so tests can dial the
timing way down without touching package globals (and therefore without
serializing parallel tests).
## Tests
- `TestRefreshToken/RefreshRetries` now disables internal retries via
`RefreshRetryTimeout = time.Nanosecond` so its existing "1 IDP call per
`RefreshToken` invocation" assertion still holds. Otherwise its
assertions are unchanged.
- New `TestRefreshToken/RefreshTokenWithBackoff` simulates 3 transient
5xx failures followed by success and verifies the refresh ultimately
succeeds with 4 total IDP attempts.
- New `TestRefreshToken/RefreshTokenBackoffPermanentError` returns
`bad_refresh_token` and verifies the refresh is **not** retried even
with a generous 1s budget.
<details>
<summary>Why the explicit <code>retryCtx.Err()</code> guard?</summary>
`retry.Retrier.Wait` `select`s between `time.After(delay)` and
`ctx.Done()`. The first call has `delay == 0`, so `time.After(0)` and an
already-cancelled context both fire immediately and Go picks the case
nondeterministically. Without the guard, a near-zero retry budget would
still trigger an unwanted extra refresh attempt roughly half the time,
which would have made the `RefreshRetries` test flaky.
</details>
This PR was opened by a Coder agent on behalf of @kylecarbs.
This change uses separate http clients/transports in TestValidateToken
subtests. Previously parallel subtests of TestValidateToken shared
a http.DefaultTransport. When one subtest's httptest.Server.Close() ran in
t.Cleanup, it called http.DefaultTransport.CloseIdleConnections, which
could interrupt connection(s) used in another subtest.
<!--
If you have used AI to produce some or all of this PR, please ensure you
have read our [AI Contribution
guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING)
before submitting.
-->
Fixes https://github.com/coder/coder/issues/17069
Builds on #24332 and #24334 which addressed token persistence and rate
limit handling.
## Problem
When multiple concurrent requests race to refresh an expiring external
auth token, providers with single-use refresh tokens (e.g., GitHub Apps)
reject all but the first refresh attempt with `bad_refresh_token`. The
losing request caches this transient error in the
`oauth_refresh_failure_reason` database column and clears the refresh
token, blocking all subsequent refresh attempts until the user manually
re-authenticates.
This is common for users with multiple terminals, IDE connections, or
workspaces open, all of which poll the external auth endpoint and
trigger concurrent refreshes when the token nears expiry. Database
analysis showed 5 of 7 affected users failed within 5-10 seconds of
token expiry, matching the Go oauth2 library's `expiryDelta` window.
## Fix
Before caching a `bad_refresh_token` failure, re-read the external auth
link from the database. If the refresh token has changed (indicating a
concurrent caller already refreshed successfully), return the winner's
updated link instead of writing a failure. An empty-string guard ensures
a token cleared by another loser isn't mistaken for a winner's
successful refresh.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Garrett Delfosse <garrett@coder.com>
ValidateToken treated all 403 responses as "token invalid," including
GitHub rate limits. isFailedRefresh included 403 in the status code
fallthrough, destroying tokens on rate-limited refresh attempts.
Split the combined 401/403 check in ValidateToken into a switch on
status code. On 403, inspect X-RateLimit-Remaining and Retry-After
headers; if either indicates a rate limit, return optimistically valid.
Handle 429 the same way. Plain 403 without rate-limit headers preserves
the existing invalid-token behavior.
Add incorrect_client_credentials and invalid_client to isFailedRefresh
error code switch. Remove 403 from the status code fallthrough since no
known provider returns 403 from the token endpoint.
GitHub rotates refresh tokens on use, invalidating the old token
immediately. If post-refresh validation fails (e.g. rate-limited
403 from /user), the new token was silently discarded because the
DB save only happened after successful validation. The next refresh
attempt would use the stale refresh token, fail permanently, and
destroy the token.
Move the UpdateExternalAuthLink call to immediately after
TokenSource.Token() succeeds. The post-validation save block is
removed (dead code after the early save). The DB write uses a
detached context (context.WithoutCancel) so a canceled request
cannot prevent persistence of the already-consumed refresh token.
Adds the `head_branch` field (the source/feature branch name of a PR) to
the diff status pipeline. Previously only `base_branch` (target branch)
and the head commit SHA were captured from the GitHub API, but not the
head branch name itself.
## Changes
- **Migration 438**: Add `head_branch` nullable TEXT column to
`chat_diff_statuses`
- **gitprovider**: Parse `head.ref` from the GitHub API response
(alongside `head.sha`) and add `HeadBranch` to `PRStatus`
- **gitsync**: Wire `HeadBranch` through `refreshOne()` into the DB
upsert params
- **worker**: Map `HeadBranch` in `chatDiffStatusFromRow()`
- **coderd**: Convert `HeadBranch` in `convertChatDiffStatus()`
- **codersdk**: Expose as `head_branch` (`*string`, omitempty) in
`ChatDiffStatus` API response
- **Tests**: Updated `github_test.go` pull JSON fixtures and assertions
_Disclaimer: implemented by a Coder Agent using Claude Opus 4.6._
Marks the injected MCP approach in AI Bridge as deprecated across the
codebase.
## Changes
- **`codersdk/deployment.go`**: Deprecated `ExternalAuthConfig.MCPURL`,
`.MCPToolAllowRegex`, `.MCPToolDenyRegex` fields; deprecated and hid the
`--aibridge-inject-coder-mcp-tools` server flag; deprecated
`AIBridgeConfig.InjectCoderMCPTools`.
- **`coderd/externalauth/externalauth.go`**: Deprecated `Config.MCPURL`,
`.MCPToolAllowRegex`, `.MCPToolDenyRegex`.
- **`enterprise/aibridgedserver/aibridgedserver.go`**: Added runtime
deprecation warning when `CODER_AIBRIDGE_INJECT_CODER_MCP_TOOLS` is
enabled; deprecated `getCoderMCPServerConfig`.
- **`enterprise/aibridged/mcp.go`**: Deprecated `MCPProxyBuilder`
interface and `MCPProxyFactory` struct.
- **`docs/ai-coder/ai-bridge/mcp.md`**: Added deprecation warning
banner.
Adds `pull_request_title` and `pull_request_draft` to the chat diff
status pipeline (DB → provider → SDK → frontend). The GitHub provider
now fetches the PR title alongside existing status fields.
The agents sidebar now displays PR-state-aware icons for chats that have
a linked pull request (when the chat is in waiting/completed state):
- **Open PR**: `GitPullRequestArrow` (green)
- **Draft PR**: `GitPullRequestDraft` (gray)
- **Merged PR**: `GitMerge` (purple)
- **Closed PR**: `GitPullRequestClosed` (red)
Running/pending/paused/error chats keep their existing activity icons
(spinner, pause, error triangle).
### Changes
**Database migration** (`000432`): Adds `pull_request_title TEXT` and
`pull_request_draft BOOLEAN` columns to `chat_diff_statuses`.
**Backend pipeline**:
- `gitprovider.PRStatus` gains a `Title` field
- GitHub provider decodes the `title` from the API response
- `gitsync` and `coderd/chats.go` pass title + draft through to the DB
upsert
- `codersdk.ChatDiffStatus` exposes both new fields in the API response
**Frontend** (`AgentsSidebar.tsx`): New `getPRIconConfig()` function
resolves the appropriate Lucide git icon based on `pull_request_state`
and `pull_request_draft`. Only applies when the chat is in a terminal
state (waiting/completed).
**Real-time sync**: No changes needed — the existing
`diff_status_change` pubsub event already propagates the full
`ChatDiffStatus` including the new fields.
- Adds `_API_BASE_URL` to `CODER_EXTERNAL_AUTH_CONFIG_`
- Extracts and refactors existing GitHub PR sync logic to new packages
`coderd/gitsync` and `coderd/externalauth/gitprovider`
- Associated wiring and tests
Created using Opus 4.6
## Problem
When multiple concurrent callers (e.g., parallel workspace builds) read
the same single-use OAuth2 refresh token from the database and race to
exchange it with the provider, the first caller succeeds but subsequent
callers get `bad_refresh_token`. The losing caller then **clears the
valid new token** from the database, permanently breaking the auth link
until the user manually re-authenticates.
This is reliably reproducible when launching multiple workspaces
simultaneously with GitHub App external auth and user-to-server token
expiration enabled.
## Solution
Two layers of protection:
### 1. Singleflight deduplication (`Config.RefreshToken` +
`ObtainOIDCAccessToken`)
Concurrent callers for the same user/provider share a single refresh
call via `golang.org/x/sync/singleflight`, keyed by `userID`. The
singleflight callback re-reads the link from the database to pick up any
token already refreshed by a prior in-flight call, avoiding redundant
IDP round-trips entirely.
### 2. Optimistic locking on `UpdateExternalAuthLinkRefreshToken`
The SQL `WHERE` clause now includes `AND oauth_refresh_token =
@old_oauth_refresh_token`, so if two replicas (HA) race past
singleflight, the loser's destructive UPDATE is a harmless no-op rather
than overwriting the winner's valid token.
## Changes
| File | Change |
|------|--------|
| `coderd/externalauth/externalauth.go` | Added `singleflight.Group` to
`Config`; split `RefreshToken` into public wrapper +
`refreshTokenInner`; pass `OldOauthRefreshToken` to DB update |
| `coderd/provisionerdserver/provisionerdserver.go` | Wrapped OIDC
refresh in `ObtainOIDCAccessToken` with package-level singleflight |
| `coderd/database/queries/externalauth.sql` | Added optimistic lock
(`WHERE ... AND oauth_refresh_token = @old_oauth_refresh_token`) |
| `coderd/database/queries.sql.go` | Regenerated |
| `coderd/database/querier.go` | Regenerated |
| `coderd/database/dbauthz/dbauthz_test.go` | Updated test params for
new field |
| `coderd/externalauth/externalauth_test.go` | Added
`ConcurrentRefreshDedup` test; updated existing tests for singleflight
DB re-read |
## Testing
- **New test `ConcurrentRefreshDedup`**: 5 goroutines call
`RefreshToken` concurrently, asserts IDP refresh called exactly once,
all callers get same token.
- All existing `TestRefreshToken/*` subtests updated and passing.
- `TestObtainOIDCAccessToken` passing.
- `dbauthz` tests passing.
Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
**Breaking Change:** Existing oauth apps might now use PKCE. If an
unknown IdP type was being used, and it does not support PKCE, it will
break.
To fix, set the PKCE methods on the external auth to `none`
```
export CODER_EXTERNAL_AUTH_1_PKCE_METHODS=none
```
Solves #15575
Adds OAuth access token revocation when unlinking external auth
provider. Due to revocation not being consistently implemented by
providers this is only best effort attempt. Unsuccessful revocation
won't influence link removal.
External auth refresh errors lose the original error thrown on the first
refresh. This PR saves that error to the database to be raised on
subsequent refresh attempts
- Update go.mod to use Go 1.24.1
- Update GitHub Actions setup-go action to use Go 1.24.1
- Fix linting issues with golangci-lint by:
- Updating to golangci-lint v1.57.1 (more compatible with Go 1.24.1)
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <claude@anthropic.com>
https://github.com/coder/coder/pull/15608 introduced a buggy behaviour
with dbcrypt enabled.
When clearing an oauth refresh token, we had been setting the value to
the empty string.
The database encryption package considers decrypting an empty string to
be an error, as an empty encrypted string value will still have a nonce
associated with it and thus not actually be empty when stored at rest.
Instead of 'deleting' the refresh token, 'update' it to be the empty
string.
This plays nicely with dbcrypt.
It also adds a 'utility test' in the dbcrypt package to help encrypt a
value. This was useful when manually fixing users affected by this bug
on our dogfood instance.
Once a token refresh fails, we remove the `oauth_refresh_token` from the
database. This will prevent the token from hitting the IDP for
subsequent refresh attempts.
Without this change, a bad script can cause a failing token to hit a
remote IDP repeatedly with each `git` operation. With this change, after
the first hit, subsequent hits will fail locally, and never contact the
IDP.
The solution in both cases is to authenticate the external auth link. So
the resolution is the same as before.
* chore: add github.com user id association
This will eventually be used to show an indicator in the UI
to star the repository if you've been using Coder for a while
and have not starred the repo.
If you have, we'll never show a thing!
* gen
* Fix model query
* Fix linting
* Ignore auditing github.com user id
* Add test
* Fix gh url var name
* Update migration
* Update coderd/database/dbauthz/dbauthz.go
Co-authored-by: Steven Masley <Emyrk@users.noreply.github.com>
* Fix updating to when the token changes
* Fix migration
---------
Co-authored-by: Steven Masley <Emyrk@users.noreply.github.com>
* chore: dynamically determine gitlab external auth defaults
Static defaults work for github cloud, but not self hosted.
Self hosted setups will now have sane defaults if omitted.
* chore: verify pass through external auth query params
Unit test added to verify behavior of query params set in the
auth url for external apps. This behavior is intended to specifically
support Auth0 audience query param.
Cian Johnston