mirror of
https://github.com/coder/coder.git
synced 2026-09-22 05:05:20 +08:00
Yevhenii Shcherbina
ab69fa2f0d
A Bedrock provider that assumes an IAM role kept failing with `AssumeRole` `AccessDenied` for several minutes after its target role's trust policy was changed, and only recovered on a gateway restart or a long wait. The request itself was correct: the AWS CLI, using the same identity and the same `ExternalId`/role/region, accepted the identical request immediately against the same endpoint. The difference is the connection. The Go SDK reuses a keep-alive connection for the STS client, so every `AssumeRole` rides one connection pinned to a single STS endpoint. After a trust-policy change, that connection kept returning `AccessDenied` for minutes while a fresh connection (the AWS CLI) accepted the identical request at once; it recovered only when the connection recycled or the process restarted. The exact STS-internal reason is unconfirmed (likely per-endpoint propagation of the change) — what is verified is that a fresh connection per call recovers promptly. Disable keep-alive on the STS client so each `AssumeRole` opens a fresh connection and a trust-policy update takes effect quickly. `AssumeRole` runs at most once per credential-cache lifetime, so keep-alive bought nothing here. The change is scoped to the STS client only; Bedrock model requests are signed by a separate client and keep their connection pooling. ## What the data proves | | CLI | Gateway | |--------------------|-------------------------------------------|------------------------------------------| | Identity / key | `bedrock-base-user-useless` / `AKIA…44NL` | same | | STS endpoint | `sts.us-east-2.amazonaws.com` | same | | Request params | `ExternalId=QL53…`, role, session, 900 | same | | Recovery after fix | 7 seconds (21:27:54) | ~4.5 minutes (21:32:17) | | Re-hitting AWS? | new call each time | yes — 77 fresh `AssumeRole`s, all denied | Same identity, params, and endpoint, concurrent — yet the gateway was denied for ~4.5 minutes while the CLI recovered in 7 seconds, and the gateway made a fresh `AssumeRole` on every request (so it was not caching a failure). The only difference was connection reuse. After disabling keep-alive, the same break/fix experiment brought gateway recovery down from ~4.5 minutes to ~7 seconds, in lockstep with the AWS CLI.