fix: remove 403 from key failover and cooldown on 401 (#27419)

## Problem

When a key returned 401 or 403, the pool marked it permanently
unavailable for the lifetime of that in-memory pool. This is bad UX: a
transient auth failure or a briefly-misconfigured key could take a key
out of rotation until the operator either restarted Coder or
reconfigured the key (even re-saving the same working value).

## Changes

- **403 removed from key failover**: it's a per-request authorization
failure, not a key-level problem, so it's surfaced to the caller as-is
without marking the key or failing over.
- **401 now applies a temporary cooldown** (like 429) so the key
recovers on its own instead of staying blocked.
- When every key is in an auth-failure cooldown, the pool reports a
`502` with no `Retry-After`, but the keys still recover automatically
once the cooldown elapses.

Closes
https://linear.app/codercom/issue/AIGOV-421/ai-gateway-a-quarantined-centralized-key-never-recovers-without-a
Closes
https://linear.app/codercom/issue/AIGOV-533/403s-misclassifying-keys-as-permanently-down-in-ai-gateway

> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by
@ssncferreira
This commit is contained in:
Susana Ferreira
2026-07-27 12:06:01 +01:00
committed by GitHub
parent 6f2011af88
commit dba45cede7
18 changed files with 254 additions and 169 deletions
+9 -9
View File
@@ -322,18 +322,18 @@ a maximum of **5 keys**.
### Failover behavior
Every request starts with the first key in the list. If a key is rate-limited
or returns an authentication error, AI Gateway automatically retries the request
with the next available key.
> [!WARNING]
> A key that fails with an authentication error (`401 Unauthorized` or
> `403 Forbidden`) is permanently disabled and will not be used again until the
> server is restarted or the provider configuration is reloaded.
(`429 Too Many Requests`) or fails authentication (`401 Unauthorized`), AI
Gateway puts that key on a temporary cooldown and retries the request with the
next available key. Keys recover automatically when the cooldown elapses, so
failover stays transparent to end users. Any other response, including a
`403 Forbidden`, is returned to the caller unchanged.
If all keys in the pool are exhausted, AI Gateway returns:
- `429 Too Many Requests` when at least one key is rate-limited, with a `Retry-After` header set to the shortest cooldown across all keys.
- `502 Bad Gateway` when every key has failed permanently.
- `429 Too Many Requests` when at least one key is rate-limited, with a `Retry-After`
header set to the shortest cooldown across all keys.
- `502 Bad Gateway` when every key is in an authentication-failure cooldown.
The keys still recover automatically once their cooldowns elapse, so no `Retry-After` is sent.
## Bring Your Own Key