fix: remove 403 from key failover and cooldown on 401 (#27419)

## Problem

When a key returned 401 or 403, the pool marked it permanently
unavailable for the lifetime of that in-memory pool. This is bad UX: a
transient auth failure or a briefly-misconfigured key could take a key
out of rotation until the operator either restarted Coder or
reconfigured the key (even re-saving the same working value).

## Changes

- **403 removed from key failover**: it's a per-request authorization
failure, not a key-level problem, so it's surfaced to the caller as-is
without marking the key or failing over.
- **401 now applies a temporary cooldown** (like 429) so the key
recovers on its own instead of staying blocked.
- When every key is in an auth-failure cooldown, the pool reports a
`502` with no `Retry-After`, but the keys still recover automatically
once the cooldown elapses.

Closes
https://linear.app/codercom/issue/AIGOV-421/ai-gateway-a-quarantined-centralized-key-never-recovers-without-a
Closes
https://linear.app/codercom/issue/AIGOV-533/403s-misclassifying-keys-as-permanently-down-in-ai-gateway

> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by
@ssncferreira
This commit is contained in:
Susana Ferreira
2026-07-27 12:06:01 +01:00
committed by GitHub
parent 6f2011af88
commit dba45cede7
18 changed files with 254 additions and 169 deletions
+1 -1
View File
@@ -62,7 +62,7 @@ func categorizeInterceptionError(c errorCategorizer, err error) (recorder.ErrorT
switch keyPoolErr.Kind {
case keypool.ErrorKindRateLimited:
return recorder.ErrorTypeRateLimited, msg
case keypool.ErrorKindPermanent:
case keypool.ErrorKindPermanent, keypool.ErrorKindUnauthorized:
return recorder.ErrorTypeUnauthorized, msg
default:
return recorder.ErrorTypeUnknown, msg