mirror of
https://github.com/coder/coder.git
synced 2026-09-24 15:04:27 +08:00
feat(chatd): add LLM stream retry with exponential backoff (#22418)
## Summary Adds automatic retry with exponential backoff for transient LLM errors during chat streaming and title generation. Inspired by [coder/mux](https://github.com/coder/mux)'s retry mechanism. ## Key Behaviors - **Infinite retries** with exponential backoff: 1s → 2s → 4s → ... → 60s cap - **Deterministic delays** (no jitter) - **Error classification**: retryable (429, 5xx, overloaded, rate limit, network errors) vs non-retryable (auth, quota, context exceeded, model not found, canceled) - **Retry status published to SSE stream** so frontend can show "Retrying in Xs..." UI - **Title generation** retries silently (best-effort, nil onRetry callback) ## New Package: `coderd/chatd/chatretry/` | File | Purpose | |------|---------| | `classify.go` | `IsRetryable(err)` and `StatusCodeRetryable(code)` | | `backoff.go` | `Delay(attempt)` — exponential doubling with 60s cap | | `retry.go` | `Retry(ctx, fn, onRetry)` — infinite loop with context-aware timer | ## Test Helpers: `coderd/chatd/chattest/errors.go` Anthropic and OpenAI error response builders for use in chattest providers: - `AnthropicErrorResponse()`, `AnthropicOverloadedResponse()`, `AnthropicRateLimitResponse()` - `OpenAIErrorResponse()`, `OpenAIRateLimitResponse()`, `OpenAIServerErrorResponse()` ## SDK Changes: `codersdk/chats.go` - New `ChatStreamEventType: "retry"` - New `ChatStreamRetry` struct with `Attempt`, `DelayMs`, `Error`, `RetryingAt` fields - TypeScript types auto-generated ## Changed Files - `coderd/chatd/chatloop/chatloop.go` — wraps `agent.Stream()` in `chatretry.Retry()` - `coderd/chatd/chatd.go` — publishes retry events to SSE stream with logging - `coderd/chatd/title.go` — wraps `model.Generate()` in silent retry - `coderd/chatd/chattest/anthropic.go` / `openai.go` — error injection support ## Tests 42 tests covering classification (33), backoff (9), and retry scenarios (8).
This commit is contained in:
@@ -444,6 +444,7 @@ const (
|
||||
ChatStreamEventTypeStatus ChatStreamEventType = "status"
|
||||
ChatStreamEventTypeError ChatStreamEventType = "error"
|
||||
ChatStreamEventTypeQueueUpdate ChatStreamEventType = "queue_update"
|
||||
ChatStreamEventTypeRetry ChatStreamEventType = "retry"
|
||||
)
|
||||
|
||||
// ChatQueuedMessage represents a queued message waiting to be processed.
|
||||
@@ -470,6 +471,19 @@ type ChatStreamError struct {
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// ChatStreamRetry represents an auto-retry status event in the stream.
|
||||
// Published when the server automatically retries a failed LLM call.
|
||||
type ChatStreamRetry struct {
|
||||
// Attempt is the 1-indexed retry attempt number.
|
||||
Attempt int `json:"attempt"`
|
||||
// DelayMs is the backoff delay in milliseconds before the retry.
|
||||
DelayMs int64 `json:"delay_ms"`
|
||||
// Error is the error message from the failed attempt.
|
||||
Error string `json:"error"`
|
||||
// RetryingAt is the timestamp when the retry will be attempted.
|
||||
RetryingAt time.Time `json:"retrying_at" format:"date-time"`
|
||||
}
|
||||
|
||||
// ChatStreamEvent represents a real-time update for chat streaming.
|
||||
type ChatStreamEvent struct {
|
||||
Type ChatStreamEventType `json:"type"`
|
||||
@@ -478,6 +492,7 @@ type ChatStreamEvent struct {
|
||||
MessagePart *ChatStreamMessagePart `json:"message_part,omitempty"`
|
||||
Status *ChatStreamStatus `json:"status,omitempty"`
|
||||
Error *ChatStreamError `json:"error,omitempty"`
|
||||
Retry *ChatStreamRetry `json:"retry,omitempty"`
|
||||
QueuedMessages []ChatQueuedMessage `json:"queued_messages,omitempty"`
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user