fix: replay stateless OpenAI reasoning across chat turns (#30298)

OpenAI Responses chats default to `store=false`, and chatd only replayed
reasoning as an `item_reference`, which needs provider-side storage.
Reasoning was therefore dropped between tool steps and turns, so
reasoning models lost their chain of thought mid-task.

This pins coder/fantasy `coder_2_33` at 6ba8681, which includes fantasy
#63 (and #64, #65, and #66). #63 captures each completed reasoning item
(ID, `encrypted_content`, summary) and replays it in full regardless of
`store`. In chatd:

- Every Responses call requests `reasoning.encrypted_content`. The
`store` default and configured includes are unchanged, and Chat
Completions requests are untouched.
- Prompt rebuilds keep reasoning parts without visible text when they
carry finalized encrypted state, so reasoning survives durable history
and coderd restarts.
- The provider-switch sanitizer also strips OpenAI reasoning from
history produced by a different or unknown provider instance, since
reasoning IDs and encrypted content only resolve where they were issued.
- A compaction model override on a different provider instance, or with
a different model, gets a compaction prompt without the chat's OpenAI
reasoning.
- Each saved reasoning part records the provider instance, model, and
credential (the deployment's or the user's personal API key) that
produced it, and the sanitizer keeps it only when all three match the
current turn, because another model, or the organization behind another
credential, can reject the encrypted content. Switching models, adding
or removing a personal key, or editing a config's model or provider
drops the earlier reasoning; switching back restores it. The record is
never returned by the API. Reasoning saved before this change has no
record and follows its model config.
- If OpenAI rejects replayed reasoning it cannot decrypt
(`invalid_encrypted_content`), for example after the deployment's key is
replaced with one from another organization, the request is retried once
without reasoning, and the reasoning that step produces is saved with
the provider's next rejection epoch. Epochs are kept separately for each
provider, model, and credential. Later steps and turns leave out
reasoning from earlier epochs, on any replica and after restarts, so
only the first request after the change is rejected and newer reasoning
keeps replaying. A retried step that returns no reasoning saves its
epoch on its first part. The advisor's prompt never includes OpenAI
reasoning, as on `main`.

## Testing

Rebased onto main at 98332c739b as 5735acb36ed (same tree as merging
the pre-rebase head a378071862a with main; older SHAs below refer to the
pre-rebase history). 3c17d807a33 then adds the model to the reasoning
record, 21156b2b32e pins Fantasy d55a867, which treats a response that
echoes `store: false` as unstored, bf795daa923 adds the retry and the
advisor change, 7eb8b1d029a moves the pin to 88cffb5 (`coder_2_33`
merged into #63; its OpenAI provider code is identical to d55a867), and
f4f40734a41 pins `coder_2_33` at 6ba8681 once #63 merged. That adds
fantasy #66, which replays the Responses message phase on assistant
text. abccede81f7 makes the retry remember rejected reasoning in the
process. The branch was then rebased onto main at abc2faed60, so every
SHA above refers to the pre-rebase history. 5fb3ce1689 replaces that
memory with the credential and a rejection epoch saved on each reasoning
part, and 62b5cb532a keeps epochs per provider, model, and credential
and saves the epoch when the retried step returns no reasoning.
1057beb327 hides the four internal fields from the Swagger spec and API
reference, and 29a705e099 merges main at d68b2c8b25. Its only
conflicts were the fantasy pin (kept at 6ba8681, whose `coder_2_33`
history includes main's 3510552) and test imports.

- Tests in `chatprompt`, `chatprovider`, `chatsanitize`, the
provider-switch sanitizer, and `integration_responses_test.go` cover
tool-step and durable replay, legacy and incomplete metadata, provider
switches, and a web search chat switching from `store=false` to
`store=true`. Integration tests also cover the retry, a later tool step
after it, the next turn on a fresh server against the same database
(with and without reasoning in the retried step), a personal key added
mid-chat, and the advisor prompt. Red-green verified, including a config
whose model is edited in place. The compaction override's model check
has unit tests and UAT coverage but no integration test. `go vet`,
targeted race tests, and the pre-commit hook (gen, fmt, lint, build)
pass. Local pre-push hooks were skipped; CI covers them.
- Remote UAT on dogfood with real gpt-5-mini and gpt-5.4 behind a
logging proxy that checks every request body. Rounds 1 and 2 failed on
dangling item references (both fixed). Rounds 3 to 5 passed
(ee3c0bf3d60, 0580870b5ff, a378071862a): tool-step and durable replay,
restarts, interrupts, provider switches and config repoints, web search
history, and compaction overrides. Both test providers shared one
upstream account, so cross-provider checks cover request shape, not a
real rejection. Round 6 at 21156b2b32e (model record) with gpt-5-mini
and gpt-5-nano: PASS. 151 Responses requests, all 200; no request
carried reasoning from a different model except an older, unrecorded
part that follows its config by design. Same-model config switches,
switching back, edits to a config's model, a mid-turn switch, restarts,
and a same-provider compaction override with a different model all
behaved as described. Round 7 at bf795daa923 (retry and advisor): PASS.
Through AI Gateway, a corrupted saved blob is rejected with HTTP 400
`invalid_encrypted_content`. All 14 rejected requests were retried once
without reasoning and returned 200, a different 400 was not retried, and
advisor requests on the chat model, a same-provider override, and
another provider carried no reasoning.
- f4f40734a41 changes only the pin. Its new OpenAI code is fantasy #66,
which UAT rounds 1 to 7 did not run. A live check of fantasy 6ba8681
through AI Gateway, two turns with tool calls on gpt-5.5 and gpt-5-mini
with `store` false and true: all 18 requests returned 200, and the
gpt-5.5 requests replayed encrypted reasoning followed by assistant
messages with the `commentary` and `final_answer` phases. `go test
./coderd/x/chatd/... ./codersdk/` passes (21 packages) on the same
fantasy tree.
- UAT round 8 at abccede81f7 (remembered rejection, and fantasy #66
through chatd) with gpt-5.5, gpt-5-mini, and gpt-5-nano: PASS. 110
requests; the 6 rejections were 5 corrupted saved blobs, each retried
once without reasoning and answered 200, and one unrelated `temperature`
error that was not retried. After the first rejection, later steps and
turns in that chat were never rejected again and replayed newer
reasoning. A coderd restart cost one more rejection, which also dropped
reasoning saved after the first one, and a second chat was unaffected.
gpt-5.5 requests replayed assistant messages with the phases the model
returned, with `store` false and true.
- UAT round 9 at 62b5cb532a (saved rejection epoch, personal key
switch) with gpt-5.5, gpt-5-mini, and gpt-5-nano: PASS. 158 Responses
and 7 Chat Completions requests; the 6 rejections were corrupted saved
blobs, each retried once without reasoning and answered 200, and an
unrelated `temperature` error was not retried. After a coderd restart,
and on a second coderd against the same database, the chat was not
rejected again and kept replaying the reasoning saved after the
rejection. Adding a personal key left out the deployment key's reasoning
before sending, and removing it brought that reasoning back. A retried
step without reasoning, two chats rejected at once, compaction, a model
switch after a rejection, and a switch to Chat Completions and back all
behaved as described.

## Review record

- Independent full-branch test audit at bbe1a1b27a: PASS.
- Codex through a378071862: compaction overrides kept foreign reasoning
and configs could be repointed to another provider (both fixed); a
request for documentation TODOs and two behaviors that already exist on
`main` (declined).
- Codex on 5735acb36e: two findings. Reasoning replayed after a
same-provider model switch: fixed in 3c17d807a33. A provider's base URL
edited in place keeps its reasoning: covered by the retry in
bf795daa923.
- Codex on 21156b2b32, bf795daa92, and 7eb8b1d029: no major issues.
Codex on f4f40734a4: rejected reasoning stayed in the chat's history, so
each later request in that chat was rejected and retried again without
any reasoning. Fixed in abccede81f7. Codex on abccede81f: that memory
was lost on a restart or another replica, where the next rejection also
dropped newer reasoning, and it grew without bound. Fixed in
5fb3ce1689. Codex on 5fb3ce1689: a rejection advanced the epoch for
every model and credential on the provider, and a retried step without
reasoning lost the new epoch. Both fixed in 62b5cb532a. Codex on
62b5cb532a: keep reasoning out of a config switched from Responses to
Chat Completions (declined: the Chat Completions converter already skips
reasoning parts, confirmed with a probe test). Codex on 29a705e099: no
major issues.
- ethanndickson: retry without reasoning on `invalid_encrypted_content`,
and keep OpenAI reasoning out of the advisor prompt. Both fixed in
bf795daa923. Then: in-process memory is not enough because dogfood
restarts on every merge and runs several replicas. Fixed in 5fb3ce1689
with his proposed rejection epoch in chat history, plus the credential
record, and tested with the next turn on a fresh server. Then: hide the
internal fields from Swagger, fixed in 1057beb327. Approved on
62b5cb532a.
- coder/fantasy#63 review: honoring an echoed `store: false` is fixed in
d55a867; replaying web searches that had no preceding reasoning is a
pre-existing gap, tracked in #30528.
- CI on 5735acb36ed: green after rerunning two failures outside this
change (a `cli` `TestSSH` data race, and a `test-go-pg` job timeout that
also fails on `main`). CI on 21156b2b32e: all checks pass. CI on
bf795daa923: all checks pass. CI on 7eb8b1d029a: all checks pass. CI on
f4f40734a41: all checks pass. CI on abccede81f7 did not run, because the
branch conflicted with `main` (#30492 changed the provider-switch
sanitizer to return database errors). CI on 62b5cb532a: all checks
pass. CI on 29a705e099: all checks pass.

> Xum, an AI coding agent, implemented, tested, and opened this PR on
behalf of @ibetitsmike.
This commit is contained in:
Michael Suchacz
2026-10-13 13:49:12 +02:00
committed by GitHub
parent 71e8e526fa
commit b2c8798e0e
21 changed files with 1537 additions and 171 deletions
+6 -1
View File
@@ -1495,7 +1495,9 @@ func partsToMessageParts(
})
case codersdk.ChatMessagePartTypeReasoning:
opts := providerMetadataToOptions(logger, part.ProviderMetadata)
if strings.TrimSpace(part.Text) == "" && !chatsanitize.HasAnthropicSignedReasoningOptions(opts) {
if strings.TrimSpace(part.Text) == "" &&
!chatsanitize.HasAnthropicSignedReasoningOptions(opts) &&
!chatsanitize.HasOpenAIReasoningOptions(opts) {
continue
}
result = append(result, fantasy.ReasoningPart{
@@ -1913,6 +1915,9 @@ var partNulFields = []partNulField{
{name: "ToolCallID", policy: nulReject, str: func(p *codersdk.ChatMessagePart) *string { return &p.ToolCallID }},
{name: "ToolName", policy: nulReject, str: func(p *codersdk.ChatMessagePart) *string { return &p.ToolName }},
{name: "ParsedCommands", policy: nulReject, grid: func(p *codersdk.ChatMessagePart) [][]string { return p.ParsedCommands }},
{name: "ProviderIdentity", policy: nulReject, str: func(p *codersdk.ChatMessagePart) *string { return &p.ProviderIdentity }},
{name: "ProviderModel", policy: nulReject, str: func(p *codersdk.ChatMessagePart) *string { return &p.ProviderModel }},
{name: "ProviderCredential", policy: nulReject, str: func(p *codersdk.ChatMessagePart) *string { return &p.ProviderCredential }},
{name: "SourceID", policy: nulReject, str: func(p *codersdk.ChatMessagePart) *string { return &p.SourceID }},
{name: "URL", policy: nulReject, str: func(p *codersdk.ChatMessagePart) *string { return &p.URL }},
{name: "MediaType", policy: nulReject, str: func(p *codersdk.ChatMessagePart) *string { return &p.MediaType }},
+96 -29
View File
@@ -11,6 +11,7 @@ import (
"charm.land/fantasy"
fantasyanthropic "charm.land/fantasy/providers/anthropic"
fantasygoogle "charm.land/fantasy/providers/google"
fantasyopenai "charm.land/fantasy/providers/openai"
"github.com/google/uuid"
"github.com/sqlc-dev/pqtype"
"github.com/stretchr/testify/assert"
@@ -39,42 +40,108 @@ func testMsg(role codersdk.ChatMessageRole, raw pqtype.NullRawMessage) database.
}
}
func TestConvertMessagesWithFilesPreservesEmptyRedactedReasoning(t *testing.T) {
func TestConvertMessagesWithFilesPreservesEmptyReplayableReasoning(t *testing.T) {
t.Parallel()
metadata, err := json.Marshal(fantasy.ProviderMetadata{
fantasyanthropic.Name: &fantasyanthropic.ReasoningOptionMetadata{
RedactedData: "redacted-payload",
},
})
require.NoError(t, err)
content, err := chatprompt.MarshalParts([]codersdk.ChatMessagePart{
encrypted := "encrypted-blob"
tests := []struct {
name string
metadata fantasy.ProviderMetadata
kept bool
}{
{
Type: codersdk.ChatMessagePartTypeReasoning,
ProviderMetadata: metadata,
name: "AnthropicRedacted",
metadata: fantasy.ProviderMetadata{
fantasyanthropic.Name: &fantasyanthropic.ReasoningOptionMetadata{
RedactedData: "redacted-payload",
},
},
kept: true,
},
codersdk.ChatMessageText("done"),
})
require.NoError(t, err)
prompt, err := chatprompt.ConvertMessagesWithFiles(context.Background(), []database.ChatMessage{
{
Role: database.ChatMessageRoleAssistant,
Visibility: database.ChatMessageVisibilityBoth,
Content: content,
ContentVersion: chatprompt.CurrentContentVersion,
name: "OpenAIFinalized",
metadata: fantasy.ProviderMetadata{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{
ItemID: "rs_1",
EncryptedContent: &encrypted,
Summary: []string{},
Finalized: true,
},
},
kept: true,
},
}, nil, slogtest.Make(t, nil), nil, uuid.NullUUID{})
require.NoError(t, err)
require.Len(t, prompt, 1)
require.Len(t, prompt[0].Content, 2)
{
name: "OpenAIUnfinalized",
metadata: fantasy.ProviderMetadata{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{
ItemID: "rs_1",
EncryptedContent: &encrypted,
},
},
},
{
name: "OpenAIFinalizedWithoutEncryptedContent",
metadata: fantasy.ProviderMetadata{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{
ItemID: "rs_1",
Finalized: true,
},
},
},
{
name: "OpenAIFinalizedWithEmptyEncryptedContent",
metadata: fantasy.ProviderMetadata{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{
ItemID: "rs_1",
EncryptedContent: new(""),
Finalized: true,
},
},
},
{
name: "OpenAIWithoutItemID",
metadata: fantasy.ProviderMetadata{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{
EncryptedContent: &encrypted,
Finalized: true,
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
reasoning, ok := fantasy.AsMessagePart[fantasy.ReasoningPart](prompt[0].Content[0])
require.True(t, ok)
require.Empty(t, reasoning.Text)
reasoningMetadata := fantasyanthropic.GetReasoningMetadata(reasoning.ProviderOptions)
require.NotNil(t, reasoningMetadata)
require.Equal(t, "redacted-payload", reasoningMetadata.RedactedData)
content, err := chatprompt.MarshalParts([]codersdk.ChatMessagePart{
chatprompt.PartFromContent(fantasy.ReasoningContent{ProviderMetadata: tt.metadata}),
codersdk.ChatMessageText("done"),
})
require.NoError(t, err)
prompt, err := chatprompt.ConvertMessagesWithFiles(context.Background(), []database.ChatMessage{
{
Role: database.ChatMessageRoleAssistant,
Visibility: database.ChatMessageVisibilityBoth,
Content: content,
ContentVersion: chatprompt.CurrentContentVersion,
},
}, nil, slogtest.Make(t, nil), nil, uuid.NullUUID{})
require.NoError(t, err)
require.Len(t, prompt, 1)
if !tt.kept {
require.Len(t, prompt[0].Content, 1)
_, ok := fantasy.AsMessagePart[fantasy.TextPart](prompt[0].Content[0])
require.True(t, ok)
return
}
require.Len(t, prompt[0].Content, 2)
reasoning, ok := fantasy.AsMessagePart[fantasy.ReasoningPart](prompt[0].Content[0])
require.True(t, ok)
require.Empty(t, reasoning.Text)
require.Equal(t, fantasy.ProviderOptions(tt.metadata), reasoning.ProviderOptions)
})
}
}
func TestConvertMessagesWithFilesRoundTripsAnthropicInterleavedWebSearch(t *testing.T) {
+21 -1
View File
@@ -960,7 +960,27 @@ func ProviderOptionsForCall(
) fantasy.ProviderOptions {
options := providerOptionsFromChatModelConfig(model, config.ProviderOptions)
effort := ResolveReasoningEffort(requestedEffort, config.ReasoningEffort)
return applyReasoningEffort(model, options, effort)
options = applyReasoningEffort(model, options, effort)
return ensureOpenAIResponsesIncludes(model, options)
}
// ensureOpenAIResponsesIncludes requests encrypted reasoning on every
// Responses call, so reasoning can be replayed without provider-side storage.
// An options block created here leaves Store unset, keeping the provider's
// default.
func ensureOpenAIResponsesIncludes(
model Model,
options fantasy.ProviderOptions,
) fantasy.ProviderOptions {
if !model.Valid() || !model.transport.UsesResponses() {
return options
}
if options == nil {
options = fantasy.ProviderOptions{}
}
responsesOptions := ensureProviderOptions[fantasyopenai.ResponsesProviderOptions](options, fantasyopenai.Name)
responsesOptions.Include = chatopenai.EnsureResponseIncludes(responsesOptions.Include)
return options
}
func providerOptionsFromChatModelConfig(
@@ -312,3 +312,93 @@ func TestModelFromConfig_OpenAIReasoningModelOverride(t *testing.T) {
})
}
}
// Encrypted reasoning is the only replayable reasoning state when responses
// are not stored, so every Responses call must request it without changing
// the configured or default store setting.
func TestProviderOptionsForCall_OpenAIResponsesEncryptedReasoning(t *testing.T) {
t.Parallel()
const encryptedReasoning = string(fantasyopenai.IncludeReasoningEncryptedContent)
const fileSearchResults = string(fantasyopenai.IncludeFileSearchCallResults)
effort := &codersdk.ChatModelReasoningEffortConfig{
Default: new(codersdk.ChatModelReasoningEffortHigh),
Max: new(codersdk.ChatModelReasoningEffortHigh),
}
for _, tc := range []struct {
name string
useResponses bool
config codersdk.ChatModelCallConfig
wantInclude []any
wantStore any
}{
{
name: "Default",
useResponses: true,
wantInclude: []any{encryptedReasoning},
wantStore: false,
},
{
name: "EffortOnly",
useResponses: true,
config: codersdk.ChatModelCallConfig{ReasoningEffort: effort},
wantInclude: []any{encryptedReasoning},
wantStore: false,
},
{
name: "ConfiguredStoreAndInclude",
useResponses: true,
config: codersdk.ChatModelCallConfig{
ReasoningEffort: effort,
ProviderOptions: &codersdk.ChatModelProviderOptions{
OpenAI: &codersdk.ChatModelOpenAIProviderOptions{
Store: new(true),
Include: []string{fileSearchResults},
},
},
},
wantInclude: []any{fileSearchResults, encryptedReasoning},
wantStore: true,
},
{
name: "ChatCompletions",
config: codersdk.ChatModelCallConfig{ReasoningEffort: effort},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
bodies := make(chan []byte, 1)
serverURL := chattest.NewOpenAI(t, func(req *chattest.OpenAIRequest) chattest.OpenAIResponse {
bodies <- req.RawBody
return chattest.OpenAINonStreamingResponse("ok")
})
model, err := chatprovider.ModelFromConfig(
fantasyopenai.Name, "gpt-5",
chatprovider.ProviderAPIKeys{
ByProvider: map[string]string{fantasyopenai.Name: "test-key"},
BaseURLByProvider: map[string]string{fantasyopenai.Name: serverURL},
},
chatprovider.UserAgent(), nil, nil,
&codersdk.ChatModelCallConfig{OpenAIConfig: &codersdk.ChatModelOpenAIConfig{UseResponsesAPI: &tc.useResponses}},
)
require.NoError(t, err)
_, err = model.LanguageModel().Generate(t.Context(), fantasy.Call{
Prompt: []fantasy.Message{{Role: fantasy.MessageRoleUser, Content: []fantasy.MessagePart{fantasy.TextPart{Text: "hello"}}}},
ProviderOptions: chatprovider.ProviderOptionsForCall(model, tc.config, nil),
})
require.NoError(t, err)
var body map[string]any
require.NoError(t, json.Unmarshal(<-bodies, &body))
if !tc.useResponses {
require.NotContains(t, body, "include")
return
}
require.Equal(t, tc.wantInclude, body["include"])
require.Equal(t, tc.wantStore, body["store"])
})
}
}
+22
View File
@@ -0,0 +1,22 @@
package chatsanitize
import (
"charm.land/fantasy"
fantasyopenai "charm.land/fantasy/providers/openai"
)
// HasOpenAIReasoningOptions reports whether options carry finalized encrypted
// reasoning that can be replayed without server-side storage.
func HasOpenAIReasoningOptions(options fantasy.ProviderOptions) bool {
metadata := fantasyopenai.GetReasoningMetadata(options)
return metadata != nil && metadata.ItemID != "" && metadata.Finalized &&
metadata.EncryptedContent != nil && *metadata.EncryptedContent != ""
}
// HasOpenAIReasoningState reports whether options carry an OpenAI reasoning
// item. Its item ID and encrypted content only resolve on the provider
// instance that issued them.
func HasOpenAIReasoningState(options fantasy.ProviderOptions) bool {
metadata := fantasyopenai.GetReasoningMetadata(options)
return metadata != nil && metadata.ItemID != ""
}
+8 -5
View File
@@ -11,6 +11,7 @@ import (
type ErrorResponse struct {
StatusCode int
Type string
Code string
Message string
}
@@ -19,12 +20,14 @@ type ErrorResponse struct {
func writeErrorResponse(t testing.TB, w http.ResponseWriter, errResp *ErrorResponse) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(errResp.StatusCode)
body := map[string]interface{}{
"error": map[string]interface{}{
"type": errResp.Type,
"message": errResp.Message,
},
errorBody := map[string]interface{}{
"type": errResp.Type,
"message": errResp.Message,
}
if errResp.Code != "" {
errorBody["code"] = errResp.Code
}
body := map[string]interface{}{"error": errorBody}
if err := json.NewEncoder(w).Encode(body); err != nil {
t.Errorf("writeErrorResponse: failed to encode error response: %v", err)
}
+42 -2
View File
@@ -21,7 +21,8 @@ func sameCompactionProviderIdentity(chatConfig, overrideConfig database.ChatMode
}
// sanitizeCompactionPrompt adapts a prompt built for the chat model to a
// differing compaction model. The input messages are never mutated; the
// differing compaction model. reasoningModel is the model whose OpenAI
// reasoning the prompt may carry. The input messages are never mutated; the
// assistant generation keeps using the original prompt.
func sanitizeCompactionPrompt(
ctx context.Context,
@@ -31,11 +32,16 @@ func sanitizeCompactionPrompt(
configuredProvider string,
chatConfig database.ChatModelConfig,
overrideConfig database.ChatModelConfig,
reasoningModel string,
) []fantasy.Message {
messages := prompt
if !sameCompactionProviderIdentity(chatConfig, overrideConfig) {
sameProvider := sameCompactionProviderIdentity(chatConfig, overrideConfig)
if !sameProvider {
messages = flattenProviderExecutedToolParts(ctx, logger, messages)
}
if !sameProvider || compactionModel.ModelID() != reasoningModel {
messages = dropOpenAIReasoningParts(ctx, logger, messages)
}
messages = replaceUnsupportedFileParts(ctx, logger, messages, compactionModel.AcceptsFilePartMediaType)
messages = replaceUnsupportedToolMedia(ctx, logger, messages, compactionModel, configuredProvider)
sanitized, stats := chatsanitize.SanitizeAnthropicProviderToolHistory(
@@ -121,6 +127,40 @@ func flattenProviderExecutedToolParts(
return out
}
// dropOpenAIReasoningParts removes OpenAI reasoning parts from a copy of
// messages, since another provider cannot resolve their item IDs and another
// model or organization can reject their encrypted content. Messages emptied
// by the drop are removed.
func dropOpenAIReasoningParts(
ctx context.Context,
logger slog.Logger,
messages []fantasy.Message,
) []fantasy.Message {
dropped := 0
out := make([]fantasy.Message, 0, len(messages))
for _, msg := range messages {
parts := make([]fantasy.MessagePart, 0, len(msg.Content))
for _, part := range msg.Content {
if reasoning, ok := part.(fantasy.ReasoningPart); ok && chatsanitize.HasOpenAIReasoningState(reasoning.ProviderOptions) {
dropped++
continue
}
parts = append(parts, part)
}
if len(parts) == 0 && len(msg.Content) > 0 {
continue
}
msg.Content = parts
out = append(out, msg)
}
if dropped > 0 {
logger.Debug(ctx, "dropped OpenAI reasoning from prompt",
slog.F("dropped_parts", dropped),
)
}
return out
}
// stringifyToolResultOutput renders a tool result as prompt text. Media
// payloads are summarized so base64 data does not enter the prompt.
func stringifyToolResultOutput(output fantasy.ToolResultOutputContent) string {
@@ -4,6 +4,7 @@ import (
"testing"
"charm.land/fantasy"
fantasyopenai "charm.land/fantasy/providers/openai"
"github.com/google/uuid"
"github.com/stretchr/testify/require"
@@ -72,7 +73,7 @@ func TestSanitizeCompactionPrompt_FlattensForeignProviderExecutedToolParts(t *te
}
compactionModel := chatprovider.NewModel(&chattest.FakeModel{ProviderName: "openai", ModelName: "gpt-4.1-mini"}, nil)
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(uuid.New()), configWithProvider(uuid.New()))
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(uuid.New()), configWithProvider(uuid.New()), compactionModel.ModelID())
require.Len(t, sanitized, 3)
// Provider-executed parts are flattened to text so the summary keeps
@@ -94,6 +95,61 @@ func TestSanitizeCompactionPrompt_FlattensForeignProviderExecutedToolParts(t *te
}, prompt[1].Content[1])
}
func TestSanitizeCompactionPrompt_OpenAIReasoning(t *testing.T) {
t.Parallel()
encrypted := "encrypted-blob"
finalized := fantasy.ReasoningPart{ProviderOptions: fantasy.ProviderOptions{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{ItemID: "rs_final", EncryptedContent: &encrypted, Finalized: true},
}}
legacy := fantasy.ReasoningPart{Text: "summary", ProviderOptions: fantasy.ProviderOptions{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{ItemID: "rs_legacy", Summary: []string{"summary"}},
}}
prompt := []fantasy.Message{
{Role: fantasy.MessageRoleUser, Content: []fantasy.MessagePart{fantasy.TextPart{Text: "hi"}}},
{Role: fantasy.MessageRoleAssistant, Content: []fantasy.MessagePart{finalized, fantasy.TextPart{Text: "answer"}}},
{Role: fantasy.MessageRoleAssistant, Content: []fantasy.MessagePart{legacy}},
}
compactionModel := chatprovider.NewModel(&chattest.FakeModel{ProviderName: "openai", ModelName: "gpt-5-mini"}, nil)
t.Run("ForeignProviderDropsReasoning", func(t *testing.T) {
t.Parallel()
ctx := testutil.Context(t, testutil.WaitShort)
logger := slogtest.Make(t, &slogtest.Options{IgnoreErrors: true})
// Reasoning IDs and encrypted content only resolve on the provider
// instance that issued them.
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(uuid.New()), configWithProvider(uuid.New()), compactionModel.ModelID())
require.Len(t, sanitized, 2)
require.Equal(t, []fantasy.MessagePart{fantasy.TextPart{Text: "answer"}}, sanitized[1].Content)
require.Equal(t, finalized, prompt[1].Content[0])
})
t.Run("SameModelKeepsReasoning", func(t *testing.T) {
t.Parallel()
ctx := testutil.Context(t, testutil.WaitShort)
logger := slogtest.Make(t, &slogtest.Options{IgnoreErrors: true})
sharedProviderID := uuid.New()
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(sharedProviderID), configWithProvider(sharedProviderID), compactionModel.ModelID())
require.Equal(t, prompt, sanitized)
})
t.Run("SameProviderDifferentModelDropsReasoning", func(t *testing.T) {
t.Parallel()
ctx := testutil.Context(t, testutil.WaitShort)
logger := slogtest.Make(t, &slogtest.Options{IgnoreErrors: true})
sharedProviderID := uuid.New()
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(sharedProviderID), configWithProvider(sharedProviderID), "gpt-5.4")
require.Len(t, sanitized, 2)
require.Equal(t, []fantasy.MessagePart{fantasy.TextPart{Text: "answer"}}, sanitized[1].Content)
})
}
func TestSanitizeCompactionPrompt_DropsNonAssistantProviderExecutedParts(t *testing.T) {
t.Parallel()
@@ -121,7 +177,7 @@ func TestSanitizeCompactionPrompt_DropsNonAssistantProviderExecutedParts(t *test
}
compactionModel := chatprovider.NewModel(&chattest.FakeModel{ProviderName: "openai", ModelName: "gpt-4.1-mini"}, nil)
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(uuid.New()), configWithProvider(uuid.New()))
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(uuid.New()), configWithProvider(uuid.New()), compactionModel.ModelID())
require.Len(t, sanitized, 1)
require.Equal(t, fantasy.MessageRoleUser, sanitized[0].Role)
@@ -151,7 +207,7 @@ func TestSanitizeCompactionPrompt_ReplacesUnsupportedFileParts(t *testing.T) {
// placeholder while the prompt stays otherwise intact.
compactionModel := chatprovider.NewModel(&chattest.FakeModel{ProviderName: "mistral", ModelName: "mistral-large"}, nil)
sharedProviderID := uuid.New()
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(sharedProviderID), configWithProvider(sharedProviderID))
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(sharedProviderID), configWithProvider(sharedProviderID), compactionModel.ModelID())
require.Len(t, sanitized, 1)
require.Len(t, sanitized[0].Content, 2)
@@ -182,7 +238,7 @@ func TestSanitizeCompactionPrompt_ReplacesUnsupportedToolMedia(t *testing.T) {
// not, and the filter must run even when both models share a provider.
compactionModel := chatprovider.NewModel(&chattest.FakeModel{ProviderName: "openai-compat", ModelName: "model"}, nil)
sharedProviderID := uuid.New()
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, "google", configWithProvider(sharedProviderID), configWithProvider(sharedProviderID))
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, "google", configWithProvider(sharedProviderID), configWithProvider(sharedProviderID), compactionModel.ModelID())
require.Len(t, sanitized, 1)
result, ok := sanitized[0].Content[0].(fantasy.ToolResultPart)
@@ -218,7 +274,7 @@ func TestSanitizeCompactionPrompt_SameProviderKeepsProviderExecutedParts(t *test
compactionModel := chatprovider.NewModel(&chattest.FakeModel{ProviderName: "openai", ModelName: "gpt-4.1-mini"}, nil)
sharedProviderID := uuid.New()
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(sharedProviderID), configWithProvider(sharedProviderID))
sanitized := sanitizeCompactionPrompt(ctx, logger, prompt, compactionModel, compactionModel.Provider(), configWithProvider(sharedProviderID), configWithProvider(sharedProviderID), compactionModel.ModelID())
require.Len(t, sanitized, 1)
require.Len(t, sanitized[0].Content, 2)
@@ -0,0 +1,84 @@
package chatd
import (
"context"
"errors"
"sync/atomic"
"charm.land/fantasy"
"github.com/google/uuid"
"github.com/openai/openai-go/v3"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/coderd/x/chatd/chatprovider"
)
func isInvalidEncryptedContentError(err error) bool {
var apiErr *openai.Error
return errors.As(err, &apiErr) && apiErr.Code == "invalid_encrypted_content"
}
// withEncryptedReasoningFallback wraps OpenAI Responses models in an
// encryptedReasoningFallbackModel that sets rejected when it retries.
func (p *Server) withEncryptedReasoningFallback(model chatprovider.Model, chatID uuid.UUID, rejected *atomic.Bool) chatprovider.Model {
if !model.Transport().UsesResponses() {
return model
}
return model.WithLanguageModel(&encryptedReasoningFallbackModel{
LanguageModel: model.LanguageModel(),
logger: p.logger.With(slog.F("chat_id", chatID)),
rejected: rejected,
})
}
// encryptedReasoningFallbackModel retries a request without OpenAI reasoning
// when the provider rejects its encrypted content. Encrypted reasoning only
// decrypts for the OpenAI organization that produced it, which can change
// without a provider, model, or credential change, for example when an admin
// replaces the deployment key with one from another organization. The
// rejected reasoning stays in the chat history, so the step stamps its new
// reasoning with the next epoch, and later steps leave out reasoning from
// earlier epochs.
type encryptedReasoningFallbackModel struct {
fantasy.LanguageModel
logger slog.Logger
rejected *atomic.Bool
}
// Stream retries only when the rejection arrives before any output.
func (m *encryptedReasoningFallbackModel) Stream(ctx context.Context, call fantasy.Call) (fantasy.StreamResponse, error) {
stream, err := m.LanguageModel.Stream(ctx, call)
if err != nil {
return nil, err
}
return func(yield func(fantasy.StreamPart) bool) {
var rejection error
started := false
for part := range stream {
if !started && part.Type == fantasy.StreamPartTypeError && isInvalidEncryptedContentError(part.Error) {
rejection = part.Error
break
}
started = started || part.Type != fantasy.StreamPartTypeWarnings
if !yield(part) {
return
}
}
if rejection == nil {
return
}
m.rejected.Store(true)
m.logger.Warn(ctx, "retrying model call without rejected encrypted reasoning", slogError(rejection))
call.Prompt = dropOpenAIReasoningParts(ctx, m.logger, call.Prompt)
retry, err := m.LanguageModel.Stream(ctx, call)
if err != nil {
yield(fantasy.StreamPart{Type: fantasy.StreamPartTypeError, Error: err})
return
}
for part := range retry {
if !yield(part) {
return
}
}
}, nil
}
+25 -6
View File
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"strings"
"sync/atomic"
"time"
"charm.land/fantasy"
@@ -71,7 +72,11 @@ type generationPrepared struct {
// ResolvedProvider is the configured provider identity used to label
// user-facing errors. See chatloop.GenerateAssistantOptions.ErrorProvider.
ResolvedProvider string
ResolvedProvider string
ReasoningProvenance reasoningProvenance
// ReasoningRejected reports whether the provider rejected the
// encrypted reasoning replayed into this step's model call.
ReasoningRejected *atomic.Bool
StageModel chatloop.StageModel
@@ -93,6 +98,17 @@ type generationPrepared struct {
Debug *generationDebug
}
// stepReasoningProvenance returns the provenance to stamp on the step's
// reasoning. A rejection starts the provider's next epoch, so later steps
// leave out the reasoning the rejected call replayed.
func (p generationPrepared) stepReasoningProvenance() reasoningProvenance {
provenance := p.ReasoningProvenance
if p.ReasoningRejected != nil && p.ReasoningRejected.Load() {
provenance.Epoch++
}
return provenance
}
// generationCompaction contains compaction inputs prepared for generation.
type generationCompaction struct {
// Override, when non-nil, is the compaction model override resolved at
@@ -966,6 +982,7 @@ func (s *taskStarter) generateAssistant(
logger: s.opts.Logger,
contentVersion: chatprompt.CurrentContentVersion,
hookRewrittenToolCalls: preflight.Overrides,
reasoningProvenance: prepared.stepReasoningProvenance(),
})
if err != nil {
return s.finishGenerationError(ctx, machine, input, err, requireGenerationAttempt(attempt.number))
@@ -1195,11 +1212,12 @@ func (s *taskStarter) executeLocalTools(
chathooks.RestoreToolCallOrder(outcome.Content, decision.localToolCalls)
step := stepDataFromPersisted(outcome)
messages, err := buildCommitStepMessages(buildCommitStepMessagesInput{
modelConfigID: prepared.ModelConfigID,
step: step,
toolNameToConfigID: prepared.ToolNameToConfigID,
logger: s.opts.Logger,
contentVersion: chatprompt.CurrentContentVersion,
modelConfigID: prepared.ModelConfigID,
step: step,
toolNameToConfigID: prepared.ToolNameToConfigID,
logger: s.opts.Logger,
contentVersion: chatprompt.CurrentContentVersion,
reasoningProvenance: prepared.stepReasoningProvenance(),
})
if err != nil {
return s.finishGenerationError(ctx, machine, input, err, requireGenerationAttempt(attempt.number))
@@ -1297,6 +1315,7 @@ func (s *taskStarter) generateCompaction(
overrideModel.resolvedProvider,
prepared.Compaction.ChatModelConfig,
overrideModel.dbConfig,
prepared.ReasoningProvenance.Model,
)
}
preResult, err := s.server.hooks.Trigger(ctx, chathooks.ChatFor(prepared.Chat, input.hookTurnID()), chathooks.Message{}, agenthooks.EventPreCompact, dispatch.CapacityClassGeneration)
+21 -7
View File
@@ -138,6 +138,7 @@ func (server *Server) prepareGeneration(
// The chat config keeps driving compaction, sanitization, and debug
// attribution even when computer use swaps the resolved call below.
modelConfig := resolved.dbConfig
providerIdentity := modelConfigProviderIdentity(modelConfig, chatprovider.NormalizeProvider(string(resolved.route.Provider.Type)))
// Computer-use turns swap in a specialized model, so the substitution
// must happen before anything model-sensitive runs: file-part
@@ -174,6 +175,11 @@ func (server *Server) prepareGeneration(
}
model := resolved.model
callConfig := resolved.callConfig
reasoningSource := reasoningProvenance{
ProviderIdentity: providerIdentity,
Model: model.ModelID(),
Credential: reasoningCredential(resolved.route.ProviderAuth),
}
modelRoute := resolved.route
currentPlanMode := chat.PlanMode
@@ -232,7 +238,10 @@ func (server *Server) prepareGeneration(
if advisorRuntime == nil {
return
}
advisorPromptSnapshot = slices.Clone(msgs)
// The advisor reads the transcript as text. Replayed OpenAI
// reasoning can be foreign to the advisor model, and textualized
// tool calls would leave it without its required following item.
advisorPromptSnapshot = dropOpenAIReasoningParts(ctx, logger, msgs)
}
currentChat := chat
@@ -324,11 +333,13 @@ func (server *Server) prepareGeneration(
planPathBlock string
)
// Drop provider-executed tool history produced by a different provider
// before building the prompt. A provider that shares another's wire format
// (e.g. Bedrock and Anthropic) can still reject the other's
// provider-executed blocks, so a mid-chat provider switch must not replay
// them.
// Drop provider-executed tool history produced by a different provider,
// and OpenAI reasoning state produced by a different provider, model,
// or credential, or before the latest rejection of that source's
// encrypted reasoning, before building the prompt. A provider
// that shares another's wire format (e.g. Bedrock and Anthropic) can
// still reject the other's provider-specific blocks, so a mid-chat
// provider switch must not replay them.
//
// The pending-user segment (trailing user rows the assistant has not
// answered yet) is handled separately from here through prompt
@@ -341,7 +352,8 @@ func (server *Server) prepareGeneration(
// sanitizing everything, since the tail is user-only and the
// sanitizer only rewrites assistant rows.
pendingRowsStart := pendingUserSegmentStart(promptRows)
sanitizedHead, err := server.sanitizeForeignProviderExecutedToolRows(ctx, logger, promptRows[:pendingRowsStart], chat.OwnerID, modelConfig.ID)
reasoningSource.Epoch = currentReasoningEpoch(promptRows[:pendingRowsStart], reasoningSource)
sanitizedHead, err := server.sanitizeForeignProviderStateRows(ctx, logger, promptRows[:pendingRowsStart], chat.OwnerID, reasoningSource)
if err != nil {
return generationPrepared{}, err
}
@@ -906,6 +918,8 @@ func (server *Server) prepareGeneration(
ProviderTools: providerTools,
ModelBuildOptions: modelOpts,
ResolvedProvider: resolved.resolvedProvider,
ReasoningProvenance: reasoningSource,
ReasoningRejected: resolved.reasoningRejected,
StageModel: resolved.stageModel(),
ModelConfigID: modelConfig.ID,
CallTemplate: resolved.newCall(),
+598 -34
View File
@@ -3,7 +3,10 @@ package chatd_test
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"net/http"
"slices"
"strings"
"sync"
"testing"
@@ -14,6 +17,7 @@ import (
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/database/db2sdk"
"github.com/coder/coder/v2/coderd/database/dbgen"
"github.com/coder/coder/v2/coderd/database/dbtestutil"
dbpubsub "github.com/coder/coder/v2/coderd/database/pubsub"
"github.com/coder/coder/v2/coderd/x/chatd"
@@ -25,47 +29,144 @@ import (
func TestOpenAIResponsesNoStaleWebSearchReplay(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
destinationStore bool
summary string
}{
{name: "stateless", summary: "checked provider-side search state"},
{name: "switch to stored", destinationStore: true},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
db, ps := dbtestutil.NewDB(t)
ctx := testutil.Context(t, testutil.WaitLong)
const (
reasoningID = "rs_no_stale_reasoning"
webSearchID = "ws_no_stale_search"
)
var recorder responsesRequestRecorder
openAIURL := chattest.NewOpenAI(t, func(req *chattest.OpenAIRequest) chattest.OpenAIResponse {
if !req.Stream {
return chattest.OpenAINonStreamingResponse("title")
}
requestNumber := recorder.record(req)
switch requestNumber {
case 1:
resp := chattest.OpenAIStreamingResponse(
chattest.OpenAITextChunks("search result summary")...,
)
resp.ResponseID = "resp_no_stale_first"
resp.Reasoning = &chattest.OpenAIReasoningItem{
ID: reasoningID,
Summary: tc.summary,
EncryptedContent: "encrypted-no-stale",
}
resp.WebSearch = &chattest.OpenAIWebSearchCall{
ID: webSearchID,
Query: "coder changelog",
}
return resp
default:
resp := chattest.OpenAIStreamingResponse(
chattest.OpenAITextChunks("follow-up answer")...,
)
resp.ResponseID = "resp_no_stale_second"
return resp
}
})
user, org, _ := seedChatDependenciesWithProvider(t, db, "openai", openAIURL)
model := insertOpenAIResponsesModelConfig(t, db, user.ID, false, true)
followupModel := model
if tc.destinationStore {
followupModel = insertOpenAIResponsesModelConfig(t, db, user.ID, true, false)
}
factory := chattest.NewMockAIBridgeTransport(t, openAIURL)
server := newOpenAIResponsesTestServer(t, db, ps, func(cfg *chatd.Config) {
cfg.AIBridgeTransportFactory = chatAIGatewayTransportFactoryPointer(factory)
})
chat, err := server.CreateChat(ctx, chatd.CreateOptions{
OrganizationID: org.ID,
OwnerID: user.ID,
Title: uniqueResponsesTitle(t, "no-stale"),
ModelConfigID: model.ID,
InitialUserContent: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("search for the latest Coder docs"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
require.Len(t, recorder.all(), 1)
_, err = server.SendMessage(ctx, chatd.SendMessageOptions{
ChatID: chat.ID,
CreatedBy: user.ID,
ModelConfigID: followupModel.ID,
Content: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("summarize the result without searching again"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
requests := recorder.all()
require.Len(t, requests, 2)
followup := requests[1]
require.NotNil(t, followup.Store)
require.Equal(t, tc.destinationStore, *followup.Store)
require.NotEmpty(t, followup.Prompt)
requireNoResponsesProviderItemReplay(t, followup.Prompt, webSearchID)
require.NotContains(t, promptItemTypes(followup.Prompt), "web_search_call")
requireInlineReasoningItem(t, followup.Prompt, reasoningID, "encrypted-no-stale",
tc.summary)
})
}
}
func TestOpenAIResponsesStatelessReasoningReplay(t *testing.T) {
t.Parallel()
db, ps := dbtestutil.NewDB(t)
ctx := testutil.Context(t, testutil.WaitLong)
const (
reasoningID = "rs_no_stale_reasoning"
webSearchID = "ws_no_stale_search"
reasoningID = "rs_stateless_reasoning"
blob = "encrypted-stateless"
)
var recorder responsesRequestRecorder
openAIURL := chattest.NewOpenAI(t, func(req *chattest.OpenAIRequest) chattest.OpenAIResponse {
if !req.Stream {
return chattest.OpenAINonStreamingResponse("title")
}
requestNumber := recorder.record(req)
switch requestNumber {
switch recorder.record(req) {
case 1:
resp := chattest.OpenAIStreamingResponse(
chattest.OpenAITextChunks("search result summary")...,
chattest.OpenAIToolCallChunk("list_templates", `{}`),
)
resp.ResponseID = "resp_no_stale_first"
resp.Reasoning = &chattest.OpenAIReasoningItem{
ID: reasoningID,
Summary: "checked provider-side search state",
EncryptedContent: "encrypted-no-stale",
}
resp.WebSearch = &chattest.OpenAIWebSearchCall{
ID: webSearchID,
Query: "coder changelog",
EncryptedContent: blob,
}
return resp
default:
resp := chattest.OpenAIStreamingResponse(
chattest.OpenAITextChunks("follow-up answer")...,
return chattest.OpenAIStreamingResponse(
chattest.OpenAITextChunks("done")...,
)
resp.ResponseID = "resp_no_stale_second"
return resp
}
})
user, org, _ := seedChatDependenciesWithProvider(t, db, "openai", openAIURL)
model := insertOpenAIResponsesModelConfig(t, db, user.ID, false, true)
// An empty call config is the default deployment shape: no stored
// responses and no requested reasoning summary.
model := insertChatModelConfigWithCallConfig(t, db, user.ID, "openai", "gpt-4o",
codersdk.ChatModelCallConfig{})
factory := chattest.NewMockAIBridgeTransport(t, openAIURL)
server := newOpenAIResponsesTestServer(t, db, ps, func(cfg *chatd.Config) {
cfg.AIBridgeTransportFactory = chatAIGatewayTransportFactoryPointer(factory)
@@ -74,23 +175,23 @@ func TestOpenAIResponsesNoStaleWebSearchReplay(t *testing.T) {
chat, err := server.CreateChat(ctx, chatd.CreateOptions{
OrganizationID: org.ID,
OwnerID: user.ID,
Title: uniqueResponsesTitle(t, "no-stale"),
Title: uniqueResponsesTitle(t, "stateless-reasoning"),
ModelConfigID: model.ID,
InitialUserContent: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("search for the latest Coder docs"),
codersdk.ChatMessageText("list the templates"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
require.Len(t, recorder.all(), 1)
require.Len(t, recorder.all(), 2)
_, err = server.SendMessage(ctx, chatd.SendMessageOptions{
ChatID: chat.ID,
CreatedBy: user.ID,
ModelConfigID: model.ID,
Content: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("summarize the result without searching again"),
codersdk.ChatMessageText("thanks"),
},
})
require.NoError(t, err)
@@ -98,13 +199,432 @@ func TestOpenAIResponsesNoStaleWebSearchReplay(t *testing.T) {
requireResponsesChatWaiting(ctx, t, db, chat.ID)
requests := recorder.all()
require.Len(t, requests, 2)
followup := requests[1]
require.NotNil(t, followup.Store)
require.False(t, *followup.Store)
require.NotEmpty(t, followup.Prompt)
requireNoResponsesProviderItemReplay(t, followup.Prompt, reasoningID, webSearchID)
require.NotContains(t, promptItemTypes(followup.Prompt), "web_search_call")
require.Len(t, requests, 3)
for _, request := range requests {
require.NotNil(t, request.Store)
require.False(t, *request.Store)
require.Contains(t, request.Include, "reasoning.encrypted_content")
}
// requests[1] continues the tool step from in-memory history;
// requests[2] rebuilds the prompt from persisted messages.
for _, request := range requests[1:] {
reasoningIndex := requireInlineReasoningItem(t, request.Prompt, reasoningID, blob, "")
require.NotContains(t, promptItemTypes(request.Prompt), "item_reference")
functionCallIndex := slices.IndexFunc(request.Prompt, func(item interface{}) bool {
itemMap, ok := item.(map[string]interface{})
return ok && chattest.StringResponseField(itemMap, "type") == "function_call"
})
require.NotEqual(t, -1, functionCallIndex, "missing function_call item")
require.Less(t, reasoningIndex, functionCallIndex, "types=%v", promptItemTypes(request.Prompt))
}
}
func TestOpenAIResponsesReasoningNotReplayedAfterModelConfigChange(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
change func(cfg *database.UpdateChatModelConfigParams, otherProvider uuid.UUID)
}{
{
name: "ProviderMoved",
change: func(cfg *database.UpdateChatModelConfigParams, otherProvider uuid.UUID) {
cfg.AIProviderID = uuid.NullUUID{UUID: otherProvider, Valid: true}
},
},
{
name: "ModelChanged",
change: func(cfg *database.UpdateChatModelConfigParams, _ uuid.UUID) {
cfg.Model = "gpt-4.1"
},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
db, ps := dbtestutil.NewDB(t)
ctx := testutil.Context(t, testutil.WaitLong)
const (
reasoningID = "rs_changed_config"
blob = "encrypted-changed-config"
)
var recorder responsesRequestRecorder
openAIURL := chattest.NewOpenAI(t, func(req *chattest.OpenAIRequest) chattest.OpenAIResponse {
if !req.Stream {
return chattest.OpenAINonStreamingResponse("title")
}
resp := chattest.OpenAIStreamingResponse(chattest.OpenAITextChunks("answer")...)
if recorder.record(req) == 1 {
resp.Reasoning = &chattest.OpenAIReasoningItem{
ID: reasoningID,
EncryptedContent: blob,
}
}
return resp
})
user, org, _ := seedChatDependenciesWithProvider(t, db, "openai", openAIURL)
model := insertChatModelConfigWithCallConfig(t, db, user.ID, "openai", "gpt-4o",
codersdk.ChatModelCallConfig{})
otherProvider := dbgen.ChatProvider(t, db, database.ChatProvider{
Provider: "openai",
BaseUrl: openAIURL,
})
factory := chattest.NewMockAIBridgeTransport(t, openAIURL)
server := newOpenAIResponsesTestServer(t, db, ps, func(cfg *chatd.Config) {
cfg.AIBridgeTransportFactory = chatAIGatewayTransportFactoryPointer(factory)
})
chat, err := server.CreateChat(ctx, chatd.CreateOptions{
OrganizationID: org.ID,
OwnerID: user.ID,
Title: uniqueResponsesTitle(t, "changed-config"),
ModelConfigID: model.ID,
InitialUserContent: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("hello"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
require.Len(t, recorder.all(), 1)
update := database.UpdateChatModelConfigParams{
ID: model.ID,
Model: model.Model,
DisplayName: model.DisplayName,
Enabled: model.Enabled,
IsDefault: model.IsDefault,
ContextLimit: model.ContextLimit,
CompressionThreshold: model.CompressionThreshold,
Options: model.Options,
AIProviderID: model.AIProviderID,
}
tc.change(&update, otherProvider.ID)
_, err = db.UpdateChatModelConfig(ctx, update)
require.NoError(t, err)
_, err = server.SendMessage(ctx, chatd.SendMessageOptions{
ChatID: chat.ID,
CreatedBy: user.ID,
ModelConfigID: model.ID,
Content: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("thanks"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
requests := recorder.all()
require.Len(t, requests, 2)
followup := requests[1]
require.NotEmpty(t, followup.Prompt)
require.NotContains(t, promptItemTypes(followup.Prompt), "reasoning")
requireNoResponsesProviderItemReplay(t, followup.Prompt, reasoningID)
encoded, err := json.Marshal(followup.Prompt)
require.NoError(t, err)
require.NotContains(t, string(encoded), blob)
})
}
}
func TestOpenAIResponsesRetriesWithoutRejectedEncryptedReasoning(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// freshReasoningAt is the request whose response carries the
// first reasoning produced after the rejection.
freshReasoningAt int
}{
{name: "RetryReturnsReasoning", freshReasoningAt: 3},
{name: "RetryReturnsNoReasoning", freshReasoningAt: 4},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
db, ps := dbtestutil.NewDB(t)
ctx := testutil.Context(t, testutil.WaitLong)
const (
staleID = "rs_rejected_reasoning"
staleBlob = "encrypted-by-another-organization"
freshID = "rs_fresh_reasoning"
freshBlob = "encrypted-by-current-organization"
)
var recorder responsesRequestRecorder
openAIURL := chattest.NewOpenAI(t, func(req *chattest.OpenAIRequest) chattest.OpenAIResponse {
if !req.Stream {
return chattest.OpenAINonStreamingResponse("title")
}
n := recorder.record(req)
// Later turns reach an organization that cannot decrypt the
// reasoning from the first turn.
if n > 1 && slices.ContainsFunc(req.Prompt, func(item interface{}) bool {
itemMap, ok := item.(map[string]interface{})
return ok && chattest.StringResponseField(itemMap, "id") == staleID
}) {
resp := chattest.OpenAIErrorResponse(http.StatusBadRequest, "invalid_request_error",
"The encrypted content could not be verified.")
resp.Error.Code = "invalid_encrypted_content"
return resp
}
resp := chattest.OpenAIStreamingResponse(chattest.OpenAITextChunks("done")...)
if n == 3 {
resp = chattest.OpenAIStreamingResponse(chattest.OpenAIToolCallChunk("list_templates", `{}`))
}
switch n {
case 1:
resp.Reasoning = &chattest.OpenAIReasoningItem{ID: staleID, EncryptedContent: staleBlob}
case tc.freshReasoningAt:
resp.Reasoning = &chattest.OpenAIReasoningItem{ID: freshID, EncryptedContent: freshBlob}
}
return resp
})
user, org, _ := seedChatDependenciesWithProvider(t, db, "openai", openAIURL)
model := insertChatModelConfigWithCallConfig(t, db, user.ID, "openai", "gpt-4o",
codersdk.ChatModelCallConfig{})
factory := chattest.NewMockAIBridgeTransport(t, openAIURL)
newServer := func() *chatd.Server {
return newOpenAIResponsesTestServer(t, db, ps, func(cfg *chatd.Config) {
cfg.AIBridgeTransportFactory = chatAIGatewayTransportFactoryPointer(factory)
})
}
server := newServer()
sendMessage := func(server *chatd.Server, chatID uuid.UUID, text string) {
t.Helper()
_, err := server.SendMessage(ctx, chatd.SendMessageOptions{
ChatID: chatID,
CreatedBy: user.ID,
ModelConfigID: model.ID,
Content: []codersdk.ChatMessagePart{
codersdk.ChatMessageText(text),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chatID, server)
requireResponsesChatWaiting(ctx, t, db, chatID)
}
chat, err := server.CreateChat(ctx, chatd.CreateOptions{
OrganizationID: org.ID,
OwnerID: user.ID,
Title: uniqueResponsesTitle(t, "rejected-reasoning"),
ModelConfigID: model.ID,
InitialUserContent: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("hello"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
sendMessage(server, chat.ID, "list the templates")
// The next turn runs on a server that never saw the rejection,
// like a restarted or different replica.
require.NoError(t, server.Close())
sendMessage(newServer(), chat.ID, "thanks")
// Only the first request after the change is rejected.
requests := recorder.all()
require.Len(t, requests, 5)
requireInlineReasoningItem(t, requests[1].Prompt, staleID, staleBlob, "")
retry := requests[2]
require.NotEmpty(t, retry.Prompt)
require.NotContains(t, promptItemTypes(retry.Prompt), "reasoning")
// requests[3] is the step after the tool call; requests[4] is
// the next turn on the fresh server.
for i, request := range requests[3:] {
requireNoResponsesProviderItemReplay(t, request.Prompt, staleID)
if i+3 < tc.freshReasoningAt {
require.NotContains(t, promptItemTypes(request.Prompt), "reasoning")
continue
}
requireInlineReasoningItem(t, request.Prompt, freshID, freshBlob, "")
}
})
}
}
func TestOpenAIResponsesReasoningNotReplayedAfterCredentialChange(t *testing.T) {
t.Parallel()
db, ps := dbtestutil.NewDB(t)
ctx := testutil.Context(t, testutil.WaitLong)
const (
userAPIKey = "sk-user-reasoning"
deploymentID = "rs_deployment_key"
deploymentBlob = "encrypted-by-deployment-organization"
personalKeyID = "rs_personal_key"
personalKeyBlob = "encrypted-by-personal-organization"
)
var (
recorder responsesRequestRecorder
authMu sync.Mutex
authByCall = map[int]string{}
)
openAIURL := chattest.NewOpenAI(t, func(req *chattest.OpenAIRequest) chattest.OpenAIResponse {
if !req.Stream {
return chattest.OpenAINonStreamingResponse("title")
}
n := recorder.record(req)
authMu.Lock()
authByCall[n] = req.Header.Get("Authorization")
authMu.Unlock()
resp := chattest.OpenAIStreamingResponse(chattest.OpenAITextChunks("answer")...)
switch n {
case 1:
resp.Reasoning = &chattest.OpenAIReasoningItem{ID: deploymentID, EncryptedContent: deploymentBlob}
case 2:
resp.Reasoning = &chattest.OpenAIReasoningItem{ID: personalKeyID, EncryptedContent: personalKeyBlob}
}
return resp
})
user, org, _ := seedChatDependenciesWithProvider(t, db, "openai", openAIURL)
model := insertChatModelConfigWithCallConfig(t, db, user.ID, "openai", "gpt-4o",
codersdk.ChatModelCallConfig{})
factory := chattest.NewMockAIBridgeTransport(t, openAIURL)
server := newOpenAIResponsesTestServer(t, db, ps, func(cfg *chatd.Config) {
cfg.AIBridgeTransportFactory = chatAIGatewayTransportFactoryPointer(factory)
})
chat, err := server.CreateChat(ctx, chatd.CreateOptions{
OrganizationID: org.ID,
OwnerID: user.ID,
Title: uniqueResponsesTitle(t, "credential-change"),
ModelConfigID: model.ID,
InitialUserContent: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("hello"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
_, err = db.UpsertUserAIProviderKey(ctx, database.UpsertUserAIProviderKeyParams{
ID: uuid.New(),
UserID: user.ID,
AIProviderID: model.AIProviderID.UUID,
APIKey: userAPIKey,
})
require.NoError(t, err)
for _, text := range []string{"switched keys", "thanks"} {
_, err = server.SendMessage(ctx, chatd.SendMessageOptions{
ChatID: chat.ID,
CreatedBy: user.ID,
ModelConfigID: model.ID,
Content: []codersdk.ChatMessagePart{
codersdk.ChatMessageText(text),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
}
requests := recorder.all()
require.Len(t, requests, 3)
authMu.Lock()
require.NotEqual(t, "Bearer "+userAPIKey, authByCall[1])
require.Equal(t, "Bearer "+userAPIKey, authByCall[2])
authMu.Unlock()
// Reasoning from the deployment's key is left out before the
// personal key's organization can reject it.
require.NotContains(t, promptItemTypes(requests[1].Prompt), "reasoning")
requireNoResponsesProviderItemReplay(t, requests[1].Prompt, deploymentID)
requireInlineReasoningItem(t, requests[2].Prompt, personalKeyID, personalKeyBlob, "")
requireNoResponsesProviderItemReplay(t, requests[2].Prompt, deploymentID)
}
func TestOpenAIResponsesAdvisorPromptOmitsReasoning(t *testing.T) {
t.Parallel()
db, ps := dbtestutil.NewDB(t)
ctx := testutil.Context(t, testutil.WaitLong)
const (
reasoningID = "rs_parent_reasoning"
blob = "encrypted-parent-reasoning"
)
var recorder responsesRequestRecorder
openAIURL := chattest.NewOpenAI(t, func(req *chattest.OpenAIRequest) chattest.OpenAIResponse {
if !req.Stream {
return chattest.OpenAINonStreamingResponse("title")
}
switch recorder.record(req) {
case 1:
resp := chattest.OpenAIStreamingResponse(chattest.OpenAITextChunks("answer")...)
resp.Reasoning = &chattest.OpenAIReasoningItem{
ID: reasoningID,
EncryptedContent: blob,
}
return resp
case 2:
return chattest.OpenAIStreamingResponse(
chattest.OpenAIToolCallChunk("advisor", `{"question":"what next?"}`),
)
case 3:
return chattest.OpenAIStreamingResponse(chattest.OpenAITextChunks("advice")...)
default:
return chattest.OpenAIStreamingResponse(chattest.OpenAITextChunks("done")...)
}
})
user, org, _ := seedChatDependenciesWithProvider(t, db, "openai", openAIURL)
model := insertChatModelConfigWithCallConfig(t, db, user.ID, "openai", "gpt-4o",
codersdk.ChatModelCallConfig{})
seedAdvisorConfig(ctx, t, db, codersdk.AdvisorConfig{
Enabled: true,
MaxUsesPerRun: 1,
MaxOutputTokens: 1024,
})
factory := chattest.NewMockAIBridgeTransport(t, openAIURL)
server := newOpenAIResponsesTestServer(t, db, ps, func(cfg *chatd.Config) {
cfg.AIBridgeTransportFactory = chatAIGatewayTransportFactoryPointer(factory)
})
chat, err := server.CreateChat(ctx, chatd.CreateOptions{
OrganizationID: org.ID,
OwnerID: user.ID,
Title: uniqueResponsesTitle(t, "advisor-reasoning"),
ModelConfigID: model.ID,
InitialUserContent: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("hello"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
_, err = server.SendMessage(ctx, chatd.SendMessageOptions{
ChatID: chat.ID,
CreatedBy: user.ID,
ModelConfigID: model.ID,
Content: []codersdk.ChatMessagePart{
codersdk.ChatMessageText("ask the advisor"),
},
})
require.NoError(t, err)
waitForChatProcessed(ctx, t, db, chat.ID, server)
requireResponsesChatWaiting(ctx, t, db, chat.ID)
requests := recorder.all()
require.Len(t, requests, 4)
requireInlineReasoningItem(t, requests[1].Prompt, reasoningID, blob, "")
advisorRequest := requests[2]
require.NotEmpty(t, advisorRequest.Prompt)
require.NotContains(t, promptItemTypes(advisorRequest.Prompt), "reasoning")
encoded, err := json.Marshal(advisorRequest.Prompt)
require.NoError(t, err)
require.Contains(t, string(encoded), "what next?")
require.NotContains(t, string(encoded), blob)
requireInlineReasoningItem(t, requests[3].Prompt, reasoningID, blob, "")
}
func TestOpenAIResponsesPersistsProviderResponseID(t *testing.T) {
@@ -320,8 +840,9 @@ func TestOpenAIResponsesNarrationKeepsPhase(t *testing.T) {
}
type recordedResponsesRequest struct {
Prompt []interface{}
Store *bool
Prompt []interface{}
Store *bool
Include []string
}
type responsesRequestRecorder struct {
@@ -338,9 +859,14 @@ func (r *responsesRequestRecorder) record(req *chattest.OpenAIRequest) int {
value := *req.Store
store = &value
}
var body struct {
Include []string `json:"include"`
}
_ = json.Unmarshal(req.RawBody, &body)
r.requests = append(r.requests, recordedResponsesRequest{
Prompt: append([]interface{}(nil), req.Prompt...),
Store: store,
Prompt: append([]interface{}(nil), req.Prompt...),
Store: store,
Include: body.Include,
})
return len(r.requests)
}
@@ -456,12 +982,17 @@ func assertNoResponsesProviderItemReplay(
if key == "type" && text == "web_search_call" {
require.FailNow(t, "prompt replayed web_search_call provider item")
}
if key == "type" && text == "item_reference" {
require.FailNow(t, "prompt replayed provider item reference")
}
if key == "id" || key == "call_id" || key == "item_id" {
if _, isStale := staleIDs[text]; isStale {
require.FailNowf(t, "prompt replayed stale provider item ID",
"field %q contained stale provider ID %q", key, text)
}
if strings.HasPrefix(text, "ws_") || strings.HasPrefix(text, "rs_") {
// Finalized reasoning is replayed inline, so only hosted
// tool items are provider-managed state here.
if strings.HasPrefix(text, "ws_") {
require.FailNowf(t, "prompt replayed provider item ID",
"field %q contained provider-managed ID %q", key, text)
}
@@ -476,6 +1007,39 @@ func assertNoResponsesProviderItemReplay(
}
}
// requireInlineReasoningItem asserts that the prompt replays the reasoning
// item in full and returns its index. An empty summary must still be sent
// as an empty array.
func requireInlineReasoningItem(
t *testing.T,
prompt []interface{},
id string,
encryptedContent string,
summary string,
) int {
t.Helper()
for index, item := range prompt {
itemMap, ok := item.(map[string]interface{})
if !ok || chattest.StringResponseField(itemMap, "type") != "reasoning" ||
chattest.StringResponseField(itemMap, "id") != id {
continue
}
require.Equal(t, encryptedContent, chattest.StringResponseField(itemMap, "encrypted_content"))
summaryItems, ok := itemMap["summary"].([]interface{})
require.True(t, ok, "reasoning summary must be an array, got %T", itemMap["summary"])
var texts []string
for _, summaryItem := range summaryItems {
summaryMap, ok := summaryItem.(map[string]interface{})
require.True(t, ok)
texts = append(texts, chattest.StringResponseField(summaryMap, "text"))
}
require.Equal(t, summary, strings.Join(texts, ""))
return index
}
require.FailNowf(t, "missing inline reasoning item", "id=%q types=%v", id, promptItemTypes(prompt))
return -1
}
func requirePromptItemReferenceOrder(
t *testing.T,
prompt []interface{},
+21
View File
@@ -33,6 +33,14 @@ type buildCommitStepMessagesInput struct {
logger slog.Logger
contentVersion int16
hookRewrittenToolCalls map[string]json.RawMessage
reasoningProvenance reasoningProvenance
}
func stampReasoningProvenance(part *codersdk.ChatMessagePart, provenance reasoningProvenance) {
part.ProviderIdentity = provenance.ProviderIdentity
part.ProviderModel = provenance.Model
part.ProviderCredential = provenance.Credential
part.ReasoningEpoch = provenance.Epoch
}
type stepMessagesForCommit struct {
@@ -50,6 +58,19 @@ func buildCommitStepMessages(input buildCommitStepMessagesInput) (stepMessagesFo
assistantBlocks, toolResults := splitStepContent(input.step.Content)
assistantParts := buildAssistantParts(input.logger, assistantBlocks, toolResults, input.step, input.toolNameToConfigID, input.hookRewrittenToolCalls)
stamped := false
for i := range assistantParts {
if assistantParts[i].Type == codersdk.ChatMessagePartTypeReasoning {
stampReasoningProvenance(&assistantParts[i], input.reasoningProvenance)
stamped = true
}
}
// Without a stamp, a step that produced no reasoning would lose the
// rejection epoch, and the next step would replay the rejected
// reasoning again.
if !stamped && input.reasoningProvenance.Epoch > 0 && len(assistantParts) > 0 {
stampReasoningProvenance(&assistantParts[0], input.reasoningProvenance)
}
messages := make([]chatstate.Message, 0, 1+len(toolResults))
if len(assistantParts) > 0 {
+14
View File
@@ -33,6 +33,12 @@ func TestBuildCommitStepMessages_AssistantTextAndReasoning(t *testing.T) {
modelConfigID: modelConfigID,
contentVersion: chatprompt.CurrentContentVersion,
logger: slog.Make(),
reasoningProvenance: reasoningProvenance{
ProviderIdentity: "provider-a",
Model: "model-a",
Credential: reasoningCredentialUser,
Epoch: 2,
},
step: stepData{
Content: []fantasy.Content{
fantasy.ReasoningContent{Text: "thinking"},
@@ -56,8 +62,16 @@ func TestBuildCommitStepMessages_AssistantTextAndReasoning(t *testing.T) {
require.Equal(t, "thinking", parts[0].Text)
require.Equal(t, startedAt, requireNotNilTime(t, parts[0].CreatedAt))
require.Equal(t, completedAt, requireNotNilTime(t, parts[0].CompletedAt))
require.Equal(t, "provider-a", parts[0].ProviderIdentity)
require.Equal(t, "model-a", parts[0].ProviderModel)
require.Equal(t, reasoningCredentialUser, parts[0].ProviderCredential)
require.Equal(t, 2, parts[0].ReasoningEpoch)
require.Equal(t, codersdk.ChatMessagePartTypeText, parts[1].Type)
require.Equal(t, "hello", parts[1].Text)
require.Empty(t, parts[1].ProviderIdentity)
require.Empty(t, parts[1].ProviderModel)
require.Empty(t, parts[1].ProviderCredential)
require.Zero(t, parts[1].ReasoningEpoch)
}
func TestBuildCommitStepMessages_LocalToolResultsBecomeToolMessages(t *testing.T) {
+6
View File
@@ -2,6 +2,7 @@ package chatd
import (
"context"
"sync/atomic"
"charm.land/fantasy"
"golang.org/x/xerrors"
@@ -72,6 +73,9 @@ type resolvedModelCall struct {
resolvedEffort string
route aiGatewayModelRoute
debugEnabled bool
// reasoningRejected reports whether the provider rejected encrypted
// reasoning replayed into a call to model.
reasoningRejected *atomic.Bool
}
// stageModel uses the configured provider type rather than
@@ -182,6 +186,8 @@ func (p *Server) resolveModelCall(ctx context.Context, spec modelCallSpec) (reso
return resolvedModelCall{}, xerrors.Errorf("create model: %w", err)
}
model = p.withThinkingDropBlock(model, out.route.Provider.ID, clientCallConfig, spec.chat.ID)
out.reasoningRejected = &atomic.Bool{}
model = p.withEncryptedReasoningFallback(model, spec.chat.ID, out.reasoningRejected)
if out.debugEnabled {
model = model.WithLanguageModel(chatdebug.WrapModel(model.LanguageModel(), debugSvc, chatdebug.RecorderOptions{
+145 -38
View File
@@ -1,22 +1,27 @@
package chatd
import (
"bytes"
"context"
"encoding/json"
"charm.land/fantasy"
"github.com/google/uuid"
"golang.org/x/xerrors"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/x/chatd/chatprompt"
"github.com/coder/coder/v2/coderd/x/chatd/chatsanitize"
"github.com/coder/coder/v2/codersdk"
)
// providerSwitchStripStats counts provider-executed tool history removed
// during a provider switch.
// providerSwitchStripStats counts provider-specific history removed during a
// provider switch.
type providerSwitchStripStats struct {
RemovedToolCalls int
RemovedToolResults int
RemovedReasoning int
DroppedMessages int
}
@@ -33,20 +38,100 @@ func modelConfigProviderIdentity(modelConfig database.ChatModelConfig, normalize
return normalizedProvider
}
// stripForeignProviderExecutedToolRows drops provider-executed tool blocks
// (calls and results) from assistant rows whose producing provider differs
// from targetIdentity. Rows with an unknown origin are treated as foreign
// (fail closed). Rows emptied by stripping are dropped; rows that fail to parse
// or re-marshal are kept unchanged.
const (
reasoningCredentialDeployment = "deployment"
reasoningCredentialUser = "user"
)
// reasoningProvenance identifies where OpenAI reasoning state was produced.
// Reasoning item IDs only resolve on the issuing provider instance, and
// encrypted content can be rejected by a different model or by the
// organization behind a different credential. Epoch counts rejections of
// the chat's encrypted reasoning from the same provider, model, and
// credential, so reasoning replayed into a rejected call is never replayed
// again.
type reasoningProvenance struct {
ProviderIdentity string
Model string
Credential string
Epoch int
}
// sameSource reports whether p and o name the same provider, model, and
// credential, regardless of epoch.
func (p reasoningProvenance) sameSource(o reasoningProvenance) bool {
return p.ProviderIdentity == o.ProviderIdentity && p.Model == o.Model && p.Credential == o.Credential
}
// stampedReasoningProvenance returns the provenance recorded on part.
func stampedReasoningProvenance(part codersdk.ChatMessagePart) reasoningProvenance {
return reasoningProvenance{
ProviderIdentity: part.ProviderIdentity,
Model: part.ProviderModel,
Credential: part.ProviderCredential,
Epoch: part.ReasoningEpoch,
}
}
// reasoningCredential names the credential an AI Gateway route sends.
func reasoningCredential(auth aiGatewayProviderAuth) string {
if len(auth.Headers) > 0 {
return reasoningCredentialUser
}
return reasoningCredentialDeployment
}
// unstampedReasoningProvenance is the provenance assumed for reasoning saved
// without a stamp: the row's model config, the target's credential, and the
// first epoch.
func unstampedReasoningProvenance(rowOrigin, target reasoningProvenance) reasoningProvenance {
return reasoningProvenance{
ProviderIdentity: rowOrigin.ProviderIdentity,
Model: rowOrigin.Model,
Credential: target.Credential,
}
}
// currentReasoningEpoch returns the highest epoch recorded in rows for the
// provider, model, and credential of target. A rejected request only
// replayed reasoning from that source, so the rejection says nothing about
// reasoning from other models or credentials.
func currentReasoningEpoch(rows []database.ChatMessage, target reasoningProvenance) int {
epoch := 0
for _, row := range rows {
if row.Role != database.ChatMessageRoleAssistant || !bytes.Contains(row.Content.RawMessage, []byte(`"reasoning_epoch"`)) {
continue
}
parts, err := chatprompt.ParseContent(row)
if err != nil {
continue
}
for _, part := range parts {
if source := stampedReasoningProvenance(part); source.sameSource(target) {
epoch = max(epoch, source.Epoch)
}
}
}
return epoch
}
// stripForeignProviderStateRows drops provider-executed tool blocks (calls and
// results) from assistant rows whose producing provider differs from the
// target's, and OpenAI reasoning state whose provenance differs from target.
// A reasoning part's recorded provenance takes precedence over its row's
// origin, since the row's model config may have been changed. Rows with an
// unknown origin are treated as foreign (fail closed). Rows emptied by
// stripping are dropped; rows that fail to parse or re-marshal are kept
// unchanged.
//
// See modelConfigProviderIdentity for how identity is derived.
func stripForeignProviderExecutedToolRows(
func stripForeignProviderStateRows(
rows []database.ChatMessage,
targetIdentity string,
originProvider func(uuid.NullUUID) (string, bool),
target reasoningProvenance,
originOf func(uuid.NullUUID) reasoningProvenance,
) ([]database.ChatMessage, providerSwitchStripStats) {
var stats providerSwitchStripStats
if targetIdentity == "" || len(rows) == 0 {
if target.ProviderIdentity == "" || len(rows) == 0 {
return rows, stats
}
@@ -56,7 +141,10 @@ func stripForeignProviderExecutedToolRows(
out = append(out, row)
continue
}
if origin, ok := originProvider(row.ModelConfigID); ok && origin == targetIdentity {
origin := originOf(row.ModelConfigID)
nativeRow := origin.ProviderIdentity == target.ProviderIdentity
// Only stamped reasoning parts can be foreign when the row origin matches.
if unstampedReasoningProvenance(origin, target) == target && !bytes.Contains(row.Content.RawMessage, []byte(`"provider_identity"`)) {
out = append(out, row)
continue
}
@@ -68,23 +156,26 @@ func stripForeignProviderExecutedToolRows(
}
kept := make([]codersdk.ChatMessagePart, 0, len(parts))
var removedCalls, removedResults int
var removedCalls, removedResults, removedReasoning int
for _, part := range parts {
switch {
case part.Type == codersdk.ChatMessagePartTypeToolCall && part.ProviderExecuted:
case !nativeRow && part.Type == codersdk.ChatMessagePartTypeToolCall && part.ProviderExecuted:
removedCalls++
case part.Type == codersdk.ChatMessagePartTypeToolResult && part.ProviderExecuted:
case !nativeRow && part.Type == codersdk.ChatMessagePartTypeToolResult && part.ProviderExecuted:
removedResults++
case part.Type == codersdk.ChatMessagePartTypeReasoning && isForeignReasoning(part, origin, target):
removedReasoning++
default:
kept = append(kept, part)
}
}
if removedCalls == 0 && removedResults == 0 {
if removedCalls == 0 && removedResults == 0 && removedReasoning == 0 {
out = append(out, row)
continue
}
stats.RemovedToolCalls += removedCalls
stats.RemovedToolResults += removedResults
stats.RemovedReasoning += removedReasoning
if len(kept) == 0 {
stats.DroppedMessages++
continue
@@ -102,28 +193,40 @@ func stripForeignProviderExecutedToolRows(
return out, stats
}
func (server *Server) sanitizeForeignProviderExecutedToolRows(
func isForeignReasoning(part codersdk.ChatMessagePart, rowOrigin, target reasoningProvenance) bool {
source := unstampedReasoningProvenance(rowOrigin, target)
if part.ProviderIdentity != "" {
source = stampedReasoningProvenance(part)
}
return source != target && hasOpenAIReasoningState(part)
}
func hasOpenAIReasoningState(part codersdk.ChatMessagePart) bool {
if len(part.ProviderMetadata) == 0 {
return false
}
var raw map[string]json.RawMessage
if err := json.Unmarshal(part.ProviderMetadata, &raw); err != nil {
return false
}
options, err := fantasy.UnmarshalProviderOptions(raw)
return err == nil && chatsanitize.HasOpenAIReasoningState(options)
}
func (server *Server) sanitizeForeignProviderStateRows(
ctx context.Context,
logger slog.Logger,
rows []database.ChatMessage,
ownerID uuid.UUID,
modelConfigID uuid.UUID,
target reasoningProvenance,
) ([]database.ChatMessage, error) {
targetCfg, targetProvider, err := server.resolveModelConfigAndNormalizedProvider(ctx, ownerID, modelConfigID)
if err != nil && !modelConfigUnavailable(err) {
return nil, xerrors.Errorf("resolve target model config: %w", err)
}
if err != nil || targetProvider == "" {
logger.Debug(ctx, "skipping provider-switch sanitization: target provider unresolved",
slog.F("model_config_id", modelConfigID),
slog.Error(err),
)
if target.ProviderIdentity == "" {
return rows, nil
}
targetIdentity := modelConfigProviderIdentity(targetCfg, targetProvider)
// An unavailable origin config maps to "", so its rows are foreign.
origins := make(map[uuid.UUID]string)
// An unavailable origin config maps to the zero provenance, so its rows
// are foreign.
origins := make(map[uuid.UUID]reasoningProvenance)
for _, row := range rows {
id := row.ModelConfigID
if row.Role != database.ChatMessageRoleAssistant || !id.Valid {
@@ -135,25 +238,29 @@ func (server *Server) sanitizeForeignProviderExecutedToolRows(
originCfg, provider, err := server.resolveModelConfigAndNormalizedProvider(ctx, ownerID, id.UUID)
switch {
case err == nil:
origins[id.UUID] = modelConfigProviderIdentity(originCfg, provider)
origins[id.UUID] = reasoningProvenance{
ProviderIdentity: modelConfigProviderIdentity(originCfg, provider),
Model: originCfg.Model,
}
case modelConfigUnavailable(err):
origins[id.UUID] = ""
origins[id.UUID] = reasoningProvenance{}
default:
return nil, xerrors.Errorf("resolve origin model config %s: %w", id.UUID, err)
}
}
originProvider := func(id uuid.NullUUID) (string, bool) {
identity := origins[id.UUID]
return identity, identity != ""
originOf := func(id uuid.NullUUID) reasoningProvenance {
return origins[id.UUID]
}
sanitized, stats := stripForeignProviderExecutedToolRows(rows, targetIdentity, originProvider)
sanitized, stats := stripForeignProviderStateRows(rows, target, originOf)
if stats != (providerSwitchStripStats{}) {
logger.Debug(ctx, "stripped foreign provider-executed tool history",
logger.Debug(ctx, "stripped foreign provider state from history",
slog.F("phase", "provider_switch"),
slog.F("target_provider_identity", targetIdentity),
slog.F("target_provider_identity", target.ProviderIdentity),
slog.F("target_model", target.Model),
slog.F("removed_tool_calls", stats.RemovedToolCalls),
slog.F("removed_tool_results", stats.RemovedToolResults),
slog.F("removed_reasoning", stats.RemovedReasoning),
slog.F("dropped_messages", stats.DroppedMessages),
)
}
@@ -4,6 +4,8 @@ import (
"encoding/json"
"testing"
"charm.land/fantasy"
fantasyopenai "charm.land/fantasy/providers/openai"
"github.com/google/uuid"
"github.com/sqlc-dev/pqtype"
"github.com/stretchr/testify/require"
@@ -13,23 +15,32 @@ import (
"github.com/coder/coder/v2/codersdk"
)
func TestStripForeignProviderExecutedToolRows(t *testing.T) {
func TestStripForeignProviderStateRows(t *testing.T) {
t.Parallel()
const (
anthropic = "anthropic"
bedrock = "bedrock"
openai = "openai"
)
const model = "model-a"
source := func(identity string) reasoningProvenance {
return reasoningProvenance{ProviderIdentity: identity, Model: model}
}
anthropic := source("anthropic")
bedrock := source("bedrock")
openai := source("openai")
anthropicCfg := uuid.New()
openAICfg := uuid.New()
unknownCfg := uuid.New()
vllmProviderID := uuid.New()
togetherProviderID := uuid.New()
vllm := source(uuid.NewString())
together := source(uuid.NewString())
vllmCfg := uuid.New()
openAIInstance := source(uuid.NewString())
otherOpenAIInstance := source(uuid.NewString())
openAIInstanceOtherModel := reasoningProvenance{ProviderIdentity: openAIInstance.ProviderIdentity, Model: "model-b"}
openAIInstanceCfg := uuid.New()
openAIInstanceOtherCfg := uuid.New()
openAIInstanceOtherModelCfg := uuid.New()
peCall := func(id string) codersdk.ChatMessagePart {
p := codersdk.ChatMessageToolCall(id, "web_search", json.RawMessage(`{"query":"x"}`))
p.ProviderExecuted = true
@@ -44,6 +55,17 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
return codersdk.ChatMessageToolCall(id, "read_file", json.RawMessage(`{}`))
}
text := codersdk.ChatMessageText
encryptedReasoning := "encrypted-blob"
openAIReasoning := chatprompt.PartFromContent(fantasy.ReasoningContent{
ProviderMetadata: fantasy.ProviderMetadata{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{
ItemID: "rs_1",
EncryptedContent: &encryptedReasoning,
Summary: []string{},
Finalized: true,
},
},
})
assistantRow := func(t *testing.T, cfg uuid.UUID, parts ...codersdk.ChatMessagePart) database.ChatMessage {
t.Helper()
@@ -67,20 +89,18 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
}
}
origin := func(providerByConfig map[uuid.UUID]string) func(uuid.NullUUID) (string, bool) {
return func(id uuid.NullUUID) (string, bool) {
if !id.Valid {
return "", false
}
provider, ok := providerByConfig[id.UUID]
return provider, ok
}
}
resolver := origin(map[uuid.UUID]string{
sourceByConfig := map[uuid.UUID]reasoningProvenance{
anthropicCfg: anthropic,
openAICfg: openai,
vllmCfg: vllmProviderID.String(),
})
vllmCfg: vllm,
openAIInstanceCfg: openAIInstance,
openAIInstanceOtherCfg: openAIInstance,
openAIInstanceOtherModelCfg: openAIInstanceOtherModel,
}
resolver := func(id uuid.NullUUID) reasoningProvenance {
return sourceByConfig[id.UUID]
}
partsOf := func(t *testing.T, row database.ChatMessage) []codersdk.ChatMessagePart {
t.Helper()
@@ -95,7 +115,7 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
userRow(t, "hi"),
assistantRow(t, anthropicCfg, peCall("ws"), peResult("ws"), text("done")),
}
got, stats := stripForeignProviderExecutedToolRows(rows, anthropic, resolver)
got, stats := stripForeignProviderStateRows(rows, anthropic, resolver)
require.Equal(t, rows, got)
require.Zero(t, stats)
})
@@ -106,7 +126,7 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
userRow(t, "hi"),
assistantRow(t, anthropicCfg, peCall("ws"), peResult("ws"), text("done")),
}
got, stats := stripForeignProviderExecutedToolRows(rows, bedrock, resolver)
got, stats := stripForeignProviderStateRows(rows, bedrock, resolver)
require.Len(t, got, 2)
require.Equal(t, []codersdk.ChatMessagePart{text("done")}, partsOf(t, got[1]))
require.Equal(t, providerSwitchStripStats{RemovedToolCalls: 1, RemovedToolResults: 1}, stats)
@@ -119,7 +139,7 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
assistantRow(t, anthropicCfg, peCall("ws")),
userRow(t, "again"),
}
got, stats := stripForeignProviderExecutedToolRows(rows, bedrock, resolver)
got, stats := stripForeignProviderStateRows(rows, bedrock, resolver)
require.Len(t, got, 2)
require.Equal(t, database.ChatMessageRoleUser, got[0].Role)
require.Equal(t, database.ChatMessageRoleUser, got[1].Role)
@@ -132,7 +152,7 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
assistantRow(t, openAICfg, peCall("os"), peResult("os"), text("openai")),
assistantRow(t, anthropicCfg, peCall("as"), peResult("as"), text("anthropic")),
}
got, stats := stripForeignProviderExecutedToolRows(rows, anthropic, resolver)
got, stats := stripForeignProviderStateRows(rows, anthropic, resolver)
require.Len(t, got, 2)
require.Equal(t, []codersdk.ChatMessagePart{text("openai")}, partsOf(t, got[0]))
require.Equal(t, rows[1], got[1])
@@ -144,7 +164,7 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
rows := []database.ChatMessage{
assistantRow(t, anthropicCfg, text("hello"), localCall("local")),
}
got, stats := stripForeignProviderExecutedToolRows(rows, bedrock, resolver)
got, stats := stripForeignProviderStateRows(rows, bedrock, resolver)
require.Equal(t, rows, got)
require.Zero(t, stats)
})
@@ -154,7 +174,7 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
rows := []database.ChatMessage{
assistantRow(t, anthropicCfg, peCall("ws"), peResult("ws")),
}
got, stats := stripForeignProviderExecutedToolRows(rows, "", resolver)
got, stats := stripForeignProviderStateRows(rows, reasoningProvenance{}, resolver)
require.Equal(t, rows, got)
require.Zero(t, stats)
})
@@ -164,7 +184,7 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
rows := []database.ChatMessage{
assistantRow(t, unknownCfg, peResult("ws"), text("done")),
}
got, stats := stripForeignProviderExecutedToolRows(rows, bedrock, resolver)
got, stats := stripForeignProviderStateRows(rows, bedrock, resolver)
require.Len(t, got, 1)
require.Equal(t, []codersdk.ChatMessagePart{text("done")}, partsOf(t, got[0]))
require.Equal(t, providerSwitchStripStats{RemovedToolResults: 1}, stats)
@@ -178,7 +198,7 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
Content: pqtype.NullRawMessage{RawMessage: []byte("{not json"), Valid: true},
ContentVersion: chatprompt.ContentVersionV1,
}}
got, stats := stripForeignProviderExecutedToolRows(rows, bedrock, resolver)
got, stats := stripForeignProviderStateRows(rows, bedrock, resolver)
require.Equal(t, rows, got)
require.Zero(t, stats)
})
@@ -189,22 +209,196 @@ func TestStripForeignProviderExecutedToolRows(t *testing.T) {
userRow(t, "hi"),
assistantRow(t, vllmCfg, peCall("ws"), peResult("ws"), text("done")),
}
got, stats := stripForeignProviderExecutedToolRows(rows, togetherProviderID.String(), resolver)
got, stats := stripForeignProviderStateRows(rows, together, resolver)
require.Len(t, got, 2)
require.Equal(t, []codersdk.ChatMessagePart{text("done")}, partsOf(t, got[1]))
require.Equal(t, providerSwitchStripStats{RemovedToolCalls: 1, RemovedToolResults: 1}, stats)
})
t.Run("same instance different config keeps OpenAI reasoning", func(t *testing.T) {
t.Parallel()
rows := []database.ChatMessage{
userRow(t, "hi"),
assistantRow(t, openAIInstanceOtherCfg, openAIReasoning, localCall("local")),
}
got, stats := stripForeignProviderStateRows(rows, openAIInstance, resolver)
require.Equal(t, rows, got)
require.Zero(t, stats)
})
t.Run("foreign instance drops OpenAI reasoning", func(t *testing.T) {
t.Parallel()
legacyReasoning := chatprompt.PartFromContent(fantasy.ReasoningContent{
Text: "summary",
ProviderMetadata: fantasy.ProviderMetadata{
fantasyopenai.Name: &fantasyopenai.ResponsesReasoningMetadata{ItemID: "rs_legacy"},
},
})
rows := []database.ChatMessage{
userRow(t, "hi"),
assistantRow(t, openAIInstanceCfg, openAIReasoning, legacyReasoning, codersdk.ChatMessageReasoning("plain"), text("visible"), localCall("local")),
}
got, stats := stripForeignProviderStateRows(rows, otherOpenAIInstance, resolver)
require.Len(t, got, 2)
require.Equal(t, rows[0], got[0])
require.Equal(t, []codersdk.ChatMessagePart{codersdk.ChatMessageReasoning("plain"), text("visible"), localCall("local")}, partsOf(t, got[1]))
require.Equal(t, providerSwitchStripStats{RemovedReasoning: 2}, stats)
})
t.Run("unknown origin drops OpenAI reasoning", func(t *testing.T) {
t.Parallel()
rows := []database.ChatMessage{
assistantRow(t, unknownCfg, openAIReasoning, text("done")),
}
got, stats := stripForeignProviderStateRows(rows, openAIInstance, resolver)
require.Len(t, got, 1)
require.Equal(t, []codersdk.ChatMessagePart{text("done")}, partsOf(t, got[0]))
require.Equal(t, providerSwitchStripStats{RemovedReasoning: 1}, stats)
})
stampedReasoning := func(source reasoningProvenance) codersdk.ChatMessagePart {
p := openAIReasoning
p.ProviderIdentity = source.ProviderIdentity
p.ProviderModel = source.Model
return p
}
t.Run("stamped foreign reasoning dropped from native row", func(t *testing.T) {
t.Parallel()
rows := []database.ChatMessage{
assistantRow(t, openAIInstanceCfg, stampedReasoning(otherOpenAIInstance), peCall("ws"), text("done")),
}
got, stats := stripForeignProviderStateRows(rows, openAIInstance, resolver)
require.Len(t, got, 1)
require.Equal(t, []codersdk.ChatMessagePart{peCall("ws"), text("done")}, partsOf(t, got[0]))
require.Equal(t, providerSwitchStripStats{RemovedReasoning: 1}, stats)
})
t.Run("stamped native reasoning kept after config moved", func(t *testing.T) {
t.Parallel()
rows := []database.ChatMessage{
assistantRow(t, openAIInstanceCfg, stampedReasoning(otherOpenAIInstance), peCall("ws"), text("done")),
}
got, stats := stripForeignProviderStateRows(rows, otherOpenAIInstance, resolver)
require.Len(t, got, 1)
require.Equal(t, []codersdk.ChatMessagePart{stampedReasoning(otherOpenAIInstance), text("done")}, partsOf(t, got[0]))
require.Equal(t, providerSwitchStripStats{RemovedToolCalls: 1}, stats)
})
t.Run("stamped reasoning from another model dropped", func(t *testing.T) {
t.Parallel()
rows := []database.ChatMessage{
assistantRow(t, openAIInstanceCfg, stampedReasoning(openAIInstanceOtherModel), peCall("ws"), text("done")),
}
got, stats := stripForeignProviderStateRows(rows, openAIInstance, resolver)
require.Len(t, got, 1)
require.Equal(t, []codersdk.ChatMessagePart{peCall("ws"), text("done")}, partsOf(t, got[0]))
require.Equal(t, providerSwitchStripStats{RemovedReasoning: 1}, stats)
})
t.Run("unstamped reasoning from another model on same instance dropped", func(t *testing.T) {
t.Parallel()
rows := []database.ChatMessage{
assistantRow(t, openAIInstanceOtherModelCfg, openAIReasoning, peCall("ws"), text("done")),
}
got, stats := stripForeignProviderStateRows(rows, openAIInstance, resolver)
require.Len(t, got, 1)
require.Equal(t, []codersdk.ChatMessagePart{peCall("ws"), text("done")}, partsOf(t, got[0]))
require.Equal(t, providerSwitchStripStats{RemovedReasoning: 1}, stats)
})
t.Run("unstamped reasoning follows row origin", func(t *testing.T) {
t.Parallel()
rows := []database.ChatMessage{
assistantRow(t, openAIInstanceCfg, stampedReasoning(openAIInstance), openAIReasoning, text("done")),
}
got, stats := stripForeignProviderStateRows(rows, openAIInstance, resolver)
require.Equal(t, rows, got)
require.Zero(t, stats)
got, stats = stripForeignProviderStateRows(rows, otherOpenAIInstance, resolver)
require.Len(t, got, 1)
require.Equal(t, []codersdk.ChatMessagePart{text("done")}, partsOf(t, got[0]))
require.Equal(t, providerSwitchStripStats{RemovedReasoning: 2}, stats)
})
t.Run("same instance keeps provider blocks", func(t *testing.T) {
t.Parallel()
rows := []database.ChatMessage{
userRow(t, "hi"),
assistantRow(t, vllmCfg, peCall("ws"), peResult("ws"), text("done")),
}
got, stats := stripForeignProviderExecutedToolRows(rows, vllmProviderID.String(), resolver)
got, stats := stripForeignProviderStateRows(rows, vllm, resolver)
require.Equal(t, rows, got)
require.Zero(t, stats)
})
stampedFull := func(source reasoningProvenance) codersdk.ChatMessagePart {
p := stampedReasoning(source)
p.ProviderCredential = source.Credential
p.ReasoningEpoch = source.Epoch
return p
}
t.Run("stamped reasoning from another credential dropped", func(t *testing.T) {
t.Parallel()
deployment := openAIInstance
deployment.Credential = reasoningCredentialDeployment
user := openAIInstance
user.Credential = reasoningCredentialUser
rows := []database.ChatMessage{
assistantRow(t, openAIInstanceCfg, stampedFull(deployment), text("deployment")),
assistantRow(t, openAIInstanceCfg, stampedFull(user), text("user")),
}
got, stats := stripForeignProviderStateRows(rows, user, resolver)
require.Len(t, got, 2)
require.Equal(t, []codersdk.ChatMessagePart{text("deployment")}, partsOf(t, got[0]))
require.Equal(t, rows[1], got[1])
require.Equal(t, providerSwitchStripStats{RemovedReasoning: 1}, stats)
})
t.Run("reasoning from an earlier epoch dropped", func(t *testing.T) {
t.Parallel()
current := openAIInstance
current.Epoch = 1
rows := []database.ChatMessage{
assistantRow(t, openAIInstanceCfg, openAIReasoning, text("unstamped")),
assistantRow(t, openAIInstanceCfg, stampedFull(openAIInstance), text("rejected")),
assistantRow(t, openAIInstanceCfg, stampedFull(current), text("current")),
}
got, stats := stripForeignProviderStateRows(rows, current, resolver)
require.Len(t, got, 3)
require.Equal(t, []codersdk.ChatMessagePart{text("unstamped")}, partsOf(t, got[0]))
require.Equal(t, []codersdk.ChatMessagePart{text("rejected")}, partsOf(t, got[1]))
require.Equal(t, rows[2], got[2])
require.Equal(t, providerSwitchStripStats{RemovedReasoning: 2}, stats)
})
t.Run("current epoch is per provider model and credential", func(t *testing.T) {
t.Parallel()
at := func(source reasoningProvenance, credential string, epoch int) reasoningProvenance {
source.Credential = credential
source.Epoch = epoch
return source
}
deployment := at(openAIInstance, reasoningCredentialDeployment, 0)
// A step with no reasoning records its epoch on its first part.
marker := text("no reasoning")
stampReasoningProvenance(&marker, at(openAIInstance, reasoningCredentialDeployment, 3))
rows := []database.ChatMessage{
userRow(t, "hi"),
assistantRow(t, openAIInstanceCfg, stampedFull(at(openAIInstance, reasoningCredentialDeployment, 2)), openAIReasoning),
assistantRow(t, openAIInstanceCfg, marker),
assistantRow(t, openAIInstanceOtherModelCfg, stampedFull(at(openAIInstanceOtherModel, reasoningCredentialDeployment, 4))),
assistantRow(t, openAIInstanceCfg, stampedFull(at(openAIInstance, reasoningCredentialUser, 5))),
assistantRow(t, openAIInstanceCfg, stampedFull(at(otherOpenAIInstance, reasoningCredentialDeployment, 6))),
}
require.Equal(t, 3, currentReasoningEpoch(rows, deployment))
require.Equal(t, 4, currentReasoningEpoch(rows, at(openAIInstanceOtherModel, reasoningCredentialDeployment, 0)))
require.Equal(t, 5, currentReasoningEpoch(rows, at(openAIInstance, reasoningCredentialUser, 0)))
require.Equal(t, 6, currentReasoningEpoch(rows, at(otherOpenAIInstance, reasoningCredentialDeployment, 0)))
require.Zero(t, currentReasoningEpoch(rows, at(vllm, reasoningCredentialDeployment, 0)))
})
}
func TestModelConfigProviderIdentity(t *testing.T) {
@@ -265,8 +459,8 @@ func TestPendingUserSegmentDerivedBeforeProviderSwitchSanitization(t *testing.T)
{Role: database.ChatMessageRoleUser, Content: userContent("pending"), ContentVersion: chatprompt.ContentVersionV1},
}
origin := func(id uuid.NullUUID) (string, bool) {
return "openai", id.Valid
origin := func(uuid.NullUUID) reasoningProvenance {
return reasoningProvenance{ProviderIdentity: "openai"}
}
// Segment first: only the truly unanswered tail is pending, and the
@@ -278,13 +472,13 @@ func TestPendingUserSegmentDerivedBeforeProviderSwitchSanitization(t *testing.T)
// The sanitizer drops the emptied foreign assistant row from the
// head; the answered user row stays in the head regardless, and the
// segment derived above is unaffected.
head, stats := stripForeignProviderExecutedToolRows(rows[:start], "anthropic", origin)
head, stats := stripForeignProviderStateRows(rows[:start], reasoningProvenance{ProviderIdentity: "anthropic"}, origin)
require.Equal(t, 1, stats.DroppedMessages)
require.Len(t, head, 1)
require.Equal(t, database.ChatMessageRoleUser, head[0].Role)
// Deriving the segment after sanitization would find no assistant
// row at all and therefore no pending segment.
sanitizedAll, _ := stripForeignProviderExecutedToolRows(rows, "anthropic", origin)
sanitizedAll, _ := stripForeignProviderStateRows(rows, reasoningProvenance{ProviderIdentity: "anthropic"}, origin)
require.Equal(t, len(sanitizedAll), pendingUserSegmentStart(sanitizedAll))
}
+24
View File
@@ -506,6 +506,26 @@ type ChatMessagePart struct {
// (e.g. Anthropic cache control hints) as raw JSON. Internal
// only: stripped by db2sdk before API responses.
ProviderMetadata json.RawMessage `json:"provider_metadata,omitempty" typescript:"-"`
// ProviderIdentity records the provider that produced a reasoning
// part, because a model config's provider can change after the
// message was written. Internal only: stripped before API responses.
ProviderIdentity string `json:"provider_identity,omitempty" typescript:"-" swaggerignore:"true"`
// ProviderModel records the model that produced a reasoning part,
// because a model config's model can change after the message was
// written. Internal only: stripped before API responses.
ProviderModel string `json:"provider_model,omitempty" typescript:"-" swaggerignore:"true"`
// ProviderCredential records whether a reasoning part was produced
// with the deployment's credential or the user's personal API key,
// because encrypted reasoning only decrypts for the organization
// behind that credential. Internal only: stripped before API responses.
ProviderCredential string `json:"provider_credential,omitempty" typescript:"-" swaggerignore:"true"`
// ReasoningEpoch records how many times the provider had rejected the
// chat's encrypted reasoning from this provider, model, and credential
// before this reasoning part was produced. A step that produced no
// reasoning after a rejection records its epoch, provider, model, and
// credential on its first part instead. Internal only: stripped before
// API responses.
ReasoningEpoch int `json:"reasoning_epoch,omitempty" typescript:"-" swaggerignore:"true"`
// ProviderExecuted indicates the tool call was executed by
// the provider (e.g. Anthropic computer use).
ProviderExecuted bool `json:"provider_executed,omitempty" variants:"tool-call?,tool-result?"`
@@ -596,6 +616,10 @@ type ChatMessagePart struct {
// the advisor streaming callbacks.
func (p *ChatMessagePart) StripInternal() {
p.ProviderMetadata = nil
p.ProviderIdentity = ""
p.ProviderModel = ""
p.ProviderCredential = ""
p.ReasoningEpoch = 0
if p.FileID.Valid {
p.Data = nil
}
+17 -5
View File
@@ -199,14 +199,22 @@ func TestChatMessagePart_StripInternal(t *testing.T) {
t.Run("StripsProviderMetadata", func(t *testing.T) {
t.Parallel()
part := codersdk.ChatMessagePart{
Type: codersdk.ChatMessagePartTypeToolCall,
ToolCallID: "call-1",
ToolName: "some_tool",
Args: json.RawMessage(`{"key":"value"}`),
ProviderMetadata: json.RawMessage(`{"type":"ephemeral"}`),
Type: codersdk.ChatMessagePartTypeToolCall,
ToolCallID: "call-1",
ToolName: "some_tool",
Args: json.RawMessage(`{"key":"value"}`),
ProviderMetadata: json.RawMessage(`{"type":"ephemeral"}`),
ProviderIdentity: "provider-1",
ProviderModel: "model-1",
ProviderCredential: "user",
ReasoningEpoch: 2,
}
part.StripInternal()
assert.Nil(t, part.ProviderMetadata)
assert.Empty(t, part.ProviderIdentity)
assert.Empty(t, part.ProviderModel)
assert.Empty(t, part.ProviderCredential)
assert.Zero(t, part.ReasoningEpoch)
// Public fields preserved.
assert.Equal(t, codersdk.ChatMessagePartTypeToolCall, part.Type)
assert.Equal(t, "call-1", part.ToolCallID)
@@ -303,6 +311,10 @@ func TestChatMessagePartVariantTags(t *testing.T) {
excludedFields := map[string]string{
"type": "discriminant, added automatically by codegen",
"provider_metadata": "internal only, stripped by db2sdk before API responses",
"provider_identity": "internal only, stripped by db2sdk before API responses",
"provider_model": "internal only, stripped by db2sdk before API responses",
"provider_credential": "internal only, stripped by db2sdk before API responses",
"reasoning_epoch": "internal only, stripped by db2sdk before API responses",
"context_file_content": "internal only, stripped before API responses (typescript:\"-\")",
"context_file_os": "internal only, used during prompt expansion (typescript:\"-\")",
"context_file_directory": "internal only, used during prompt expansion (typescript:\"-\")",
+6 -2
View File
@@ -111,8 +111,12 @@ replace github.com/spf13/afero => github.com/aslilac/afero v0.0.0-20250403163713
// with no results so chatd keeps them in saved history.
// 13) coder/fantasy#66, capture OpenAI Responses message phase on text
// parts and replay it on assistant input messages in both store modes.
// See: https://github.com/coder/fantasy/commits/351055280157
replace charm.land/fantasy => github.com/coder/fantasy v0.0.0-20261009165329-351055280157
// 14) coder/fantasy#63, capture finalized OpenAI Responses reasoning
// (item ID, encrypted content, summary) and replay it in full
// regardless of store. Skip hosted search references from unstored
// source responses.
// See: https://github.com/coder/fantasy/commits/6ba868142390
replace charm.land/fantasy => github.com/coder/fantasy v0.0.0-20261011102151-6ba868142390
// coder/coder uses a fork of charmbracelet's fork of the Anthropic Go SDK
// with performance improvements and Bedrock header cleanup.
+2 -2
View File
@@ -333,8 +333,8 @@ github.com/coder/bubbletea v1.2.2-0.20241212190825-007a1cdb2c41 h1:SBN/DA63+ZHwu
github.com/coder/bubbletea v1.2.2-0.20241212190825-007a1cdb2c41/go.mod h1:I9ULxr64UaOSUv7hcb3nX4kowodJCVS7vt7VVJk/kW4=
github.com/coder/clistat v1.2.1 h1:P9/10njXMyj5cWzIU5wkRsSy5LVQH49+tcGMsAgWX0w=
github.com/coder/clistat v1.2.1/go.mod h1:m7SC0uj88eEERgvF8Kn6+w6XF21BeSr+15f7GoLAw0A=
github.com/coder/fantasy v0.0.0-20261009165329-351055280157 h1:t1DbZyWlLOPISsLzcFFnKfvSToWpXFIL1pkpFovlNh0=
github.com/coder/fantasy v0.0.0-20261009165329-351055280157/go.mod h1:Hd5DBPecZ+CGc8O9bFAV03S7jbEmqd9xXVmjKCzpD7g=
github.com/coder/fantasy v0.0.0-20261011102151-6ba868142390 h1:l0MyL3NG85LofRJj7pvXaPINmUqGI89dTQXxg2fJMNs=
github.com/coder/fantasy v0.0.0-20261011102151-6ba868142390/go.mod h1:Hd5DBPecZ+CGc8O9bFAV03S7jbEmqd9xXVmjKCzpD7g=
github.com/coder/flog v1.1.0 h1:kbAes1ai8fIS5OeV+QAnKBQE22ty1jRF/mcAwHpLBa4=
github.com/coder/flog v1.1.0/go.mod h1:UQlQvrkJBvnRGo69Le8E24Tcl5SJleAAR7gYEHzAmdQ=
github.com/coder/go-httpstat v0.0.0-20230801153223-321c88088322 h1:m0lPZjlQ7vdVpRBPKfYIFlmgevoTkBxB10wv6l2gOaU=