Vertex AI's Anthropic endpoint rejects unknown anthropic-beta tokens with
HTTP 400. buildUpstreamRequestAnthropicVertex forwarded the client header
verbatim via the allowedHeaders whitelist, so recent Claude Code CLIs that
send advisor-tool-2026-03-01, prompt-caching-scope-2026-01-05,
redact-thinking-2026-02-12 and thinking-token-count-2026-05-13 broke every
Vertex service_account request, even though plain account-test requests passed.
This is the only upstream builder that bypassed beta filtering: the
OAuth/API-key path uses computeFinalAnthropicBeta and the Bedrock path uses
filterBedrockBetaTokens. Close the gap with a Vertex-specific whitelist
(vertexSupportedBetaTokens) mirroring bedrockSupportedBetaTokens, plus the
existing BetaPolicy block check:
- evaluateBetaPolicy block check (symmetric to resolveBedrockBetaTokensForRequest)
- filterVertexBetaTokens strips policy-filtered + non-whitelisted tokens
- body context_management sanitize now keys on the final beta, not the raw client value
- overwrite the anthropic-beta header after the whitelist copy loop with the final value
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a use-it-or-lose-it scheduling strategy: prefer accounts whose
session window resets soonest, so near-reset accounts get drained first
instead of accounts whose reset is still far away.
Both schedulers, opt-in, default behavior unchanged:
- Anthropic (gateway_service.go): new GatewaySchedulingConfig
.PreferSoonestReset flag. When on, the layered load-aware selection
inserts a filterBySoonestReset stage (priority -> soonest-reset ->
load -> LRU). Accounts with no active SessionWindowEnd are treated as
lowest priority; ties fall through to LRU.
- OpenAI/Codex (openai_account_scheduler.go): new "reset" score weight
in GatewayOpenAIWSSchedulerScoreWeights. Soonest-reset accounts score
higher; weight defaults to 0 (no effect).
SessionWindowEnd (upstream 5h/quota ResetsAt) is already carried in the
scheduler snapshot, so no snapshot changes are needed.
Documented in deploy/config.example.yaml. Adds unit tests for the
Anthropic filter and the OpenAI reset factor.
DeriveUpstreamEndpoint maps every OpenAI-platform request to /v1/responses, but
API-key accounts whose upstream only speaks Chat Completions
(!ShouldUseResponsesAPI) are forwarded directly to /v1/chat/completions. The
messages, responses and cyber-policy recording sites derived the endpoint via the
bare GetUpstreamEndpoint, so usage/ops records mislabeled those requests as
/v1/responses. Generalize the existing resolveRawCCUpstreamEndpoint into
resolveOpenAIUpstreamEndpoint and use it at every OpenAI recording site, matching
the already-correct chat-completions client path.
Add the :Z (private unshared) SELinux label to all bind-mounted
directories in docker-compose.local.yml and docker-compose.dev.yml.
On systems with SELinux in Enforcing mode (e.g. Fedora, RHEL, CentOS),
containers using bind mounts are denied access to host directories
because the default 'user_home_t' context is not accessible to
container processes. The :Z label tells the container runtime to
relabel the mount point with 'container_file_t' so the container
can read/write it.
Named volumes in docker-compose.yml are not affected because the
runtime already handles their labels automatically.
Fixes permission-denied errors on:
- ./data:/app/data
- ./postgres_data:/var/lib/postgresql/data
- ./redis_data:/data
When an Anthropic upstream returned HTTP 200 but then emitted an SSE
`event: error` frame (overloaded_error / rate_limit_error / api_error /
etc.), Forward's stream branch matched on `err.Error() == "have error in
stream"` and returned `UpstreamFailoverError{StatusCode: 403}` with no
ResponseBody. That dropped three pieces of evidence:
- handleFailoverExhausted → ExtractUpstreamErrorMessage(nil) = "" →
ops_error_logs.upstream_error_message was empty.
- errorPassthroughService.MatchRule(_, 403, nil) could only match rules
without keywords, so keyword-based passthrough rules silently never
fired.
- upstream_errors carried no stream_error record, leaving ops looking at
a generic 403 with no clue whether the upstream was throttled,
overloaded, or rejecting the request. ping-during-slot-wait amplified
this by skipping failover (writerSizeBeforeForward guard), so 403s
ballooned in the ops view well past the upstream's actual rate.
Fix:
- Introduce *sseStreamErrorEventError that carries the SSE data line.
Error() still returns "have error in stream" so existing log searches
keep working.
- Forward extracts via errors.As, appends an OpsUpstreamErrorEvent
(kind="stream_error", with the sanitized message and a truncated raw
body honoring LogUpstreamErrorBody*), and returns
UpstreamFailoverError{StatusCode: 403, ResponseBody: rawJSON}.
StatusCode 403 is preserved verbatim: mapUpstreamError, failover
decisions (shouldFailoverUpstreamError(403)=true), client-visible message,
RetryableOnSameAccount, and rateLimitService side-effects (this path
already didn't invoke them) all match prior behavior. OAuth and API Key
accounts share this path; the API-Key passthrough branch is independent
and already forwards SSE error frames untouched, so it's unaffected.
Adds four unit tests: typed-error contract + RawData, empty data line,
event:error after partial stream output (streamStarted=true), and
non-JSON data line.
The function signature was changed to require a mappedModel parameter
for protocol-aware thinking-block filtering, but this test call site
was not updated. Without the fix, the unit test build fails on CI.
The gateway's thinking-block handling was designed for Anthropic's strict
semantics (drop blocks with missing/invalid signature), but third-party
Anthropic-compatible upstreams have INVERTED semantics:
* DeepSeek `/anthropic`, Kimi `/coding`, GLM, Moonshot, qwen-*-thinking
require ALL historical thinking blocks to round-trip verbatim.
* Stripping any of them produces:
400 "The content[].thinking in the thinking mode must be passed back
to the API"
Without this fix, every multi-turn request from a thinking-capable client
(Claude Code, pi, etc.) to such upstreams loses its thinking blocks and
fails. This becomes especially painful when an account's model_mapping
maps `claude-sonnet-4-6 → deepseek-v4-pro` — `reqModel` looks Anthropic
but the upstream contract is the opposite.
Approach
--------
Branch all thinking-block transforms by the *mapped* model id (after
account model_mapping is applied), classifying into three families:
* `anthropic-strict` claude-/opus-/sonnet-/haiku- → existing behaviour
* `passback-required` deepseek-/kimi-/moonshot-/glm-/ → preserve verbatim
qwen-*-thinking
* `unknown` other models → conservative
(preserve, no retry)
Affected entry points (all guarded):
* Pre-filter on outbound: `FilterThinkingBlocks`
Previously dropped blocks with missing/invalid signature; now skips
entirely for non-strict families. Pre-filter is needed because the
post-error retry path can run out of budget on long conversations
(maxRetryElapsed = 10 s).
* 400 retry rectifier: `FilterThinkingBlocksForRetry`
Disables top-level thinking and converts thinking → text. Now skips
for passback-required (those 400s aren't signature errors and any
transformation breaks the round-trip contract).
* 400 retry rectifier (tools): `FilterSignatureSensitiveBlocksForRetry`
Same family-aware short-circuit.
* 400 detector: `shouldRectifySignatureError`
Returns false for passback-required, so the retry path doesn't even
fire.
Tests
-----
* `thinking_protocol_test.go` — classifier across all known vendor
prefixes plus edge cases (empty, case, qwen non-thinking).
* `thinking_protocol_filter_integration_test.go` — locks in that the
three filter entry points return the body byte-for-byte unchanged
when the model id is passback-required or unknown, and still strip
invalid blocks for anthropic-strict.
This PR supersedes #1350 (which only added the pre-filter without the
upstream-family awareness, and would have made third-party upstreams
worse). Once merged, please close#1350.
Reference issues:
- NousResearch/hermes-agent#16748 — DeepSeek /anthropic strip behaviour
- NousResearch/hermes-agent#15700 — DeepSeek thinking:disabled requirement
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pi maps defaultThinkingLevel 'xhigh' → 'max' for deepseek-v4-pro via
thinkingLevelMap, but normalizeOpenAIReasoningEffort did not recognize
'max', causing all reasoning_effort to be dropped (0% fill rate since
traffic switched from claude to deepseek on May 1).
Add 'max' → 'xhigh' mapping to match Claude's NormalizeClaudeOutputEffort
behavior.
Address copilot-pull-request-reviewer feedback on #2157:
- billing_service_test.go: extend TestGetFallbackPricing_FamilyMatching
with optional expectedOutput / expectedCacheRead fields and assert
full Input/Output/CacheRead pricing for all 4 DeepSeek cases
(v4-pro, v4-flash, deepseek-chat→flash, deepseek-reasoner→flash),
preventing silent regression to 0 output/cache cost.
- billing_service.go: rewrite the DeepSeek block comment to explicitly
describe its scope (V4 Pro/Flash + chat/reasoner aliases, no
unknown-deepseek fallback) and tighten the OpenAI comment to make
it unambiguous that it only describes the OpenAI/Codex branch
immediately below it.
Three independent CI blockers landed on main from concurrent PR merges:
- openai_quota_service.go (introduced by b8169492): const block spacing
not gofmt-compliant + trailing blank line. golangci-lint v2.9 flagged it
on every push after the merge.
- openai_images_failover_test.go (introduced by PR #3155, da30c599):
NewOpenAIGatewayHandler call missing the opsService argument added by
PR #3230 (b62b573f). Test was authored before #3230 and merged without
rebase, causing "not enough arguments" compile error.
- account_quota_reset_test.go: TestIsFixedDailyPeriodExpired_NotExpired
and TestIsFixedWeeklyPeriodExpired_NotExpired used time.Now()-1min as
periodStart, which crosses the 09:00 UTC reset boundary when CI runs in
the 09:00:00-09:00:59 window. Anchoring periodStart to today's 12:00
UTC removes the race.
Adds an admin-side action that mirrors the Codex Desktop "rate-limit reset"
flow against chatgpt.com upstream for OpenAI OAuth accounts.
Backend
- OpenAIQuotaService.QueryUsage / ResetCredit hit /wham/usage and
/wham/rate-limit-reset-credits/consume with the Codex Desktop header set,
reusing OpenAITokenProvider for refreshed tokens and PrivacyClientFactory
for the impersonated Chrome TLS fingerprint.
- Honors the account's configured proxy by reading the eager-loaded
account.Proxy directly (falls back to proxyRepo only when missing).
- GET /api/v1/admin/openai/accounts/:id/quota
POST /api/v1/admin/openai/accounts/:id/reset-quota
- Wire DI for the new service + handler dependency.
Frontend
- OpenAIQuotaResetCell renders a single action row in AccountUsageCell's
OpenAI section: the existing local "查询" (active sampling) is injected
via #pre-actions, alongside a "次数 N" button that doubles as the
upstream query trigger and the available-credit indicator, and a "重置"
button that consumes one credit.
- No duplicate 5h/7d window display; the local UsageProgressBar owns those
bars to avoid confusion.
#3255 introduced payload-aware dedup_key (sha256 over event_type, account_id,
group_id, payload_json). enqueueSchedulerOutbox checks "if payload != nil"
before json.Marshal — but Go interface containing a typed-nil map (returned
by buildSchedulerGroupPayload(empty)) is NOT == nil at the interface level.
So an ungrouped account's account_changed event went through this path:
payload := buildSchedulerGroupPayload(account.GroupIDs) // typed-nil map
enqueueSchedulerOutbox(..., payload) // interface != nil
→ json.Marshal(typedNilMap) = "null"
→ dedup_key hash = sha256(... + "null")
While other call sites pass literal nil:
enqueueSchedulerOutbox(..., nil) // interface == nil
→ payloadJSON stays empty
→ dedup_key hash = sha256(... + "")
The two dedup_keys differ for what should be the same logical event,
silently degrading dedup effectiveness in bursts on ungrouped accounts.
Fix: change buildSchedulerGroupPayload return type from map[string]any to
any so empty input returns true untyped-nil. All call sites pass the
result straight to enqueueSchedulerOutbox(payload any) — no inspection,
no breakage.
Adds regression test TestEnqueueSchedulerOutbox_UngroupedAccountDedupesWithLiteralNilPayload
asserting (1) typed-nil regression doesn't sneak back, (2) dedup_key for
empty-groups payload matches the literal-nil-payload key.
