feat(providers): prompt caching capability + usage-based cache pricing (#5922)

* improvement(providers): validation pass, and stream tool loop improvements

* remove deploy options correctly

* fix

* feat(providers): prompt caching capability and usage-based cache pricing

Replace the arbitrary cached-rate heuristic with a single cache-aware pricing
function, and add prompt caching as an opt-in capability for Anthropic.

Pricing: priceModelUsage in cost-policy.ts is now the only place cache
arithmetic happens. Provider adapters normalize their wire shape into
ModelUsage (input always excludes cache buckets); the pricing function never
branches on provider. This removes five divergent behaviors, including the
!!request.context heuristic that gave Router and Evaluator an unearned 10x
input discount, and the overwrite that silently billed Anthropic cache reads
and writes at zero. Also parses OpenAI cache_write_tokens, previously ignored.

Caching: Anthropic gets a capability-gated advanced switch that places
cache_control on the last tool and last system block; system is now always a
TextBlockParam array. OpenAI gets a stable per-block prompt_cache_key with no
UI, since its caching is automatic.

* fix(providers): route OpenAI and Gemini block cost through cache-aware pricing

Cache-aware pricing only reached trace segments. The billable block cost still
called calculateCost on the cache-inclusive prompt total, so OpenAI cache hits
and Gemini implicit-cache hits were charged at the full input rate and GPT-5.6+
cache writes went unbilled.

Both providers now accumulate cache buckets and price through priceModelUsage,
matching the Anthropic token convention where input excludes cache reads and
writes. Cached counts are clamped to the prompt total so an over-reporting
payload cannot bill more input than the request contained.

* fix(streaming): redact tool payloads on selected outputs in public chat

Redaction only ran on the empty-selection branch, but a deployment almost
always selects outputs, so it was dead in the case it exists for. Selecting
toolCalls streamed the raw arguments and results to a public chat client in a
chunk frame, and providerTiming carried thinking content the same way.

Both paths now extract from the sanitized block output rather than the raw log:
the streamed selected output, which is the reachable vector, and the final
envelope. Sanitizing the source rather than per selected path means a newly
selectable field cannot reopen the hole.

* refactor(providers): drop unreachable billing fallbacks

Every provider pricing helper took a policy parameter no caller passed. Worse
than dead: passing one would have double-applied the margin the central layer
already applies. Removed, so providers can only price at list.

Also removed guards that cannot fire. The central fallback normalized cache
buckets no provider can reach it with (all three that report cache usage price
themselves) and did so at a 1x write multiplier no vendor charges.
priceModelUsage re-validated token counts the adapter had already clamped, and
applyModelCostPolicy defaulted a required total field.

Validation now happens once, in the adapter that parses the vendor payload and
is the only layer that knows cache buckets are a subset of the prompt total.
This commit is contained in:
Vikhyath Mondreti
2026-07-24 15:46:11 -07:00
committed by GitHub
parent 290e52c3f3
commit 17d77795b4
168 changed files with 31086 additions and 5220 deletions
@@ -449,6 +449,7 @@ export const schemaMock = {
allowedEmails: 'allowedEmails',
outputConfigs: 'outputConfigs',
includeThinking: 'includeThinking',
includeToolCalls: 'includeToolCalls',
archivedAt: 'archivedAt',
createdAt: 'createdAt',
updatedAt: 'updatedAt',