songkuan-zheng 17fd745898 Normalize usage token conventions per source ...

normalizeUsageInputTokens decides whether input_tokens already includes the
cached prefix, and it asked the upstream provider protocol first. Both usage
call sites merge the billing headers and the response body into one snapshot
and then normalize that merge once — but on a translated response the two
sources use different conventions:

  - x-gateway-billing-* headers restate the upstream provider's own counters
    verbatim, so for an OpenAI-compatible upstream they are cache-inclusive.
  - The response body is whatever the gateway emitted. An Anthropic body is
    cache-exclusive regardless of what it was translated from.

So an Anthropic response served from an OpenAI-compatible upstream had the
body's already-excluded prefix subtracted a second time. Non-streaming
responses were unaffected by luck: the headers win the merge and are
inclusive, making the subtraction correct. Streaming responses carry no
billing headers at all, leaving the body alone to be over-subtracted and then
clamped by Math.max(0, ...) — on a long cached conversation input_tokens
reports 0 on nearly every turn, and the derived cache ratio is pinned at
100%.

Tag each source with a UsageConventionSource and normalize the two separately
before merging them. Reordering the precedence instead is not sufficient: it
moves the defect onto the non-streaming path, which depends on the headers
being reduced.

Raw-trace updates carry no provider protocol, so their billing headers keep
falling back to the request path; only the body side changes there.

2026-08-17 05:21:30 +00:00

..

usage-normalization.test.mjs

Normalize usage token conventions per source

2026-08-17 05:21:30 +00:00