mirror of
https://github.com/coder/coder.git
synced 2026-09-21 20:51:01 +08:00
Yevhenii Shcherbina
b6fcb9a30a
Implements https://linear.app/codercom/issue/AIGOV-286/add-interception-cost-calculation-to-aibridge-token-usages Adds spend attribution to AI Gateway. After the upstream response, each token-usage record now captures the user's effective group, the per-token prices in effect at that moment, and a computed cost — so spend is recorded as an immutable, point-in-time snapshot. Concretely, `aibridge_token_usages` gains `effective_group_id`, `input_price_micros`, `output_price_micros`, `cache_read_price_micros`, `cache_write_price_micros`, and `cost_micros`. When a usage record is written, the effective group is resolved (per-user override, else the deployment budget policy), the `(provider, model)` price is looked up and snapshotted onto the row, and cost is computed from the provider-reported token counts. A model that isn't in the price table records its tokens with a `NULL` cost; any *other* resolution failure fails the write, so a `NULL` cost unambiguously means "model not priced" rather than "lookup errored." All values are stored in micro-units (1 unit = 1,000,000 micro-units; Phase 1 assumes USD, so 1 micro-unit = $0.000001). Prices are quoted per million tokens. This also grants the AI Bridge RBAC subject `read` on `ai_model_prices` (the per-interception price lookup needs it; it previously only had `update` for the startup seeder). ## Cost precision Cost is computed per token category as `tokens × price / 1_000_000` with integer division, then the four categories are summed. The division is done **per category** (not once over the summed numerator) on purpose: it keeps the per-category line items summing exactly to the stored total — no "the parts don't add up to the whole" in reporting). Integer division truncates sub-micro-unit fractions. For example, a cheap model at $0.10 per million tokens is a price of `100_000`; 9 tokens cost `9 × 100_000 / 1_000_000 = 900_000 / 1_000_000 = 0` (the true 0.9 micro-units floors to 0). At real list prices this rarely bites — $3/M input is a price of `3_000_000`, so even a single token is 3 micro-units. The per-record under-count is bounded below 1 micro-unit per category, so under $0.000004 total across the four categories, which is acceptable for list-price-based cost approximation. ## Overflow safety `cost_micros` is a `BIGINT` (int64), and the largest intermediate value is a single category's `tokens × price` before division. int64's ceiling is ≈ `9.223e18`. - At a steep $75/M model (price `75_000_000`), overflow would require ~123 billion tokens in one response: `123e9 × 75e6 = 9.225e18`, just over the limit. `122e9` stays under at `9.15e18`. - A realistically maxed-out Opus 4.8 response (≈1M input + 128K output at list prices) costs about $15, with a numerator around `1.5e13` — roughly six orders of magnitude below the ceiling. So overflow is unreachable from real token counts. ### Multi-currency support In the future, we may encounter issues with multi-currency support, especially when dealing with currencies that have very large exchange rates relative to USD, for example: IRR: ~1,300,000 IRR ≈ 1 USD VND: ~26,000 VND ≈ 1 USD For currencies with such large denominations, numeric overflow is technically possible, considering that we have only about six orders of magnitude of headroom before reaching the limit (see above). ## `effective_group_id` has no foreign key `effective_group_id` records the group a spend was attributed to, as an immutable historical fact. It is intentionally **not** a foreign key, so the record survives deletion of the group. Alternatives were considered and rejected: - **`ON DELETE SET NULL`** would mutate an "immutable" record — deleting a group silently erases that interception's attribution and under-counts the group's historical spend. - **`RESTRICT` / `NO ACTION`** would block group deletion entirely (groups are hard-deleted). - **`CASCADE`** would delete spend history when a group is deleted — the worst outcome for an audit record. There is also no insert-time check that the group still exists: the id comes from a budget that was just resolved, meaning it was valid at some point. ## Open question: group name snapshotting Should we also snapshot the group *name* onto each record? Two options: - **Denormalize it now** — readable in historical reports even after a group is deleted, but the snapshot can drift from the current name on rename, raising a "show point-in-time vs. current name" question. - **Postpone until needed** — it's a purely additive column later, and the name is display-only (not correctness-bearing like the price). The cost: names of groups deleted before the column is added can't be backfilled. Leaning toward postponing until a concrete reporting need settles the drift question.