mirror of
https://github.com/coder/coder.git
synced 2026-09-21 20:51:01 +08:00
4b7494be72
Closes CODAGT-835 ## Summary `chat_messages.runtime_ms` becomes the billing source of truth for Coder Agents runtime (summed hourly by #27312), but it was built for debugging: the June refactor (#26270) silently stopped recording tool-step runtime, compaction was never measured, and interrupted turns lost their partial runtime entirely. This PR defines the billable metric, closes the paths that dropped it, and documents the definition where the data lives. ## The billable definition **`runtime_ms` is the wall-clock duration of the model invocation that produced the persisted message content**, measured from just before the provider stream opens until it is fully consumed. What counts: - Assistant generation steps, in top-level and sub-agent chats (sub-agents are ordinary chats on the same generation path). - Compaction summarization calls, persisted on the compaction assistant message (**new**). - Interrupted attempts: the message-part episode's lifetime is persisted on the partial assistant message committed by `FinishInterruption`, so partial generation time survives interruption (**new**; measured via a new `Buffer.EpisodeDuration`, which works even though the generation goroutine and the interrupt task are different tasks). What deliberately does not count (each is documented in code and docs): - **Local tool execution.** Tool wall time includes idle waits, most importantly `wait_agent` polling a sub-agent chat that already bills its own model invocations; billing the batch would double count, and excluding one tool from a concurrent batch's wall time is ill-defined. Pre-refactor instrumentation did include tool time; this makes the exclusion an explicit product definition instead of a silent regression. - **Failed model calls whose output is discarded** (retried attempts, terminal errors, content-filter refusals). They persist no content, so they bill nothing; billing errs toward undercounting. Notably a stream-silence timeout can burn 10 idle minutes before a retry, which should not be billable "active generation". If product later wants failed attempts billed, that needs a place to persist runtime on error turns (`FinishError` inserts no rows today) and is a deliberate follow-up, not instrumentation drift. - **Ancillary calls that produce no chat messages** (title generation, advisor, turn summaries) and all idle/parked time (`requires_action`, queueing). The definition is documented as `COMMENT ON COLUMN chat_messages.runtime_ms` (migration 000551, surfacing as a Go doc comment on `ChatMessage.RuntimeMs`), on `chatloop.PersistedStep.Runtime`, in the chatd architecture doc, and in the Spend Management docs page. ## Index for the hourly scan None needed: `GetTotalChatMessageRuntimeMsInRange` (#27312) filters an hour-wide `created_at` range, which the existing `idx_chat_messages_created_at` b-tree already serves; the residual `runtime_ms IS NOT NULL` filter applies to one hour of rows. A partial index would add permanent write amplification for a query that runs once an hour. > [!NOTE] > Migration 000551 is also claimed by #27312; whichever merges second renumbers via `fix_migration_numbers.sh`. ## Tests - End-to-end: the existing full-server generation test now asserts `RuntimeMs.Valid` on the committed assistant row (it previously read `.Int64` without checking `.Valid`, so it passed on NULL). - Interrupted turn: full task-level test (real DB, mock clock) asserting the partial assistant message persists the attempt's runtime. - Errored stream: asserts a failed invocation yields no step and no runtime. - Tool-using turn: asserts runtime lands on the assistant row only and tool rows stay NULL. - Compaction: asserts the summarization call duration is recorded and lands on the compaction assistant message only. - `messagepartbuffer.EpisodeDuration` unit coverage. Blocks: CODAGT-843 (B3), CODAGT-838 (D8). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Hugo Dutka <hugo@coder.com>
85 lines
3.4 KiB
Markdown
85 lines
3.4 KiB
Markdown
# Usage Data Reporting
|
|
|
|
[AI Governance](./ai-governance.md) requires reporting usage data to Tallyman, a Coder-managed server for billing and reporting purposes. Coder only captures and sends the following information, related to your deployment ID:
|
|
|
|
- number of agent workspace builds consumed
|
|
- number of AI Governance seats consumed
|
|
- total Coder Agent runtime consumed per hour, in milliseconds
|
|
|
|
No user-identifiable information or additional metrics are sent to Tallyman. This information is also shared with [Metronome](https://metronome.com), a Stripe product and Coder partner for usage-based billing and reporting.
|
|
|
|
To send usage data, your Coder deployment must be able to make outbound HTTPS requests to `https://tallyman-prod.coder.com`. Usage data is sent approximately every 17 minutes and can be monitored via `coderd` logs.
|
|
|
|
Example of a successful request (requires debug logging enabled [`CODER_LOG_FILTER=.*`](../reference/cli/server.md#-l---log-filter)):
|
|
|
|
```sh
|
|
[debu] published usage events to tallyman accepted=5 rejected=0
|
|
```
|
|
|
|
Example of a request payload:
|
|
|
|
```txt
|
|
POST /api/v1/events/ingest HTTP/1.1
|
|
Host: tallyman-prod.coder.com
|
|
Content-Type: application/json
|
|
Coder-License-Key: <license-jwt> # your license JWT for verification
|
|
Coder-Deployment-ID: 8a4e92f1-3b7c-4d5e-9f12-abc123def456 # your deployment ID
|
|
|
|
{
|
|
"events": [
|
|
{
|
|
"id": "550e8400-e29b-41d4-a716-446655440000", # unique event ID generated by Coder
|
|
"event_type": "dc_managed_agents_v1", # aka. agent workspace builds
|
|
"event_data": {
|
|
"count": 1
|
|
},
|
|
"created_at": "2025-01-15T14:30:00Z"
|
|
},
|
|
{
|
|
"id": "hb_agent_runtime_v1:2025-01-15_14:00:00", # unique event ID generated by Coder
|
|
"event_type": "hb_agent_runtime_v1", # aka. Coder Agent runtime
|
|
"event_data": {
|
|
"runtime_ms": 3600000
|
|
},
|
|
"created_at": "2025-01-15T14:00:00Z" # start of the hour the runtime was consumed in
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Example of a failed request (e.g. Tallyman Server is blocked by your network):
|
|
|
|
```sh
|
|
[warn] failed to send publish request to tallyman count=5 error="Post \"https://tallyman-prod.coder.com/api/v1/events/ingest\": dial tcp: lookup tallyman-prod.coder.com: no such host"
|
|
```
|
|
|
|
> [!NOTE]
|
|
> Air-gapped deployments and/or those with legal restrictions around usage reporting can [contact us](https://coder.com/contact) to discuss alternative methods.
|
|
|
|
## Agent runtime measurement
|
|
|
|
Total Coder Agent runtime is summed from per-message generation time
|
|
(`runtime_ms` on chat messages).
|
|
|
|
A message's runtime is the wall-clock duration of the model invocation that
|
|
produced its content, measured from just before the request to the model
|
|
provider opens until the response is fully consumed.
|
|
|
|
What counts:
|
|
|
|
- Assistant generation steps, in both top-level chats and sub-agent chats.
|
|
- Context compaction (summarization) model calls.
|
|
- Interrupted generation: the time streamed before the interrupt is kept on
|
|
the partial assistant message.
|
|
|
|
What does not count:
|
|
|
|
- Local tool execution, including waiting on sub-agents. A sub-agent is its
|
|
own chat and records its own model invocations, so counting the parent's
|
|
wait would double count.
|
|
- Idle time: chats waiting for user input or external tool results.
|
|
- Failed model calls whose output was discarded. Retried and errored
|
|
attempts persist no content, so they record no runtime.
|
|
- Ancillary model calls that produce no chat messages, such as title
|
|
generation.
|