6 Commits

Include renames

Author	SHA1	Message	Date
3 people	4b7494be72	feat: harden chat generation runtime instrumentation for billing (#27451) ... Closes CODAGT-835 ## Summary `chat_messages.runtime_ms` becomes the billing source of truth for Coder Agents runtime (summed hourly by #27312), but it was built for debugging: the June refactor (#26270) silently stopped recording tool-step runtime, compaction was never measured, and interrupted turns lost their partial runtime entirely. This PR defines the billable metric, closes the paths that dropped it, and documents the definition where the data lives. ## The billable definition **`runtime_ms` is the wall-clock duration of the model invocation that produced the persisted message content**, measured from just before the provider stream opens until it is fully consumed. What counts: - Assistant generation steps, in top-level and sub-agent chats (sub-agents are ordinary chats on the same generation path). - Compaction summarization calls, persisted on the compaction assistant message (**new**). - Interrupted attempts: the message-part episode's lifetime is persisted on the partial assistant message committed by `FinishInterruption`, so partial generation time survives interruption (**new**; measured via a new `Buffer.EpisodeDuration`, which works even though the generation goroutine and the interrupt task are different tasks). What deliberately does not count (each is documented in code and docs): - **Local tool execution.** Tool wall time includes idle waits, most importantly `wait_agent` polling a sub-agent chat that already bills its own model invocations; billing the batch would double count, and excluding one tool from a concurrent batch's wall time is ill-defined. Pre-refactor instrumentation did include tool time; this makes the exclusion an explicit product definition instead of a silent regression. - **Failed model calls whose output is discarded** (retried attempts, terminal errors, content-filter refusals). They persist no content, so they bill nothing; billing errs toward undercounting. Notably a stream-silence timeout can burn 10 idle minutes before a retry, which should not be billable "active generation". If product later wants failed attempts billed, that needs a place to persist runtime on error turns (`FinishError` inserts no rows today) and is a deliberate follow-up, not instrumentation drift. - **Ancillary calls that produce no chat messages** (title generation, advisor, turn summaries) and all idle/parked time (`requires_action`, queueing). The definition is documented as `COMMENT ON COLUMN chat_messages.runtime_ms` (migration 000551, surfacing as a Go doc comment on `ChatMessage.RuntimeMs`), on `chatloop.PersistedStep.Runtime`, in the chatd architecture doc, and in the Spend Management docs page. ## Index for the hourly scan None needed: `GetTotalChatMessageRuntimeMsInRange` (#27312) filters an hour-wide `created_at` range, which the existing `idx_chat_messages_created_at` b-tree already serves; the residual `runtime_ms IS NOT NULL` filter applies to one hour of rows. A partial index would add permanent write amplification for a query that runs once an hour. > [!NOTE] > Migration 000551 is also claimed by #27312; whichever merges second renumbers via `fix_migration_numbers.sh`. ## Tests - End-to-end: the existing full-server generation test now asserts `RuntimeMs.Valid` on the committed assistant row (it previously read `.Int64` without checking `.Valid`, so it passed on NULL). - Interrupted turn: full task-level test (real DB, mock clock) asserting the partial assistant message persists the attempt's runtime. - Errored stream: asserts a failed invocation yields no step and no runtime. - Tool-using turn: asserts runtime lands on the assistant row only and tool rows stay NULL. - Compaction: asserts the summarization call duration is recorded and lands on the compaction assistant message only. - `messagepartbuffer.EpisodeDuration` unit coverage. Blocks: CODAGT-843 (B3), CODAGT-838 (D8). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Hugo Dutka <hugo@coder.com>	2026-08-06 16:09:38 +07:00
Jaayden Halko	54d5eb7ec2	feat: add hourly hb_agent_runtime_v1 usage events for Coder Agent runtime (#27312) ... closes CODAGT-839 closes CODAGT-843 closes CODAGT-773 ## Summary Adds a new heartbeat usage event type, `hb_agent_runtime_v1`, measuring the total agent-loop runtime of Coder Agents (chats) per UTC hour, plus a reconciler that generates one event per hour with self-healing backfill over a trailing 7-day window. Events flow to Tallyman through the existing publisher unchanged. This measures the new Coder Agents (the `chats` tables), not the deprecated Tasks counted by `dc_managed_agents_v1`. Independent of #27508, which fixes the dead ai-seats cron registration. Both PRs carry the identical `usage_event` create permission hunk for the usage-publisher subject (this feature's generator and the ai-seats cron each need it for heartbeat inserts), so they can land in either order and the overlap merges cleanly. > [!WARNING] > **Do not include this in a release until Tallyman accepts `hb_agent_runtime_v1`.** The publisher marks permanently rejected events as done-forever, and the generator then sees those buckets as complete locally, so their usage would be silently and permanently lost. ## Details Each event's payload is `{"runtime_ms": N}`: the sum of `chat_messages.runtime_ms` for messages created in the hour bucket `[H, H+1)`, across all chats (sub-agents, API-created, archived, and soft-deleted messages included). Events use deterministic IDs (`hb_agent_runtime_v1:<bucket start>`) with `created_at` set to the bucket start, so concurrent replicas race safely via `ON CONFLICT (id) DO NOTHING` without locking, and daily rollups attribute backfilled hours to the correct day. Idle hours produce zero-valued events. A bucket becomes eligible 5 minutes after it closes; hours missing for longer than the 7-day window are forfeited, which can only undercount. Note that this makes `usage_events.created_at` explicitly the *event occurrence time* rather than the row insertion time; the two only diverge for backfilled events. It already behaved as the occurrence timestamp (it drives the daily rollup day and is shipped to Tallyman/Metronome as the event timestamp), and the migration now documents this with a `COMMENT ON COLUMN`, which also surfaces as a Go doc comment on `UsageEvent.CreatedAt`. The new `usage.Generator` runs unconditionally in enterprise builds; the `publish_usage_data` license flag continues to gate egress only, so air-gapped deployments still fill their local ledger. The `aggregate_usage_event()` trigger sums `runtime_ms` per day into `usage_events_daily` (unlike `hb_ai_seats_v1`, which takes the daily max). `InsertHeartbeatUsageEvent` now takes an explicit `createdAt` so generators can backfill historical buckets; the cron passes `clock.Now()` to preserve its existing behavior. ## Tallyman follow-up <details> <summary>Prompt for the Tallyman-repo agent</summary> > **Task**: Add support for the new Coder usage event type `hb_agent_runtime_v1` so Tallyman accepts, validates, and forwards it to Metronome. > > **Background**: coder/coder PR (this PR) adds hourly heartbeat events measuring Coder Agent runtime. Events arrive via the existing `/api/v1/events/ingest` endpoint with: `event_type: "hb_agent_runtime_v1"`, `event_data: {"runtime_ms": <int64 >= 0>}`, deterministic `id` of the form `hb_agent_runtime_v1:2026-07-15_14:00:00` (UTC hour bucket start), and `created_at` set to the bucket start (may be up to ~8 days in the past due to backfill; within Metronome's 34-day dedup window). Zero-value events are normal (idle hours). > > **Work**: > 1. Update Tallyman's vendored/imported `coderd/usage/usagetypes` (or equivalent) to the coder/coder commit that adds `UsageEventTypeHBAgentRuntimeV1` and `HBAgentRuntime`. > 2. Ensure ingestion validation accepts the type (`Valid()` switches) and rejects negative `runtime_ms`. > 3. Ensure Metronome forwarding maps the event with transaction ID derived from the event `id` as for existing types, passing `runtime_ms` through as the property for a SUM-aggregated billable metric ("Coder Agent Hours" = `SUM(runtime_ms) / 3,600,000`). > 4. Do NOT permanently reject unknown-but-well-formed future `hb_*` types if avoidable; at minimum confirm current behavior for unknown types (temporary vs permanent rejection) and report it. > 5. Tests: ingest accept/validate, dedup by ID, Metronome payload mapping. > > **Constraint**: this must be deployed to tallyman-prod **before** any coder/coder release containing the event generator; coderd treats permanent rejections as terminal per event. </details>	2026-07-30 08:37:45 +01:00
Matt Vollmer	4987afada7	docs: present AI Governance as included with Premium (#27545) ... ## Summary AI Governance is now included with Premium licenses instead of being sold as a separate per-user add-on. This updates `docs/` to describe the new packaging, removes "Add-On" from AI Governance references, and refreshes the editions architecture diagram. ## Changes - **`docs/ai-coder/ai-governance.md`**: title is now "AI Governance"; rewrote the licensing statements (previously "a separate, per-user license... not included with a Premium subscription and must be purchased separately") to state it is included with Premium. The usage-pool section now attributes the shared Agent Workspace Build pool to Premium deployments. - **Repeated admonition (28 files under `ai-coder/agent-firewall/` and `ai-coder/ai-gateway/`)**: replaced "requires the AI Governance Add-On / as of Coder v2.32, deployments without the add-on..." with "is part of AI Governance, which is included with a Premium license." The v2.32 add-on gate no longer applies; the gate is now Premium vs. Community. - **`docs/ai-coder/index.md`, `security.md`, `tasks.md`, `usage-data-reporting.md`, `admin/licensing/index.md`, `install/releases/esr-2.29-2.34-upgrade.md`, `ai-gateway/ai-gateway-proxy/setup.md`, `ai-gateway/clients/claude-code.md`**: reworded add-on references to Premium inclusion. - **`docs/manifest.json`**: nav title "AI Governance Add-On" → "AI Governance", updated two descriptions, and swapped the 25 `"state": ["ai governance add-on"]` badges to `["premium"]` so the sidebar badge reads "Premium" instead of "AI Governance Add-On". - **`docs/images/single-region-architecture.png`**: refreshed the diagram in the **Community and Premium editions** tab on [Architecture](https://coder.com/docs/admin/infrastructure/architecture). Also deleted the unreferenced `single-region-architecture.svg` copy. ## Follow-ups outside this PR - The `"ai governance add-on"` doc-state badge is defined in `coder/coder.com` (`src/utils/docs/state.ts`). After this merges, no manifest entry uses that key, so it becomes dead config and can be removed there. - `enterprise/coderd/license/license.go:564-572` still warns admins that "The AI Governance add-on is required to use AI Gateway." That backend string will contradict these docs once shipped. ## Verification - `pnpm run lint-docs`: 0 errors across 504 files - `make lint/emdash`: clean - Vale on the changed Markdown files: 0 errors; remaining warnings are pre-existing gerund headings on untouched lines - `docs/manifest.json` validated as JSON - Confirmed the deleted SVG had no references anywhere in the repo --- PR generated with Coder Agents on behalf of @mattvollmer.	2026-07-29 08:45:20 -04:00
Nick VigilanteandDanny Kopping	61a9c4a61d	chore: Style fixes and nits across the AI Governance docs (#25793) ... - Add the "AI Governance Add-On" label across all pages - Use a generic `coder.example.com` URL across examples - Fix a few typos - Remove mentions of command access as a feature of AI Gov Fixes DOCS-262 <!-- If you have used AI to produce some or all of this PR, please ensure you have read our [AI Contribution guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING) before submitting. --> --------- Co-authored-by: Danny Kopping <danny@coder.com>	2026-06-01 13:04:14 +00:00
Jake Howell	0cea4de69e	fix: AI governance into AI Governance (#23553)	2026-03-25 20:06:48 +11:00
Ben Potter	ece531ab4e	chore: mention usage data reporting in AI Gov docs (#21664) ... <!-- If you have used AI to produce some or all of this PR, please ensure you have read our [AI Contribution guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING) before submitting. -->	2026-01-23 21:40:17 +00:00