Files
coder/enterprise/cli/server.go
T
Jaayden Halko 54d5eb7ec2 feat: add hourly hb_agent_runtime_v1 usage events for Coder Agent runtime (#27312)
closes CODAGT-839
closes CODAGT-843
closes CODAGT-773

## Summary

Adds a new heartbeat usage event type, `hb_agent_runtime_v1`, measuring
the total agent-loop runtime of Coder Agents (chats) per UTC hour, plus
a reconciler that generates one event per hour with self-healing
backfill over a trailing 7-day window. Events flow to Tallyman through
the existing publisher unchanged. This measures the new Coder Agents
(the `chats` tables), not the deprecated Tasks counted by
`dc_managed_agents_v1`.

Independent of #27508, which fixes the dead ai-seats cron registration.
Both PRs carry the identical `usage_event` create permission hunk for
the usage-publisher subject (this feature's generator and the ai-seats
cron each need it for heartbeat inserts), so they can land in either
order and the overlap merges cleanly.

> [!WARNING]
> **Do not include this in a release until Tallyman accepts
`hb_agent_runtime_v1`.** The publisher marks permanently rejected events
as done-forever, and the generator then sees those buckets as complete
locally, so their usage would be silently and permanently lost.

## Details

Each event's payload is `{"runtime_ms": N}`: the sum of
`chat_messages.runtime_ms` for messages created in the hour bucket `[H,
H+1)`, across all chats (sub-agents, API-created, archived, and
soft-deleted messages included). Events use deterministic IDs
(`hb_agent_runtime_v1:<bucket start>`) with `created_at` set to the
bucket start, so concurrent replicas race safely via `ON CONFLICT (id)
DO NOTHING` without locking, and daily rollups attribute backfilled
hours to the correct day. Idle hours produce zero-valued events. A
bucket becomes eligible 5 minutes after it closes; hours missing for
longer than the 7-day window are forfeited, which can only undercount.

Note that this makes `usage_events.created_at` explicitly the *event
occurrence time* rather than the row insertion time; the two only
diverge for backfilled events. It already behaved as the occurrence
timestamp (it drives the daily rollup day and is shipped to
Tallyman/Metronome as the event timestamp), and the migration now
documents this with a `COMMENT ON COLUMN`, which also surfaces as a Go
doc comment on `UsageEvent.CreatedAt`.

The new `usage.Generator` runs unconditionally in enterprise builds; the
`publish_usage_data` license flag continues to gate egress only, so
air-gapped deployments still fill their local ledger. The
`aggregate_usage_event()` trigger sums `runtime_ms` per day into
`usage_events_daily` (unlike `hb_ai_seats_v1`, which takes the daily
max).

`InsertHeartbeatUsageEvent` now takes an explicit `createdAt` so
generators can backfill historical buckets; the cron passes
`clock.Now()` to preserve its existing behavior.

## Tallyman follow-up

<details>
<summary>Prompt for the Tallyman-repo agent</summary>

> **Task**: Add support for the new Coder usage event type
`hb_agent_runtime_v1` so Tallyman accepts, validates, and forwards it to
Metronome.
>
> **Background**: coder/coder PR (this PR) adds hourly heartbeat events
measuring Coder Agent runtime. Events arrive via the existing
`/api/v1/events/ingest` endpoint with: `event_type:
"hb_agent_runtime_v1"`, `event_data: {"runtime_ms": <int64 >= 0>}`,
deterministic `id` of the form `hb_agent_runtime_v1:2026-07-15_14:00:00`
(UTC hour bucket start), and `created_at` set to the bucket start (may
be up to ~8 days in the past due to backfill; within Metronome's 34-day
dedup window). Zero-value events are normal (idle hours).
>
> **Work**:
> 1. Update Tallyman's vendored/imported `coderd/usage/usagetypes` (or
equivalent) to the coder/coder commit that adds
`UsageEventTypeHBAgentRuntimeV1` and `HBAgentRuntime`.
> 2. Ensure ingestion validation accepts the type (`Valid()` switches)
and rejects negative `runtime_ms`.
> 3. Ensure Metronome forwarding maps the event with transaction ID
derived from the event `id` as for existing types, passing `runtime_ms`
through as the property for a SUM-aggregated billable metric ("Coder
Agent Hours" = `SUM(runtime_ms) / 3,600,000`).
> 4. Do NOT permanently reject unknown-but-well-formed future `hb_*`
types if avoidable; at minimum confirm current behavior for unknown
types (temporary vs permanent rejection) and report it.
> 5. Tests: ingest accept/validate, dedup by ID, Metronome payload
mapping.
>
> **Constraint**: this must be deployed to tallyman-prod **before** any
coder/coder release containing the event generator; coderd treats
permanent rejections as terminal per event.

</details>
2026-07-30 08:37:45 +01:00

223 lines
7.5 KiB
Go

//go:build !slim
package cli
import (
"context"
"database/sql"
"encoding/base64"
"errors"
"io"
"time"
"golang.org/x/xerrors"
"tailscale.com/derp"
"tailscale.com/types/key"
agplcoderd "github.com/coder/coder/v2/coderd"
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/cryptorand"
"github.com/coder/coder/v2/enterprise/audit"
"github.com/coder/coder/v2/enterprise/audit/backends"
"github.com/coder/coder/v2/enterprise/coderd"
"github.com/coder/coder/v2/enterprise/coderd/dormancy"
"github.com/coder/coder/v2/enterprise/coderd/usage"
"github.com/coder/coder/v2/enterprise/dbcrypt"
"github.com/coder/coder/v2/enterprise/trialer"
"github.com/coder/coder/v2/tailnet"
"github.com/coder/quartz"
"github.com/coder/serpent"
)
func (r *RootCmd) Server(_ func()) *serpent.Command {
cmd := r.RootCmd.Server(func(ctx context.Context, options *agplcoderd.Options) (*agplcoderd.API, io.Closer, error) {
// Always generate a mesh key, even if the built-in DERP server is
// disabled. This mesh key is still used by workspace proxies running
// HA.
var meshKey string
err := options.Database.InTx(func(tx database.Store) error {
// This will block until the lock is acquired, and will be
// automatically released when the transaction ends.
err := tx.AcquireLock(ctx, database.LockIDEnterpriseDeploymentSetup)
if err != nil {
return xerrors.Errorf("acquire lock: %w", err)
}
meshKey, err = tx.GetDERPMeshKey(ctx)
if err == nil {
return nil
}
if !errors.Is(err, sql.ErrNoRows) {
return xerrors.Errorf("get DERP mesh key: %w", err)
}
meshKey, err = cryptorand.String(32)
if err != nil {
return xerrors.Errorf("generate DERP mesh key: %w", err)
}
err = tx.InsertDERPMeshKey(ctx, meshKey)
if err != nil {
return xerrors.Errorf("insert DERP mesh key: %w", err)
}
return nil
}, nil)
if err != nil {
return nil, nil, err
}
if meshKey == "" {
return nil, nil, xerrors.New("mesh key is empty")
}
if options.DeploymentValues.DERP.Server.Enable {
options.DERPServer = derp.NewServer(key.NewNode(), tailnet.Logger(options.Logger.Named("derp")))
options.DERPServer.SetMeshKey(meshKey)
}
options.Auditor = audit.NewAuditor(
options.Database,
audit.DefaultFilter,
backends.NewPostgres(options.Database, true),
backends.NewSlog(options.Logger),
)
options.TrialGenerator = trialer.New(options.Database, "https://v2-licensor.coder.com/trial", coderd.Keys)
o := &coderd.Options{
Options: options,
AuditLogging: true,
ConnectionLogging: true,
BrowserOnly: options.DeploymentValues.BrowserOnly.Value(),
SCIMAPIKey: []byte(options.DeploymentValues.SCIMAPIKey.Value()),
UseLegacySCIM: options.DeploymentValues.UseLegacySCIM.Value(),
RBAC: true,
DERPServerRelayAddress: options.DeploymentValues.DERP.Server.RelayURL.String(),
DERPServerRegionID: int(options.DeploymentValues.DERP.Server.RegionID.Value()),
ProxyHealthInterval: options.DeploymentValues.ProxyHealthStatusInterval.Value(),
DefaultQuietHoursSchedule: options.DeploymentValues.UserQuietHoursSchedule.DefaultSchedule.Value(),
ProvisionerDaemonPSK: options.DeploymentValues.Provisioner.DaemonPSK.Value(),
CheckInactiveUsersCancelFunc: dormancy.CheckInactiveUsers(ctx, options.Logger, quartz.NewReal(), options.Database, options.Auditor),
}
if encKeys := options.DeploymentValues.ExternalTokenEncryptionKeys.Value(); len(encKeys) != 0 {
keys := make([][]byte, 0, len(encKeys))
for idx, ek := range encKeys {
dk, err := base64.StdEncoding.DecodeString(ek)
if err != nil {
return nil, nil, xerrors.Errorf("decode external-token-encryption-key %d: %w", idx, err)
}
keys = append(keys, dk)
}
cs, err := dbcrypt.NewCiphers(keys...)
if err != nil {
return nil, nil, xerrors.Errorf("initialize encryption: %w", err)
}
o.ExternalTokenEncryption = cs
}
if o.LicenseKeys == nil {
o.LicenseKeys = coderd.Keys
}
closers := &multiCloser{}
// Create the enterprise API.
api, err := coderd.New(ctx, o)
if err != nil {
return nil, nil, err
}
closers.Add(api)
// Start the enterprise usage publisher routine. This won't do anything
// unless the deployment is licensed and one of the licenses has usage
// publishing enabled.
publisher := usage.NewTallymanPublisher(ctx, options.Logger, options.Database, o.LicenseKeys,
usage.PublisherWithHTTPClient(api.HTTPClient),
)
err = publisher.Start()
if err != nil {
_ = closers.Close()
return nil, nil, xerrors.Errorf("start usage publisher: %w", err)
}
closers.Add(publisher)
// usageCron are heartbeat events to the usage table. These events are eventually sent
// to Tallyman.
usageCron := usage.NewCron(quartz.NewReal(), options.Logger.Named("usage-cron"), options.Database, *options.UsageInserter.Load())
// ai-seats heartbeats track the number of users that have used an AI feature.
// These users consume a seat for the AI addon to our License.
_ = usageCron.Register(usage.CronJob{
Name: "ai-seats",
Interval: usage.AISeatsInterval,
Jitter: 10 * time.Minute,
Fn: usage.AISeatsHeartbeat(options.Database),
})
usageCron.Start(ctx)
closers.Add(usageCron)
// Usage generation is deliberately not license-gated; the
// publish_usage_data license flag only gates publishing to Tallyman.
usageGenerator := usage.NewGenerator(quartz.NewReal(), options.Logger.Named("usage-event-generator"), options.Database, *options.UsageInserter.Load())
usageGenerator.Start(ctx)
closers.Add(usageGenerator)
// In-memory AI Bridge Proxy daemon. The bridge daemon itself is
// started unconditionally by AGPL cli/server.go (chatd uses its
// in-memory roundtripper regardless of license); only the proxy
// daemon remains enterprise-gated by config.
if options.DeploymentValues.AI.BridgeProxyConfig.Enabled.Value() {
// Seed env-derived providers before the proxy daemon's reloader
// reads them back so the proxy observes them on first startup.
// options.Database is dbcrypt-wrapped at this point (set by
// coderd.New above), so env-seeded keys are also written
// encrypted. Detached ctx for the same reason as in agplcli
// below: an early return would orphan newAPI's goroutines.
// Seeding is idempotent; the agplcli path seeds again
// post-newAPI.
//nolint:gocritic // Production timeout, not a test wait.
aibridgeInitCtx, aibridgeInitCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer aibridgeInitCancel()
if err := agplcoderd.SeedAIProvidersFromEnv(
aibridgeInitCtx,
options.Database,
options.DeploymentValues.AI.BridgeConfig,
options.Logger.Named("aibridge.envseed"),
); err != nil {
return nil, nil, xerrors.Errorf("seed ai providers from env: %w", err)
}
aiBridgeProxyCloser, err := newAIBridgeProxyDaemon(api)
if err != nil {
_ = closers.Close()
return nil, nil, xerrors.Errorf("create aibridgeproxyd: %w", err)
}
closers.Add(aiBridgeProxyCloser)
}
return api.AGPL, closers, nil
})
cmd.AddSubcommands(
r.dbcryptCmd(),
)
return cmd
}
type multiCloser struct {
closers []io.Closer
}
var _ io.Closer = &multiCloser{}
func (m *multiCloser) Add(closer io.Closer) {
m.closers = append(m.closers, closer)
}
func (m *multiCloser) Close() error {
var errs []error
for _, closer := range m.closers {
if err := closer.Close(); err != nil {
errs = append(errs, xerrors.Errorf("close %T: %w", closer, err))
}
}
return errors.Join(errs...)
}