mirror of
https://github.com/coder/coder.git
synced 2026-09-23 14:03:57 +08:00
closes CODAGT-839 closes CODAGT-843 closes CODAGT-773 ## Summary Adds a new heartbeat usage event type, `hb_agent_runtime_v1`, measuring the total agent-loop runtime of Coder Agents (chats) per UTC hour, plus a reconciler that generates one event per hour with self-healing backfill over a trailing 7-day window. Events flow to Tallyman through the existing publisher unchanged. This measures the new Coder Agents (the `chats` tables), not the deprecated Tasks counted by `dc_managed_agents_v1`. Independent of #27508, which fixes the dead ai-seats cron registration. Both PRs carry the identical `usage_event` create permission hunk for the usage-publisher subject (this feature's generator and the ai-seats cron each need it for heartbeat inserts), so they can land in either order and the overlap merges cleanly. > [!WARNING] > **Do not include this in a release until Tallyman accepts `hb_agent_runtime_v1`.** The publisher marks permanently rejected events as done-forever, and the generator then sees those buckets as complete locally, so their usage would be silently and permanently lost. ## Details Each event's payload is `{"runtime_ms": N}`: the sum of `chat_messages.runtime_ms` for messages created in the hour bucket `[H, H+1)`, across all chats (sub-agents, API-created, archived, and soft-deleted messages included). Events use deterministic IDs (`hb_agent_runtime_v1:<bucket start>`) with `created_at` set to the bucket start, so concurrent replicas race safely via `ON CONFLICT (id) DO NOTHING` without locking, and daily rollups attribute backfilled hours to the correct day. Idle hours produce zero-valued events. A bucket becomes eligible 5 minutes after it closes; hours missing for longer than the 7-day window are forfeited, which can only undercount. Note that this makes `usage_events.created_at` explicitly the *event occurrence time* rather than the row insertion time; the two only diverge for backfilled events. It already behaved as the occurrence timestamp (it drives the daily rollup day and is shipped to Tallyman/Metronome as the event timestamp), and the migration now documents this with a `COMMENT ON COLUMN`, which also surfaces as a Go doc comment on `UsageEvent.CreatedAt`. The new `usage.Generator` runs unconditionally in enterprise builds; the `publish_usage_data` license flag continues to gate egress only, so air-gapped deployments still fill their local ledger. The `aggregate_usage_event()` trigger sums `runtime_ms` per day into `usage_events_daily` (unlike `hb_ai_seats_v1`, which takes the daily max). `InsertHeartbeatUsageEvent` now takes an explicit `createdAt` so generators can backfill historical buckets; the cron passes `clock.Now()` to preserve its existing behavior. ## Tallyman follow-up <details> <summary>Prompt for the Tallyman-repo agent</summary> > **Task**: Add support for the new Coder usage event type `hb_agent_runtime_v1` so Tallyman accepts, validates, and forwards it to Metronome. > > **Background**: coder/coder PR (this PR) adds hourly heartbeat events measuring Coder Agent runtime. Events arrive via the existing `/api/v1/events/ingest` endpoint with: `event_type: "hb_agent_runtime_v1"`, `event_data: {"runtime_ms": <int64 >= 0>}`, deterministic `id` of the form `hb_agent_runtime_v1:2026-07-15_14:00:00` (UTC hour bucket start), and `created_at` set to the bucket start (may be up to ~8 days in the past due to backfill; within Metronome's 34-day dedup window). Zero-value events are normal (idle hours). > > **Work**: > 1. Update Tallyman's vendored/imported `coderd/usage/usagetypes` (or equivalent) to the coder/coder commit that adds `UsageEventTypeHBAgentRuntimeV1` and `HBAgentRuntime`. > 2. Ensure ingestion validation accepts the type (`Valid()` switches) and rejects negative `runtime_ms`. > 3. Ensure Metronome forwarding maps the event with transaction ID derived from the event `id` as for existing types, passing `runtime_ms` through as the property for a SUM-aggregated billable metric ("Coder Agent Hours" = `SUM(runtime_ms) / 3,600,000`). > 4. Do NOT permanently reject unknown-but-well-formed future `hb_*` types if avoidable; at minimum confirm current behavior for unknown types (temporary vs permanent rejection) and report it. > 5. Tests: ingest accept/validate, dedup by ID, Metronome payload mapping. > > **Constraint**: this must be deployed to tallyman-prod **before** any coder/coder release containing the event generator; coderd treats permanent rejections as terminal per event. </details>
223 lines
7.5 KiB
Go
223 lines
7.5 KiB
Go
//go:build !slim
|
|
|
|
package cli
|
|
|
|
import (
|
|
"context"
|
|
"database/sql"
|
|
"encoding/base64"
|
|
"errors"
|
|
"io"
|
|
"time"
|
|
|
|
"golang.org/x/xerrors"
|
|
"tailscale.com/derp"
|
|
"tailscale.com/types/key"
|
|
|
|
agplcoderd "github.com/coder/coder/v2/coderd"
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
"github.com/coder/coder/v2/cryptorand"
|
|
"github.com/coder/coder/v2/enterprise/audit"
|
|
"github.com/coder/coder/v2/enterprise/audit/backends"
|
|
"github.com/coder/coder/v2/enterprise/coderd"
|
|
"github.com/coder/coder/v2/enterprise/coderd/dormancy"
|
|
"github.com/coder/coder/v2/enterprise/coderd/usage"
|
|
"github.com/coder/coder/v2/enterprise/dbcrypt"
|
|
"github.com/coder/coder/v2/enterprise/trialer"
|
|
"github.com/coder/coder/v2/tailnet"
|
|
"github.com/coder/quartz"
|
|
"github.com/coder/serpent"
|
|
)
|
|
|
|
func (r *RootCmd) Server(_ func()) *serpent.Command {
|
|
cmd := r.RootCmd.Server(func(ctx context.Context, options *agplcoderd.Options) (*agplcoderd.API, io.Closer, error) {
|
|
// Always generate a mesh key, even if the built-in DERP server is
|
|
// disabled. This mesh key is still used by workspace proxies running
|
|
// HA.
|
|
var meshKey string
|
|
err := options.Database.InTx(func(tx database.Store) error {
|
|
// This will block until the lock is acquired, and will be
|
|
// automatically released when the transaction ends.
|
|
err := tx.AcquireLock(ctx, database.LockIDEnterpriseDeploymentSetup)
|
|
if err != nil {
|
|
return xerrors.Errorf("acquire lock: %w", err)
|
|
}
|
|
|
|
meshKey, err = tx.GetDERPMeshKey(ctx)
|
|
if err == nil {
|
|
return nil
|
|
}
|
|
if !errors.Is(err, sql.ErrNoRows) {
|
|
return xerrors.Errorf("get DERP mesh key: %w", err)
|
|
}
|
|
meshKey, err = cryptorand.String(32)
|
|
if err != nil {
|
|
return xerrors.Errorf("generate DERP mesh key: %w", err)
|
|
}
|
|
err = tx.InsertDERPMeshKey(ctx, meshKey)
|
|
if err != nil {
|
|
return xerrors.Errorf("insert DERP mesh key: %w", err)
|
|
}
|
|
return nil
|
|
}, nil)
|
|
if err != nil {
|
|
return nil, nil, err
|
|
}
|
|
if meshKey == "" {
|
|
return nil, nil, xerrors.New("mesh key is empty")
|
|
}
|
|
|
|
if options.DeploymentValues.DERP.Server.Enable {
|
|
options.DERPServer = derp.NewServer(key.NewNode(), tailnet.Logger(options.Logger.Named("derp")))
|
|
options.DERPServer.SetMeshKey(meshKey)
|
|
}
|
|
|
|
options.Auditor = audit.NewAuditor(
|
|
options.Database,
|
|
audit.DefaultFilter,
|
|
backends.NewPostgres(options.Database, true),
|
|
backends.NewSlog(options.Logger),
|
|
)
|
|
|
|
options.TrialGenerator = trialer.New(options.Database, "https://v2-licensor.coder.com/trial", coderd.Keys)
|
|
|
|
o := &coderd.Options{
|
|
Options: options,
|
|
AuditLogging: true,
|
|
ConnectionLogging: true,
|
|
BrowserOnly: options.DeploymentValues.BrowserOnly.Value(),
|
|
SCIMAPIKey: []byte(options.DeploymentValues.SCIMAPIKey.Value()),
|
|
UseLegacySCIM: options.DeploymentValues.UseLegacySCIM.Value(),
|
|
RBAC: true,
|
|
DERPServerRelayAddress: options.DeploymentValues.DERP.Server.RelayURL.String(),
|
|
DERPServerRegionID: int(options.DeploymentValues.DERP.Server.RegionID.Value()),
|
|
ProxyHealthInterval: options.DeploymentValues.ProxyHealthStatusInterval.Value(),
|
|
DefaultQuietHoursSchedule: options.DeploymentValues.UserQuietHoursSchedule.DefaultSchedule.Value(),
|
|
ProvisionerDaemonPSK: options.DeploymentValues.Provisioner.DaemonPSK.Value(),
|
|
|
|
CheckInactiveUsersCancelFunc: dormancy.CheckInactiveUsers(ctx, options.Logger, quartz.NewReal(), options.Database, options.Auditor),
|
|
}
|
|
|
|
if encKeys := options.DeploymentValues.ExternalTokenEncryptionKeys.Value(); len(encKeys) != 0 {
|
|
keys := make([][]byte, 0, len(encKeys))
|
|
for idx, ek := range encKeys {
|
|
dk, err := base64.StdEncoding.DecodeString(ek)
|
|
if err != nil {
|
|
return nil, nil, xerrors.Errorf("decode external-token-encryption-key %d: %w", idx, err)
|
|
}
|
|
keys = append(keys, dk)
|
|
}
|
|
cs, err := dbcrypt.NewCiphers(keys...)
|
|
if err != nil {
|
|
return nil, nil, xerrors.Errorf("initialize encryption: %w", err)
|
|
}
|
|
o.ExternalTokenEncryption = cs
|
|
}
|
|
|
|
if o.LicenseKeys == nil {
|
|
o.LicenseKeys = coderd.Keys
|
|
}
|
|
|
|
closers := &multiCloser{}
|
|
|
|
// Create the enterprise API.
|
|
api, err := coderd.New(ctx, o)
|
|
if err != nil {
|
|
return nil, nil, err
|
|
}
|
|
closers.Add(api)
|
|
|
|
// Start the enterprise usage publisher routine. This won't do anything
|
|
// unless the deployment is licensed and one of the licenses has usage
|
|
// publishing enabled.
|
|
publisher := usage.NewTallymanPublisher(ctx, options.Logger, options.Database, o.LicenseKeys,
|
|
usage.PublisherWithHTTPClient(api.HTTPClient),
|
|
)
|
|
err = publisher.Start()
|
|
if err != nil {
|
|
_ = closers.Close()
|
|
return nil, nil, xerrors.Errorf("start usage publisher: %w", err)
|
|
}
|
|
closers.Add(publisher)
|
|
|
|
// usageCron are heartbeat events to the usage table. These events are eventually sent
|
|
// to Tallyman.
|
|
usageCron := usage.NewCron(quartz.NewReal(), options.Logger.Named("usage-cron"), options.Database, *options.UsageInserter.Load())
|
|
// ai-seats heartbeats track the number of users that have used an AI feature.
|
|
// These users consume a seat for the AI addon to our License.
|
|
_ = usageCron.Register(usage.CronJob{
|
|
Name: "ai-seats",
|
|
Interval: usage.AISeatsInterval,
|
|
Jitter: 10 * time.Minute,
|
|
Fn: usage.AISeatsHeartbeat(options.Database),
|
|
})
|
|
usageCron.Start(ctx)
|
|
closers.Add(usageCron)
|
|
|
|
// Usage generation is deliberately not license-gated; the
|
|
// publish_usage_data license flag only gates publishing to Tallyman.
|
|
usageGenerator := usage.NewGenerator(quartz.NewReal(), options.Logger.Named("usage-event-generator"), options.Database, *options.UsageInserter.Load())
|
|
usageGenerator.Start(ctx)
|
|
closers.Add(usageGenerator)
|
|
|
|
// In-memory AI Bridge Proxy daemon. The bridge daemon itself is
|
|
// started unconditionally by AGPL cli/server.go (chatd uses its
|
|
// in-memory roundtripper regardless of license); only the proxy
|
|
// daemon remains enterprise-gated by config.
|
|
if options.DeploymentValues.AI.BridgeProxyConfig.Enabled.Value() {
|
|
// Seed env-derived providers before the proxy daemon's reloader
|
|
// reads them back so the proxy observes them on first startup.
|
|
// options.Database is dbcrypt-wrapped at this point (set by
|
|
// coderd.New above), so env-seeded keys are also written
|
|
// encrypted. Detached ctx for the same reason as in agplcli
|
|
// below: an early return would orphan newAPI's goroutines.
|
|
// Seeding is idempotent; the agplcli path seeds again
|
|
// post-newAPI.
|
|
//nolint:gocritic // Production timeout, not a test wait.
|
|
aibridgeInitCtx, aibridgeInitCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
|
|
defer aibridgeInitCancel()
|
|
if err := agplcoderd.SeedAIProvidersFromEnv(
|
|
aibridgeInitCtx,
|
|
options.Database,
|
|
options.DeploymentValues.AI.BridgeConfig,
|
|
options.Logger.Named("aibridge.envseed"),
|
|
); err != nil {
|
|
return nil, nil, xerrors.Errorf("seed ai providers from env: %w", err)
|
|
}
|
|
aiBridgeProxyCloser, err := newAIBridgeProxyDaemon(api)
|
|
if err != nil {
|
|
_ = closers.Close()
|
|
return nil, nil, xerrors.Errorf("create aibridgeproxyd: %w", err)
|
|
}
|
|
closers.Add(aiBridgeProxyCloser)
|
|
}
|
|
|
|
return api.AGPL, closers, nil
|
|
})
|
|
|
|
cmd.AddSubcommands(
|
|
r.dbcryptCmd(),
|
|
)
|
|
return cmd
|
|
}
|
|
|
|
type multiCloser struct {
|
|
closers []io.Closer
|
|
}
|
|
|
|
var _ io.Closer = &multiCloser{}
|
|
|
|
func (m *multiCloser) Add(closer io.Closer) {
|
|
m.closers = append(m.closers, closer)
|
|
}
|
|
|
|
func (m *multiCloser) Close() error {
|
|
var errs []error
|
|
for _, closer := range m.closers {
|
|
if err := closer.Close(); err != nil {
|
|
errs = append(errs, xerrors.Errorf("close %T: %w", closer, err))
|
|
}
|
|
}
|
|
return errors.Join(errs...)
|
|
}
|