mirror of
https://github.com/coder/coder.git
synced 2026-09-01 14:53:15 +08:00
Yevhenii Shcherbina
05699c4d55
Example of generated PR: https://github.com/coder/coder/pull/28228 ## Summary The price book is regenerated by hand with `make gen/aibridge-prices`, so keeping it current depends on someone remembering to run it. Newly launched models stay unpriced and changed prices stay stale until the next manual run. This adds a weekly workflow that regenerates both artifacts from live [models.dev](https://models.dev) data and opens a pull request when the output changes. For scale, a run against `main` today produced 32 models added, 11 removed, and 43 repriced. ## What it does - Runs every Thursday at 09:00 UTC, plus `workflow_dispatch`. Exits without opening anything when regeneration produces no diff. - Reuses one branch and one pull request, force-pushing each week, so at most one refresh PR is open and it always carries the newest snapshot. - Ships `prices.json` and `knownModelsGenerated.json` together, since both come from a single upstream snapshot. - Never merges automatically. Prices are customer-visible cost numbers, so every change goes through human review. - Announces failures in Slack. The generator fails by design when upstream drops a model pinned in `overrides.jq` or curated in `curation.json`, and a silently red weekly cron would defeat the point. `scripts/aibridgepricesdiff` renders the PR body summary: counts, then the models added, removed, and repriced. Exact figures stay in the diff rather than being restated in the body. It is deterministic and unit tested rather than AI-generated, so the model lists cannot drift from the file. `CODEOWNERS` covers the two generated artifacts, which are exactly the files a refresh PR touches. ## Verification Ran on this branch via a temporary `push` trigger, since `schedule` and `workflow_dispatch` only fire from the default branch. Confirmed end to end: mise and pnpm setup, generation from live models.dev, change detection, summary and body assembly, commit under the bot identity, branch creation, force-push on the second run, and Slack delivery. `gh pr create` and `gh pr edit` remain unverified: both runs hit `HTTP 503` from `api.github.com/graphql` during a GitHub major outage. The push path itself succeeded, so the credential and permissions are proven. Also verified locally: `go test ./scripts/aibridgepricesdiff/`, `make lint/actions/actionlint`, `make lint/actions/zizmor`, and `make pre-commit`. ## Notes - The PR is opened with `secrets.CDRCI_GITHUB_TOKEN`. PRs opened with the default `GITHUB_TOKEN` do not trigger workflow runs, which would leave the refresh without CI signal on a file that feeds cost calculation. - Requires the `AIGATEWAY_PRICES_SLACK_WEBHOOK` repository secret. The notification step fails loudly if it is unset, since by then the refresh has already failed and nobody would be told. - The workflow installs pnpm dependencies because catalog generation formats its output with biome; without `site/node_modules` the formatting step silently no-ops and an unformatted file would be committed. - A refresh that lands after a release branch is cut can reach that release through the existing cherry-pick label, so no separate release-cut trigger is included. The label is not applied automatically; that stays a reviewer decision. - Identifiers keep the `aibridge` spelling where they name real paths and make targets (`coderd/aibridge/prices`, `make gen/aibridge-prices`, `scripts/aibridgepricesgen`). The workflow itself is named `aigateway-prices-refresh`, and prose says AI Gateway per the style guide. - The issue also asked for code owners on the generator. Only the generated artifacts are owned here, since those are the files a refresh PR touches. Ownership of `curation.json` and `overrides.jq` can be added separately if human edits to them should request review. <details> <summary>Implementation plan and decision log</summary> ### Goal Refresh `make gen/aibridge-prices` output on a schedule and open a human-reviewed PR when it changes, so releases never ship a stale price book. ### Schedule Weekly, Thursdays 09:00 UTC, leaving Thursday, Friday, and Monday for review before Tuesday releases. A monthly run was considered first. Release ships Tuesday morning and the freeze is the last Tuesday before EOM, so a monthly run on the 19th or 20th collapses to 1-2 usable business days in roughly 10 of 132 months: February, when the release Tuesday is the 22nd and the run day falls on a weekend. Measured across 2026-2036: | Cron day | Worst case usable business days | Months with 2 or fewer | Median | |---|---|---|---| | 20th | 1 | 10 | 6 | | 19th | 1 | 10 | 6 | | 16th | 3 | 0 | 6 | | Weekly Thursday | n/a, always a full week | 0 | n/a | The median is 6 days for every candidate, so moving to weekly costs nothing and removes the bad tail. The shipped book is never more than 7 days stale. ### Branch strategy One fixed branch, force-pushed, rather than a new branch and PR per run. `prices.json` is a full regeneration from the current upstream snapshot, not an incremental patch, so two open refresh PRs are the same file at two points in time and the older one is strictly wrong. A single PR makes merging a stale snapshot impossible, keeps reviewer load at one PR, and avoids a full CI run per week per stale PR. ### Summary generation Deterministic Go tool, not an AI summary. An LLM summarizing a JSON diff can drop or invent a row, and would be non-reproducible run to run. A jq implementation was considered and rejected as effectively untestable. The body lists which models moved; the diff in the Files tab remains the source of truth for figures. An earlier version rendered full price tables with percentage deltas. That duplicated the diff, so it was reduced to model lists. ### Token `secrets.CDRCI_GITHUB_TOKEN`, the cdrci machine user already used for bot-authored PRs in `release.yaml` and cited for exactly this reason in the commented-out `update-flake` job in `ci.yaml`. A dedicated GitHub App scoped to this repo would be tighter, but needs org admin to create. ### Commit identity `github-actions[bot]` with its numeric noreply address, matching `backport.yaml` and `cherry-pick.yaml`. An invented address would leave refresh commits with an unlinked author. ### Alerting A dedicated webhook secret rather than reusing `CI_FAILURE_SLACK_WEBHOOK`. The repo already splits alerting per domain (docs, security, dependabot each have their own webhook), and the CI-failure channel carries a user mention plus `vars.BLINK_CI_FAILURE_PROMPT` that feeds an automated triage flow this payload does not belong in. GitHub's built-in email notification was considered and rejected as the primary channel: for scheduled workflows it goes to a single user, the one who last modified the cron syntax, and cannot be routed to a team. ### Out of scope Triggering a refresh when a release branch is cut. The existing cherry-pick label already gets a late refresh into a release, so a separate mechanism is unnecessary. </details> --- Closes [AIGOV-578](https://linear.app/codercom/issue/AIGOV-578/automate-updates-to-the-shipped-ai-model-price-book). Authored by Coder Agents on behalf of @evgeniy-scherbina.
50 lines
2.2 KiB
Plaintext
50 lines
2.2 KiB
Plaintext
# These APIs are versioned, so any changes need to be carefully reviewed for
|
|
# whether to bump API major or minor versions.
|
|
agent/proto/ @spikecurtis @johnstcn
|
|
provisionerd/proto/ @spikecurtis @johnstcn
|
|
provisionersdk/proto/ @spikecurtis @johnstcn
|
|
tailnet/proto/ @spikecurtis @johnstcn
|
|
vpn/vpn.proto @spikecurtis @johnstcn
|
|
vpn/version.go @spikecurtis @johnstcn
|
|
|
|
# This caching code is particularly tricky, and one must be very careful when
|
|
# altering it.
|
|
coderd/files/ @aslilac
|
|
|
|
coderd/dynamicparameters/ @Emyrk
|
|
coderd/rbac/policy.rego @Emyrk
|
|
|
|
# Mainly dependent on coder/guts, which is maintained by @Emyrk
|
|
scripts/apitypings/ @Emyrk
|
|
scripts/gensite/ @aslilac
|
|
|
|
# The blood and guts of the autostop algorithm, which is quite complex and
|
|
# requires elite ball knowledge of most of the scheduling code to make changes
|
|
# without inadvertently affecting other parts of the codebase.
|
|
coderd/schedule/autostop.go @deansheather @DanielleMaywood
|
|
|
|
# Usage tracking code requires intimate knowledge of Tallyman and Metronome, as
|
|
# well as guidance from revenue.
|
|
coderd/usage/ @deansheather @spikecurtis
|
|
enterprise/coderd/usage/ @deansheather @spikecurtis
|
|
|
|
site/src/pages/AgentsPage/ @DanielleMaywood
|
|
|
|
.github/ @jdomeracki-coder
|
|
|
|
# Shared contract between chatd and aibridge: AI provider configuration and
|
|
# model pricing, read by both. Note aicostcontrol.sql also carries group/user
|
|
# AI budget and spend queries alongside the ai_model_prices queries.
|
|
coderd/database/queries/ai_providers.sql @ibetitsmike @johnstcn
|
|
coderd/database/queries/ai_provider_keys.sql @ibetitsmike @johnstcn
|
|
coderd/database/queries/aicostcontrol.sql @ibetitsmike @johnstcn
|
|
codersdk/aiproviders.go @ibetitsmike @johnstcn
|
|
codersdk/aiproviders_bedrock.go @ibetitsmike @johnstcn
|
|
|
|
# Generated price book and frontend model catalog. The
|
|
# aigateway-prices-refresh workflow proposes updates to both from live
|
|
# models.dev data, and they carry customer-visible cost numbers, so every
|
|
# refresh needs a review from someone who owns them.
|
|
coderd/aibridge/prices/data/prices.json @evgeniy-scherbina @ssncferreira @johnstcn
|
|
site/src/pages/AgentsPage/components/ChatModelAdminPanel/knownModels/knownModelsGenerated.json @evgeniy-scherbina @ssncferreira @johnstcn
|