mirror of
https://github.com/coder/coder.git
synced 2026-09-24 15:04:27 +08:00
feat: automate weekly AI model price book refresh (#28146)
Example of generated PR: https://github.com/coder/coder/pull/28228 ## Summary The price book is regenerated by hand with `make gen/aibridge-prices`, so keeping it current depends on someone remembering to run it. Newly launched models stay unpriced and changed prices stay stale until the next manual run. This adds a weekly workflow that regenerates both artifacts from live [models.dev](https://models.dev) data and opens a pull request when the output changes. For scale, a run against `main` today produced 32 models added, 11 removed, and 43 repriced. ## What it does - Runs every Thursday at 09:00 UTC, plus `workflow_dispatch`. Exits without opening anything when regeneration produces no diff. - Reuses one branch and one pull request, force-pushing each week, so at most one refresh PR is open and it always carries the newest snapshot. - Ships `prices.json` and `knownModelsGenerated.json` together, since both come from a single upstream snapshot. - Never merges automatically. Prices are customer-visible cost numbers, so every change goes through human review. - Announces failures in Slack. The generator fails by design when upstream drops a model pinned in `overrides.jq` or curated in `curation.json`, and a silently red weekly cron would defeat the point. `scripts/aibridgepricesdiff` renders the PR body summary: counts, then the models added, removed, and repriced. Exact figures stay in the diff rather than being restated in the body. It is deterministic and unit tested rather than AI-generated, so the model lists cannot drift from the file. `CODEOWNERS` covers the two generated artifacts, which are exactly the files a refresh PR touches. ## Verification Ran on this branch via a temporary `push` trigger, since `schedule` and `workflow_dispatch` only fire from the default branch. Confirmed end to end: mise and pnpm setup, generation from live models.dev, change detection, summary and body assembly, commit under the bot identity, branch creation, force-push on the second run, and Slack delivery. `gh pr create` and `gh pr edit` remain unverified: both runs hit `HTTP 503` from `api.github.com/graphql` during a GitHub major outage. The push path itself succeeded, so the credential and permissions are proven. Also verified locally: `go test ./scripts/aibridgepricesdiff/`, `make lint/actions/actionlint`, `make lint/actions/zizmor`, and `make pre-commit`. ## Notes - The PR is opened with `secrets.CDRCI_GITHUB_TOKEN`. PRs opened with the default `GITHUB_TOKEN` do not trigger workflow runs, which would leave the refresh without CI signal on a file that feeds cost calculation. - Requires the `AIGATEWAY_PRICES_SLACK_WEBHOOK` repository secret. The notification step fails loudly if it is unset, since by then the refresh has already failed and nobody would be told. - The workflow installs pnpm dependencies because catalog generation formats its output with biome; without `site/node_modules` the formatting step silently no-ops and an unformatted file would be committed. - A refresh that lands after a release branch is cut can reach that release through the existing cherry-pick label, so no separate release-cut trigger is included. The label is not applied automatically; that stays a reviewer decision. - Identifiers keep the `aibridge` spelling where they name real paths and make targets (`coderd/aibridge/prices`, `make gen/aibridge-prices`, `scripts/aibridgepricesgen`). The workflow itself is named `aigateway-prices-refresh`, and prose says AI Gateway per the style guide. - The issue also asked for code owners on the generator. Only the generated artifacts are owned here, since those are the files a refresh PR touches. Ownership of `curation.json` and `overrides.jq` can be added separately if human edits to them should request review. <details> <summary>Implementation plan and decision log</summary> ### Goal Refresh `make gen/aibridge-prices` output on a schedule and open a human-reviewed PR when it changes, so releases never ship a stale price book. ### Schedule Weekly, Thursdays 09:00 UTC, leaving Thursday, Friday, and Monday for review before Tuesday releases. A monthly run was considered first. Release ships Tuesday morning and the freeze is the last Tuesday before EOM, so a monthly run on the 19th or 20th collapses to 1-2 usable business days in roughly 10 of 132 months: February, when the release Tuesday is the 22nd and the run day falls on a weekend. Measured across 2026-2036: | Cron day | Worst case usable business days | Months with 2 or fewer | Median | |---|---|---|---| | 20th | 1 | 10 | 6 | | 19th | 1 | 10 | 6 | | 16th | 3 | 0 | 6 | | Weekly Thursday | n/a, always a full week | 0 | n/a | The median is 6 days for every candidate, so moving to weekly costs nothing and removes the bad tail. The shipped book is never more than 7 days stale. ### Branch strategy One fixed branch, force-pushed, rather than a new branch and PR per run. `prices.json` is a full regeneration from the current upstream snapshot, not an incremental patch, so two open refresh PRs are the same file at two points in time and the older one is strictly wrong. A single PR makes merging a stale snapshot impossible, keeps reviewer load at one PR, and avoids a full CI run per week per stale PR. ### Summary generation Deterministic Go tool, not an AI summary. An LLM summarizing a JSON diff can drop or invent a row, and would be non-reproducible run to run. A jq implementation was considered and rejected as effectively untestable. The body lists which models moved; the diff in the Files tab remains the source of truth for figures. An earlier version rendered full price tables with percentage deltas. That duplicated the diff, so it was reduced to model lists. ### Token `secrets.CDRCI_GITHUB_TOKEN`, the cdrci machine user already used for bot-authored PRs in `release.yaml` and cited for exactly this reason in the commented-out `update-flake` job in `ci.yaml`. A dedicated GitHub App scoped to this repo would be tighter, but needs org admin to create. ### Commit identity `github-actions[bot]` with its numeric noreply address, matching `backport.yaml` and `cherry-pick.yaml`. An invented address would leave refresh commits with an unlinked author. ### Alerting A dedicated webhook secret rather than reusing `CI_FAILURE_SLACK_WEBHOOK`. The repo already splits alerting per domain (docs, security, dependabot each have their own webhook), and the CI-failure channel carries a user mention plus `vars.BLINK_CI_FAILURE_PROMPT` that feeds an automated triage flow this payload does not belong in. GitHub's built-in email notification was considered and rejected as the primary channel: for scheduled workflows it goes to a single user, the one who last modified the cron syntax, and cannot be routed to a team. ### Out of scope Triggering a refresh when a release branch is cut. The existing cherry-pick label already gets a late refresh into a release, so a separate mechanism is unnecessary. </details> --- Closes [AIGOV-578](https://linear.app/codercom/issue/AIGOV-578/automate-updates-to-the-shipped-ai-model-price-book). Authored by Coder Agents on behalf of @evgeniy-scherbina.
This commit is contained in:
Yevhenii Shcherbina
@@ -0,0 +1,178 @@
|
||||
# Refreshes the AI Gateway price book from live upstream data (models.dev)
|
||||
# once a week and opens a pull request when the generated artifacts change.
|
||||
#
|
||||
# The price book seeds customer-visible cost numbers, so the refresh is never
|
||||
# merged automatically. The workflow only ever proposes a change; a human
|
||||
# reviews and merges it.
|
||||
#
|
||||
# Behavior:
|
||||
# - Runs every Thursday. If regeneration produces no diff, the run ends
|
||||
# without opening anything.
|
||||
# - Reuses a single branch and pull request, force-pushing each week, so at
|
||||
# most one refresh PR is open and it always carries the newest snapshot.
|
||||
# - Fails loudly when the generator refuses to run, which it does by design
|
||||
# when upstream drops a model pinned in overrides.jq or curated in
|
||||
# curation.json. Failures are announced in Slack.
|
||||
name: aigateway-prices-refresh
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# 09:00 UTC every Thursday, leaving three business days before Tuesday releases.
|
||||
- cron: "0 9 * * 4"
|
||||
workflow_dispatch: # allows manual runs for testing
|
||||
|
||||
permissions: {}
|
||||
|
||||
concurrency:
|
||||
group: aigateway-prices-refresh
|
||||
|
||||
env:
|
||||
REFRESH_BRANCH: bot/aigateway-prices-refresh
|
||||
PRICES_FILE: coderd/aibridge/prices/data/prices.json
|
||||
CATALOG_FILE: site/src/pages/AgentsPage/components/ChatModelAdminPanel/knownModels/knownModelsGenerated.json
|
||||
|
||||
jobs:
|
||||
refresh:
|
||||
name: Refresh price book
|
||||
runs-on: ubuntu-latest
|
||||
permissions: {}
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up mise tools
|
||||
uses: ./.github/actions/setup-mise
|
||||
with:
|
||||
install-args: "go node pnpm"
|
||||
|
||||
# Needed by catalog generation, which formats its output with biome.
|
||||
- name: Install pnpm dependencies
|
||||
uses: ./.github/actions/pnpm-install
|
||||
|
||||
- name: Snapshot the current price book
|
||||
run: cp "${PRICES_FILE}" "${RUNNER_TEMP}/prices-before.json"
|
||||
|
||||
- name: Regenerate price book and model catalog
|
||||
run: make gen/aibridge-prices
|
||||
|
||||
- name: Detect changes
|
||||
id: detect
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if git diff --quiet -- "${PRICES_FILE}" "${CATALOG_FILE}"; then
|
||||
# exit 0 => NO differences => nothing to propose
|
||||
echo "Price book already matches upstream; nothing to propose."
|
||||
echo "changed=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
# exit 1 => differences found
|
||||
git diff --stat -- "${PRICES_FILE}" "${CATALOG_FILE}"
|
||||
echo "changed=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Build pull request body
|
||||
if: steps.detect.outputs.changed == 'true'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
go run ./scripts/aibridgepricesdiff \
|
||||
-old "${RUNNER_TEMP}/prices-before.json" \
|
||||
-new "${PRICES_FILE}" > "${RUNNER_TEMP}/summary.md"
|
||||
|
||||
{
|
||||
cat "${RUNNER_TEMP}/summary.md"
|
||||
echo
|
||||
echo "## Review notes"
|
||||
echo
|
||||
echo "Regenerated by \`make gen/aibridge-prices\` from the live [models.dev](https://models.dev) catalog. Both artifacts come from one snapshot, so they ship together:"
|
||||
echo
|
||||
echo "- \`${PRICES_FILE}\`"
|
||||
echo "- \`${CATALOG_FILE}\`"
|
||||
echo
|
||||
echo "These are customer-visible cost numbers taken from upstream data, so this PR is never merged automatically. The summary above lists what moved; check the diff for exact figures before approving."
|
||||
echo
|
||||
echo "Opened automatically by the [aigateway-prices-refresh workflow](${RUN_URL})."
|
||||
} > "${RUNNER_TEMP}/body.md"
|
||||
|
||||
cat "${RUNNER_TEMP}/body.md"
|
||||
env:
|
||||
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
|
||||
- name: Open or update the refresh pull request
|
||||
if: steps.detect.outputs.changed == 'true'
|
||||
env:
|
||||
# Use cdrci's token instead of the default GITHUB_TOKEN: PRs opened
|
||||
# with GITHUB_TOKEN do not trigger workflow runs, so the refresh
|
||||
# would arrive without CI signal.
|
||||
GH_TOKEN: ${{ secrets.CDRCI_GITHUB_TOKEN }}
|
||||
PR_TITLE: "chore: refresh AI model price book"
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
# persist-credentials is disabled on checkout, so authenticate the
|
||||
# push explicitly.
|
||||
git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
|
||||
git checkout -b "${REFRESH_BRANCH}"
|
||||
git add -- "${PRICES_FILE}" "${CATALOG_FILE}"
|
||||
git commit -m "${PR_TITLE}"
|
||||
|
||||
# Force-push: the branch is regenerated from the newest upstream
|
||||
# snapshot each week, so the previous contents are always stale.
|
||||
git push --force origin "refs/heads/${REFRESH_BRANCH}"
|
||||
|
||||
# REST, not `gh pr`: those go through GraphQL, which needs a read:org
|
||||
# scope that cdrci's token lacks.
|
||||
owner="${GITHUB_REPOSITORY%%/*}"
|
||||
pr_number="$(gh api "repos/${GITHUB_REPOSITORY}/pulls?state=open&base=main&head=${owner}:${REFRESH_BRANCH}" --jq '.[0].number // empty')"
|
||||
if [ -n "${pr_number}" ]; then
|
||||
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/pulls/${pr_number}" \
|
||||
-f title="${PR_TITLE}" \
|
||||
-f body="$(cat "${RUNNER_TEMP}/body.md")" \
|
||||
--silent
|
||||
echo "Updated existing PR #${pr_number}."
|
||||
else
|
||||
gh api --method POST "repos/${GITHUB_REPOSITORY}/pulls" \
|
||||
-f title="${PR_TITLE}" \
|
||||
-f head="${REFRESH_BRANCH}" \
|
||||
-f base=main \
|
||||
-f body="$(cat "${RUNNER_TEMP}/body.md")" \
|
||||
--jq '.html_url'
|
||||
fi
|
||||
|
||||
- name: Send Slack notification
|
||||
if: failure() || steps.detect.outputs.changed == 'false'
|
||||
env:
|
||||
JOB_STATUS: ${{ job.status }}
|
||||
PRICE_BOOK_CHANGED: ${{ steps.detect.outputs.changed }}
|
||||
SLACK_WEBHOOK: ${{ secrets.AIGATEWAY_PRICES_SLACK_WEBHOOK }}
|
||||
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${SLACK_WEBHOOK}" ]; then
|
||||
echo "::error::AIGATEWAY_PRICES_SLACK_WEBHOOK is not set; the notification could not be sent."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "${JOB_STATUS}" = "failure" ]; then
|
||||
# printf, not a double-quoted literal: bash leaves \n as two
|
||||
# characters, and jq --arg then escapes the backslash, so Slack
|
||||
# would print \n as text instead of breaking the line.
|
||||
text="$(printf ':warning: *AI model price book refresh failed.*\nThe generator fails by design when upstream drops a model pinned in scripts/aibridgepricesgen/overrides.jq or curated in curation.json. Logs: %s' "${RUN_URL}")"
|
||||
elif [ "${PRICE_BOOK_CHANGED}" = "false" ]; then
|
||||
text=":white_check_mark: *AI Gateway price book refresh completed.* No generated changes were found. Run: ${RUN_URL}"
|
||||
else
|
||||
echo "::error::Unexpected notification state: status=${JOB_STATUS}, changed=${PRICE_BOOK_CHANGED}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
payload="$(jq -nc --arg text "${text}" '{text: $text}')"
|
||||
curl -fsSL -X POST -H 'Content-type: application/json' -d "${payload}" "${SLACK_WEBHOOK}"
|
||||
echo "Sent Slack notification"
|
||||
Reference in New Issue
Block a user