opensource/coder
3
0
Fork 0
mirror of https://github.com/coder/coder.git synced 2026-09-24 15:04:27 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor: build dogfood image as base + mise oci layers (#25448)

Browse Source 
Splits the dogfood image into two artifacts:

- `ghcr.io/coder/oss-dogfood-base:<distro>-<base-sha>`: Ubuntu base with
apt packages, chrome, rustup, brew, gh, and the mise binary. The
base-sha is a cache key over `Dockerfile.base` and `files/`, so commits
that don't touch those inputs reuse the previous build.
- `codercom/oss-dogfood:<final-sha>-<distro>` and rolling tags
(`:22.04`, `:26.04`, `:latest`, `:<branch>`): produced by `mise oci
build` on top of the base, with one content-addressed OCI layer per mise
tool. The rolling tag scheme is unchanged, so the workspace template
doesn't need updating.

Single-tool version bumps now invalidate only that tool's OCI layer, so
workspaces re-pull just what changed instead of the entire 5-6 GB image
on every recreate.

Also:

- Drops the build-time `pnpm dlx playwright@1.47.0 install --with-deps
chromium` step (~400 MB) and the equivalent `playwright-driver.browsers`
install from `flake.nix`. `@playwright/mcp` (used by the claude-code and
codex MCP servers in `dogfood/coder/main.tf`) does NOT auto-install
browsers, so the existing `install-deps` `coder_script` now runs two
installs on workspace start: `pnpm exec playwright install chromium` for
the site's pinned `@playwright/test`, and `npx
--package=@playwright/mcp@latest playwright-core install --no-shell
chromium` so the MCP servers find their matching browser revision.
Browser revisions coexist under
`~/.cache/ms-playwright/chromium-<rev>/`, which lives on the home volume
so both downloads happen once per workspace recreate and persist across
restarts. Net effect: same MCP behavior as before, +~1-2 min on first
workspace start. Nix devshell users running site e2e tests locally now
need `pnpm exec playwright install` once (instead of getting browsers
via nixpkgs).
- Bumps the pinned mise binary to v2026.5.12 (matching main after
#25521) and adds top-level `min_version = "2026.5.12"` to `mise.toml` so
every consumer (devs, CI, the embedded mise inside the dogfood image,
mise oci builds) fails fast on an older mise.
- Adds bison, flex, libicu-dev, libreadline-dev, uuid-dev, and
zlib1g-dev to both Ubuntu base images for source-build use cases (e.g.,
building Postgres from source).
- Replaces skopeo with crane as the registry client `mise oci push`
shells out to: crane is added to `mise.toml`, the workflow drops its
`apt-get install skopeo` and forces `--tool crane`, and the local
wrapper image stops bundling skopeo. One source of truth for tool
versions, no apt drift, smaller wrapper image, and workspace users get a
registry client on PATH for free via mise oci's tool layers.
- Removes `nix.hash`/`mise.hash` and their Makefile rules. The registry
digest already captures every effective change since CI rebuilds when
any baked-in input moves; the per-file `filesha1()` entries in
`pull_triggers` are redundant.

Supersedes #25400 (the `mise.hash` pull trigger landed there in
`2b612abe7b`; this PR removes it as part of the broader simplification).

> [!NOTE]
> `mise oci build` is experimental and requires `MISE_EXPERIMENTAL=1`
(set at job level in the workflow). The local-only
`scripts/dogfood/mise-oci-wrapper.sh` builds a tiny
`coderdev/mise-oci-wrapper:<version>` Debian image with curl-installed
mise on first invocation (cached by version tag thereafter); we don't
reuse `jdxcode/mise:latest` because that tag lags upstream GitHub
releases by days and would defeat the `min_version` enforcement above.

> [!NOTE]
> `compute-base-sha.sh` and `compute-final-sha.sh` are cache keys, not
strict content addresses: the base Dockerfile still pulls dynamic
resources at build time (gh/buildx `releases/latest`, chrome
`stable_current_amd64.deb`, apt mirror state). Two runs with identical
checked-in files can produce slightly different bytes, which is
acceptable here because the cache-hit savings on irrelevant commits
outweigh that drift.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Thomas Kosiewski
2026-05-26 14:52:21 +02:00
committed by GitHub
co-authored by Claude Opus 4.7
parent d8dc782da5
commit 51836e681e
18 changed files with 606 additions and 258 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/dogfood.yaml
+188 -97
View File
@@ -8,21 +8,24 @@ on:
#
# Effects vary by event:
#
# PRs: `build_image` builds the image variants but never pushes
# (each `depot/build-push-action` step's `push:` and the
# `Push Nix image` step are gated on `github.ref ==
# 'refs/heads/main'`). `test_image` rebuilds the Ubuntu images
# from Depot cache with `load: true` and runs `make gen`, `fmt`,
# `lint`, and a Linux build inside each image to validate that
# the baked-in tooling works. `deploy_template` runs
# `terraform init` + `validate` only; the apply step and
# SHA/title gathering are gated on main.
# PRs: `build_image` builds the base and runs `mise oci build`,
# loads the result into the local Docker daemon, and runs
# `make gen`, `fmt`, `lint`, and a Linux build inside the image
# to validate the baked-in tooling. Only the base image is pushed
# (to ghcr.io so the mise oci step can pull --from a real
# registry); the Docker Hub push is gated on
# `github.ref == 'refs/heads/main'`. Fork PRs skip the entire
# base+mise-oci pipeline since GITHUB_TOKEN is read-only for
# packages; the nix matrix entry still runs.
# `deploy_template` runs `terraform init` + `validate` only; the
# apply step and SHA/title gathering are gated on main.
#
# Pushes to main: `build_image` retags rolling tags on
# `codercom/oss-dogfood` (`:latest`, `:22.04`, `:26.04`),
# `codercom/oss-dogfood-vscode-coder` (`:latest`), and
# `codercom/oss-dogfood-nix` (`:latest`), plus a per-branch tag on
# each. `test_image` validates tooling as above.
# each. The image-tooling validation runs as above before any
# push, so a broken image never reaches Docker Hub.
# `deploy_template` runs `terraform apply` and creates new
# `coderd_template` versions on dev.coder.com whose `name` is the
# commit short SHA. Content is unchanged when neither `dogfood/**`
@@ -37,6 +40,8 @@ on:
- "flake.nix"
- "mise.toml"
- "mise.lock"
- "scripts/dogfood/**"
- "scripts/dogfood_test_image.sh"
pull_request:
paths:
- "dogfood/**"
@@ -45,6 +50,8 @@ on:
- "flake.nix"
- "mise.toml"
- "mise.lock"
- "scripts/dogfood/**"
- "scripts/dogfood_test_image.sh"
workflow_dispatch:
permissions:
@@ -58,7 +65,16 @@ jobs:
image-version: ["22.04", "26.04", "nix"]
if: github.actor != 'dependabot[bot]' # Skip Dependabot PRs
runs-on: ${{ github.repository_owner == 'coder' && 'depot-ubuntu-22.04-4' || 'ubuntu-latest' }}
runs-on: ${{ github.repository_owner == 'coder' && 'depot-ubuntu-22.04-8' || 'ubuntu-latest' }}
permissions:
contents: read
packages: write # push the dogfood base image to ghcr.io/coder/oss-dogfood-base
env:
# MISE_EXPERIMENTAL opts into the experimental `oci` subcommand.
# Trust is set via a config file (see the Install mise step
# below) rather than MISE_TRUSTED_CONFIG_PATHS so the workspace
# template can keep parity with the same file-based approach.
MISE_EXPERIMENTAL: "1"
steps:
- name: Harden Runner
uses: step-security/harden-runner@f808768d1510423e83855289c910610ca9b43176 # v2.17.0
@@ -119,6 +135,58 @@ jobs:
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
if: matrix.image-version != 'nix'
- name: Install mise
if: matrix.image-version != 'nix'
# MISE_VERSION + MISE_SHA256 match dogfood/coder/ubuntu-*/Dockerfile.base
# so the mise binary baking the image is the same one a workspace
# ships with. `min_version` in mise.toml catches downgrades.
# Write trust config to ~/.config/mise/conf.d/ instead of using
# MISE_TRUSTED_CONFIG_PATHS so the same file-based approach
# works in workspaces (where the user owns the file).
env:
MISE_VERSION: v2026.5.12
MISE_SHA256: a238972a3162d710b85b28c324372e96ca4e4b486c81fe78695000d9fbc77c48
WORKSPACE: ${{ github.workspace }}
run: |
set -euo pipefail
curl --silent --show-error --location --fail \
"https://github.com/jdx/mise/releases/download/${MISE_VERSION}/mise-${MISE_VERSION}-linux-x64" \
--output /tmp/mise
echo "${MISE_SHA256} /tmp/mise" | sha256sum -c
sudo install -m 0755 /tmp/mise /usr/local/bin/mise
rm /tmp/mise
mise --version
mkdir -p "$HOME/.config/mise/conf.d"
cat > "$HOME/.config/mise/conf.d/00-ci-trust.toml" <<EOF
[settings]
trusted_config_paths = ["$WORKSPACE"]
EOF
- name: Compute image SHAs
# Match the fork guard on the downstream consumers of these
# outputs: nothing reads `steps.shas.outputs.*` outside the
# base-push + mise-oci pipeline, which is gated below.
if: matrix.image-version != 'nix' && !github.event.pull_request.head.repo.fork
id: shas
env:
IMAGE_VERSION: ${{ matrix.image-version }}
run: |
base_sha="$(./scripts/dogfood/compute-base-sha.sh "$IMAGE_VERSION")"
final_sha="$(./scripts/dogfood/compute-final-sha.sh "$IMAGE_VERSION")"
echo "base_sha=${base_sha}" >> "$GITHUB_OUTPUT"
echo "final_sha=${final_sha}" >> "$GITHUB_OUTPUT"
- name: Login to GHCR
# Fork PRs get a read-only GITHUB_TOKEN that cannot push to
# ghcr.io. Skip the entire GHCR-dependent pipeline (base push +
# mise oci build) for fork PRs; the nix matrix entry still runs.
if: matrix.image-version != 'nix' && !github.event.pull_request.head.repo.fork
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Login to DockerHub
if: github.ref == 'refs/heads/main'
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
@@ -126,48 +194,122 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
- name: Build and push Ubuntu 22.04 image
- name: Build base image
uses: depot/build-push-action@5f3b3c2e5a00f0093de47f657aeaefcedff27d18 # v1.17.0
if: matrix.image-version != 'nix' && !github.event.pull_request.head.repo.fork
with:
project: b4q6ltmpzh
token: ${{ secrets.DEPOT_TOKEN }}
buildx-fallback: true
# Context is the repo root so the Dockerfile can COPY the
# project mise.toml that the image installs from. The
# github_token secret raises aqua's GitHub API quota during
# `mise install`.
# Context is the repo root so Dockerfile.base can COPY the
# distro-specific files/ tree and configure-chrome-flags.sh.
context: "{{defaultContext}}"
file: dogfood/coder/ubuntu-22.04/Dockerfile
secrets: |
github_token=${{ secrets.GITHUB_TOKEN }}
file: dogfood/coder/ubuntu-${{ matrix.image-version }}/Dockerfile.base
pull: true
save: true
push: ${{ github.ref == 'refs/heads/main' }}
# TODO: move the `latest` tag to 26.04 soon. we don't want to transition
# it immediately because that would make workspaces switch to it
# automatically without any grace period.
tags: "codercom/oss-dogfood:${{ steps.docker-tag-name.outputs.tag }},codercom/oss-dogfood:22.04,codercom/oss-dogfood:latest"
if: matrix.image-version == '22.04'
# Push to ghcr.io on every non-fork CI run so the downstream
# mise oci build can --from a real registry. The base-sha tag
# is a cache key (see scripts/dogfood/compute-base-sha.sh) so
# commits that don't change base inputs reuse the previous
# build.
push: true
tags: |
ghcr.io/coder/oss-dogfood-base:${{ matrix.image-version }}-${{ steps.shas.outputs.base_sha }}
ghcr.io/coder/oss-dogfood-base:${{ matrix.image-version }}-${{ steps.docker-tag-name.outputs.tag }}
- name: Build and push Ubuntu 26.04 image
uses: depot/build-push-action@5f3b3c2e5a00f0093de47f657aeaefcedff27d18 # v1.17.0
with:
project: b4q6ltmpzh
token: ${{ secrets.DEPOT_TOKEN }}
buildx-fallback: true
# Context is the repo root so the Dockerfile can COPY the
# project mise.toml that the image installs from. The
# github_token secret raises aqua's GitHub API quota during
# `mise install`.
context: "{{defaultContext}}"
file: dogfood/coder/ubuntu-26.04/Dockerfile
secrets: |
github_token=${{ secrets.GITHUB_TOKEN }}
pull: true
save: true
push: ${{ github.ref == 'refs/heads/main' }}
tags: "codercom/oss-dogfood:${{ steps.docker-tag-name.outputs.tag }},codercom/oss-dogfood:26.04"
if: matrix.image-version == '26.04'
- name: Install mise tools
if: matrix.image-version != 'nix' && !github.event.pull_request.head.repo.fork
# `mise oci build` packages already-installed tools into OCI
# layers; it does not install them. Run `mise install` first so
# the tools land in MISE_DATA_DIR on the runner.
# github_token raises aqua's API quota during tool installs.
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# --locked refuses to resolve URLs at install time and forces
# the runner to consume what mise.lock already committed,
# so a forgotten lockfile entry fails CI instead of silently
# being added on next run.
mise install --yes --locked
# Put mise's shims dir on PATH for subsequent steps so
# `mise oci push --tool crane` can find crane (and any other
# mise-managed binary it shells out to).
echo "$HOME/.local/share/mise/shims" >> "$GITHUB_PATH"
- name: Build mise oci layer
if: matrix.image-version != 'nix' && !github.event.pull_request.head.repo.fork
env:
IMAGE_VERSION: ${{ matrix.image-version }}
BASE_SHA: ${{ steps.shas.outputs.base_sha }}
FINAL_SHA: ${{ steps.shas.outputs.final_sha }}
# --output makes the OCI layout location explicit so the later
# `mise oci push --image-dir` steps point at the right path even
# if mise oci's default ever changes (it's experimental).
run: |
mise oci build \
--from "ghcr.io/coder/oss-dogfood-base:${IMAGE_VERSION}-${BASE_SHA}" \
--tag "codercom/oss-dogfood:${FINAL_SHA}-${IMAGE_VERSION}" \
--output ./mise-oci
# Load the OCI layout into the local Docker daemon so the next
# step can `docker run` it. crane lacks a direct OCI-layout-to-
# daemon command, but its built-in registry server gives us a
# simple two-hop path with no extra dependencies.
- name: Load mise oci image into Docker daemon
if: matrix.image-version != 'nix' && !github.event.pull_request.head.repo.fork
env:
IMAGE_VERSION: ${{ matrix.image-version }}
run: |
set -euo pipefail
crane registry serve --address localhost:5000 &
reg_pid=$!
trap 'kill $reg_pid 2>/dev/null || true' EXIT
for _ in 1 2 3 4 5; do
curl -sf http://localhost:5000/v2/ >/dev/null && break
sleep 1
done
crane push ./mise-oci "localhost:5000/dogfood-test:${IMAGE_VERSION}"
docker pull "localhost:5000/dogfood-test:${IMAGE_VERSION}"
docker tag "localhost:5000/dogfood-test:${IMAGE_VERSION}" "dogfood-test:${IMAGE_VERSION}"
# Validate the dogfood image's tooling by running make gen, fmt,
# lint, and a fat build inside it. Failures here block the
# Docker Hub push below so broken images never reach workspaces.
- name: Test image tooling
if: matrix.image-version != 'nix' && !github.event.pull_request.head.repo.fork
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: ./scripts/dogfood_test_image.sh "dogfood-test:${{ matrix.image-version }}"
- name: Push final Ubuntu 22.04 image
if: matrix.image-version == '22.04' && github.ref == 'refs/heads/main'
env:
FINAL_SHA: ${{ steps.shas.outputs.final_sha }}
DOCKER_TAG: ${{ steps.docker-tag-name.outputs.tag }}
# --image-dir points at the OCI layout written by the previous
# `mise oci build` step. Without it, `mise oci push` rebuilds
# from mise.toml and forgets the --from base. --tool crane
# forces the registry client mise oci shells out to, so we
# don't drift between the apt-shipped skopeo on whatever runner
# image we land on.
# TODO: move the `latest` tag to 26.04 soon. we don't want to
# transition it immediately because that would make workspaces
# switch to it automatically without any grace period.
run: |
set -euo pipefail
for tag in "${FINAL_SHA}-22.04" "$DOCKER_TAG" 22.04 latest; do
mise oci push --tool crane --image-dir ./mise-oci "codercom/oss-dogfood:$tag"
done
- name: Push final Ubuntu 26.04 image
if: matrix.image-version == '26.04' && github.ref == 'refs/heads/main'
env:
FINAL_SHA: ${{ steps.shas.outputs.final_sha }}
DOCKER_TAG: ${{ steps.docker-tag-name.outputs.tag }}
run: |
set -euo pipefail
for tag in "${FINAL_SHA}-26.04" "$DOCKER_TAG" 26.04; do
mise oci push --tool crane --image-dir ./mise-oci "codercom/oss-dogfood:$tag"
done
- name: Build and push vscode-coder image
uses: depot/build-push-action@5f3b3c2e5a00f0093de47f657aeaefcedff27d18 # v1.17.0
@@ -201,59 +343,8 @@ jobs:
env:
DOCKER_TAG: ${{ steps.docker-tag-name.outputs.tag }}
# Validate that the Ubuntu dogfood images contain working tooling.
# Failures here block template deployment (deploy_template).
test_image:
needs: build_image
strategy:
fail-fast: false
matrix:
image-version: ["22.04", "26.04"]
runs-on: ${{ github.repository_owner == 'coder' && 'depot-ubuntu-22.04-8' || 'ubuntu-latest' }}
steps:
- name: Harden Runner
uses: step-security/harden-runner@f808768d1510423e83855289c910610ca9b43176 # v2.17.0
with:
egress-policy: audit
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 1
persist-credentials: false
- name: Set up Depot CLI
uses: depot/setup-action@15c09a5f77a0840ad4bce955686522a257853461 # v1.7.1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
# Near-instant cache hit from build_image; loads into local daemon
# without pushing to a registry.
- name: Load dogfood image from Depot cache
uses: depot/build-push-action@5f3b3c2e5a00f0093de47f657aeaefcedff27d18 # v1.17.0
with:
project: b4q6ltmpzh
token: ${{ secrets.DEPOT_TOKEN }}
buildx-fallback: true
context: "{{defaultContext}}"
file: dogfood/coder/ubuntu-${{ matrix.image-version }}/Dockerfile
secrets: |
github_token=${{ secrets.GITHUB_TOKEN }}
pull: true
load: true
push: false
tags: "dogfood-test:${{ matrix.image-version }}"
- name: Test image tooling
run: ./scripts/dogfood_test_image.sh "dogfood-test:${{ matrix.image-version }}"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
deploy_template:
needs:
- build_image
- test_image
needs: build_image
runs-on: ubuntu-latest
permissions:
# Necessary for GCP authentication (https://github.com/google-github-actions/setup-gcloud#usage)