From d3986b53b908d784842c750c674d695442560e80 Mon Sep 17 00:00:00 2001 From: Kyle Carberry Date: Tue, 10 Mar 2026 08:54:32 -0700 Subject: [PATCH] perf(ci): use fast zstd compression for non-release CI builds (#22907) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Problem The `build` job on `main` takes ~7m28s for the Build step alone (~13m total). Analysis of 10 recent CI runs on `main` shows the zstd compression of the slim binary archive is the second largest bottleneck: | Phase | Avg Duration | % of Build Step | |-------|-------------|----------------| | Fat Go builds (7 binaries w/ embed) | ~205s | 45.8% | | **zstd compression (`-22 --ultra`)** | **~123s** | **27.4%** | | Parallel block (vite + slim Go builds) | ~65s | 14.5% | | Packaging + signing | ~55s | 12.3% | The `zstd -22 --ultra` setting compresses a ~350 MB tar to ~71 MB, but it is **single-threaded** and takes ~102s on 8-core CI runners. Adding `-T8` does not help at level 22 — it remains CPU-bound on a single thread. ## Solution Use `zstd -6 -T0` (multithreaded, auto-detect cores) for non-release CI builds. Release builds (`CODER_RELEASE=true`) continue using `-22 --ultra`. ### Benchmarks (349 MB slim binary tar, 8 cores) | Setting | Wall Time | Output Size | Use Case | |---------|----------|------------|----------| | `-22 --ultra` | **102.4s** | 71 MB | Release builds | | `-6 -T0` | **0.8s** | 94 MB | CI builds (new) | | `-6` | 2.4s | 94 MB | Local dev (unchanged) | The 23 MB size increase is negligible for the main branch preview images (`ghcr.io/coder/coder-preview:main`). The archive is embedded in fat binaries and extracted once by the agent at startup — decompression time is identical regardless of compression ratio. ### Expected impact ~120s savings on the Build step, bringing it from ~7m28s to ~5m30s. ## Verification All three code paths confirmed: - `CODER_RELEASE=true CI=true` → `-22 --ultra` ✅ - `CI=true` (no `CODER_RELEASE`) → `-6 -T0` ✅ - Local (no `CI`) → `-6` ✅ - `CODER_RELEASE=false CI=true` (dry run) → `-6 -T0` ✅ --- Makefile | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 2421a6ddf7..f6f9f21a47 100644 --- a/Makefile +++ b/Makefile @@ -118,11 +118,14 @@ POSTGRES_IMAGE ?= us-docker.pkg.dev/coder-v2-images-public/public/postgres:$(P # parallelism. Override: make pre-push PARALLEL_JOBS=8 PARALLEL_JOBS ?= $(shell n=$$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 8); echo $$(( n / 4 > 2 ? n / 4 : 2 ))) -# Use the highest ZSTD compression level in CI. -ifdef CI +# Use the highest ZSTD compression level in release builds to +# minimize artifact size. For non-release CI builds (e.g. main +# branch preview), use multithreaded level 6 which is ~99% faster +# at the cost of ~30% larger archives. +ifeq ($(CODER_RELEASE),true) ZSTDFLAGS := -22 --ultra else -ZSTDFLAGS := -6 +ZSTDFLAGS := -6 -T0 endif # Common paths to exclude from find commands, this rule is written so