Files
sim/scripts/setup/env-files.ts
T
Theodore Li 19c3b6f47d feat(setup): setup wizard with browser-based Chat key handoff (#5911)
* feat(setup): setup wizard with browser-based Chat key handoff

Adds `bun run setup` and `bun run doctor` for local installs, and replaces
the wizard's paste-your-Chat-key step with a browser handoff that never puts
the key in a URL.

* improvement(setup): drop the paste-a-key fallback, simplify consent copy

The browser handoff is now the only path — the wizard waits on a spinner
instead of racing a paste prompt. Consent card leads with "Connect your
terminal" and moves the match-the-code disclaimer into the description.

* fix(setup): pin kube context, keep secrets out of argv, validate reused keys

Review findings from #5911:
- helm/kubectl now run against the validated context instead of the ambient one
- helm values are piped on stdin rather than passed as --set arguments
- ENCRYPTION_KEY/API_ENCRYPTION_KEY are checked for the 64-hex format the app
  requires, not just length, so an unusable key is replaced rather than kept
- the managed Redis container's published port is read back instead of assumed

* refactor(copilot): one module for Chat API key operations

list/generate/delete each repeated the same /api/validate-key envelope in
their route. They now share callValidateKey in lib/copilot/server/api-keys.ts,
which also keeps the display masking server-side so the full key can only ever
leave at creation.

* improvement(setup): reuse shared helpers, parallelize probes, drop dead code

- PKCE verifier/state/pairing code now use generateSecureToken, generateRandomHex
  and generateShortId instead of hand-rolled randomBytes; the pairing loop's
  modulo was unbiased only because 256 % 32 == 0
- new sha256Base64Url in @sim/security/hash so both sides of the PKCE exchange
  derive the challenge from one implementation
- isUsableSecret moved beside SECRET_KEYS so setup and doctor apply the same
  rule; doctor previously passed a key setup would replace
- isTruthy narrowed to true/1, matching the app it claims to mirror — it accepted
  yes/on, so a flag could read on in doctor and off in the app
- checkLive runs its five probes concurrently (~17s serial worst case)
- detection overlaps the banner animation instead of queueing behind it
- glyph.fail/glyph.warn at 13 sites that bypassed the constant; removed unused
  prompter exports, a dead ENV_PATHS re-export, and an unused export keyword

* fix(setup): make doctor understand the compose env layout

Compose writes a single root .env (what docker-compose reads via env_file) but
the checks required the three per-app files, so a successful compose install was
followed by doctor printing three failures and exiting 1 — and the whole
coherence catalog was skipped because it keyed off apps/sim/.env existing.

Layout is now derived from what's on disk and every check consults it: file and
schema checks iterate the layout's targets, consistency reports skip when
there's only one file to mirror, and coherence/live read the layout's primary
file. The wizard's existing-config detection counts root for the same reason —
a compose install used to read as unconfigured and re-run from scratch.

* feat(cli-auth): device-authorization poll flow, drop the loopback listener

The CLI no longer binds a local port. It generates a request id + poll secret,
opens /cli/auth, and polls /api/cli/auth/poll over TLS while the user approves
in the browser — so the flow works over SSH and inside containers, where the
browser and terminal don't share a machine.

- approve stores the approval keyed by request id (session-authed, userId from
  the session only); poll verifies the secret before an atomic claim, so an
  observer of the semi-public request id can neither mint nor cancel it
- pairing code stays as the anti-phishing compare; no key ever crosses the
  browser; done page just confirms
- removes the loopback listener, /token exchange, buildCliHandoffUrl, and
  validateCliCallbackUrl (+ its tests) — nothing hands a key to a URL anymore

* fix(setup): reuse an existing managed Postgres container instead of colliding

A running sim-postgres fell through to `docker run --name sim-postgres` and died
on the name conflict; a stopped one failed with "no DATABASE_URL to reach it"
because the generated password only lived in the env files a fresh clone lacks.

Both facts are recoverable from Docker: the ladder now reads the published port
and password back via `docker inspect` and reuses the container (starting it if
stopped). A container that won't answer prompts before recreating, and never
drops the data volume silently.

* improvement(setup): audience-first run-mode hints

Each run mode now names who it's for — compose for self-hosting/evaluating, dev
for contributing to Sim, k8s for rehearsing a production deploy — with the live
detection state (Docker/kube/VM) appended.

* fix(cli-auth): retry a failed mint, port container/port fixes to Redis + k8s

Review findings from #5911:
- poll now reserves the mint with an atomic NX lock instead of deleting the
  approval up front, so a failed mint (e.g. mothership blip) is retried by the
  next poll instead of forcing a fresh browser approval; the lock still prevents
  a double-mint and its TTL frees the slot if the caller dies
- setup reuses/recreates an unhealthy managed sim-redis instead of colliding on
  the name (Redis has no data volume, so it removes and recreates without a prompt)
- k8s failure-path hints carry --context, matching the success-path hints, so a
  changed ambient context can't send diagnostics to the wrong cluster
- compose port-free waits for a killed port to actually release before
  re-checking; SIGKILL is async, so the immediate re-check re-saw the port

* fix(setup): harden mint cleanup, Windows browser, container detection, helm cwd

Review findings from #5911:
- a post-mint completeApproval failure no longer routes into releaseMint — the
  mint lock now outlives the approval (shared TTL), so a cleanup blip can't leave
  a re-mintable window and orphan a key; cleanup is best-effort after the key ships
- compose doctor --fix writes the feature-flag twin to the layout's primary env
  (root .env on a compose install), not always apps/sim/.env
- Windows opens the browser via `cmd /c start "" <url>` — `start` is a shell
  builtin, so spawning it directly ENOENT'd and the handoff never opened
- managed-container detection filters loosely and pins the exact name in code;
  Docker's `name=^x$` anchor matches the internal `/x` form and often missed,
  skipping the reuse branch
- the shared helm/kind run helper pins cwd to the repo root, matching helm test,
  so `helm upgrade --install ./helm/sim` works from any working directory

* feat(chat-keys): standalone manage page, drop from settings nav, refresh README

- Add /account/settings/chat-keys — a linkable page to view, create, and revoke Chat API keys
- Remove Chat keys from the settings sidebar (account + unified nav) and its render branches
- README: replace Docker Compose + Manual Setup with the bun run setup wizard; drop the manual COPILOT_API_KEY step, point to the manage page

* fix(setup): per-key reason in the secret-replacement warning

Cursor: the warn hardcoded '64-character hex key', but only ENCRYPTION_KEY/API_ENCRYPTION_KEY require that — BETTER_AUTH_SECRET/INTERNAL_API_SECRET only need length >= 32. Use the existing secretRequirement(key) helper so each replaced key reports its actual requirement.

* fix(setup): compose doctor schema, cross-platform binary detection, quoted context hints

- Doctor: for the compose (root) env layout, require only the secrets compose has no interpolation default for (BETTER_AUTH_SECRET/ENCRYPTION_KEY/INTERNAL_API_SECRET). DATABASE_URL/BETTER_AUTH_URL/NEXT_PUBLIC_APP_URL come from docker-compose ${VAR:-default}, so a healthy compose install no longer fails doctor.
- Binary detection: use Bun.which instead of which (which is absent on Windows), so kubectl/helm/kind/docker resolve cross-platform.
- k8s diagnostic hints: POSIX-quote the kube-context so a context with whitespace/metacharacters can't break or inject into a copied command.

* fix(setup): quote kube-context in the helm uninstall tear-down hint too

The tear-down hint used --kube-context ${context} raw while the sibling kubectl hints already used shq(); a context with whitespace/metacharacters could break or inject into the copied command. All copyable k8s hints now go through shq(context).

* feat(setup): sim lifecycle CLI — start/stop/status/logs/down/reset

Turn the setup entry into a 'sim' command umbrella so there's one place to run everything, not scattered docker/bun commands. Adds a global bin (bun link) + a bun run sim fallback.

- Detects how you're running (compose file / managed dev containers / helm release) from disk + docker/helm state — no persisted mode. Ambiguous installs prompt.
- start/stop/restart/logs work per mode; down removes containers (volumes kept); reset archives .env + wipes managed data; both destructive verbs confirm first.
- status shows detected mode, container states, and app/realtime health.
- Wizard outro + README now point at the sim commands and the one-time bun link.

* feat(setup): 'bun run sim' is the primary entry; bare invocation prints help

- Lead usage/wizard-outro/README with 'bun run sim <cmd>' (works with zero PATH setup); global bare 'sim' via bun link is an optional upgrade, with the ~/.bun/bin PATH caveat spelled out (Homebrew's bun omits it).
- Bare 'sim' now prints help instead of launching the wizard; the wizard is 'sim setup'. The 'setup' npm script passes the keyword so 'bun run setup' is unchanged.

* fix(setup): quote the auth URL for cmd /c start on Windows

Cursor (High): cmd re-parses the command line and treats & in the query string as a command separator, so cmd /c start opened a URL truncated at the first &, breaking the key flow on win32 (the handoff URL always has request/challenge/pairing). Quote the URL and pass args verbatim so & stays literal.

* fix(setup): verify kube-context is really local; lengthen CLI handoff wait

- k8s: a context named like a local cluster (kind-*, docker-desktop) can actually point at a remote API server. Verify the server host is loopback/docker-internal before defaulting the 'use this context?' confirm to yes; otherwise warn and default to no, so generated secrets can't ship to a remote cluster on a blind Enter.
- cli-auth: bump the device-flow wait from 3 to 15 minutes so first-time users have time to sign up, wait for the email OTP, and approve before the terminal stops polling. The server-side approval record keeps its own short TTL, so a longer client wait only costs cheap rate-limited polls.

* fix(setup): only manage k8s lifecycle on a verified-local context

Greptile: sim down/reset used the ambient kube-context, so switching context after setup could uninstall a same-named sim-dev release from the wrong cluster. Gate k8sInstall on the same locality check the wizard uses (API server is loopback/docker-internal) via a shared isLocalKubeContext helper — the wizard only ever deploys locally, so a remote current-context is never treated as a Sim install.

* fix(setup): doctor skips placeholder secrets when seeding; reset names its target

- checks: the missing-file autofix copied shared keys from apps/sim/.env whenever truthy, including .env.example placeholders — doctor --fix could seed unusable secrets into realtime/db env files. Skip placeholders, matching autofixForMissing.
- lifecycle: reset now names the exact install (k8s context / compose file / dev containers) in its confirm, so a destructive reset can't silently hit the wrong same-named install after a context switch (down already names the context).

* fix(cli-auth): size the poll rate limit to the poll cadence; honor Retry-After

The poll route used the default public-IP bucket (10 burst, 5/min) but the CLI polls every 2s (30/min), so it 429'd within ~20s — worse behind a slow dev cold-compile. Give the endpoint a bucket matched to its cadence (60 burst, 60/min); it's not a brute-force surface (unknown request id returns pending, minting needs the 256-bit verifier). Also make the CLI honor Retry-After and back off on 429 so a shared-NAT per-IP limit degrades gracefully instead of hammering.

* fix(setup): check ports before starting the dev server, not just compose

Local dev auto-start spawned bun run dev:full with no port check, so it silently started a server that couldn't bind when 3000/3002 were already taken (e.g. another worktree's dev server). Extract compose's port-conflict resolver into a shared ensurePortsFree(ports) and run it before the dev start too — kill/recheck/leave, same as compose. Leaving the ports skips the auto-start with guidance instead of failing; compose still treats it as fatal.

* fix(setup): verify the kube cluster is reachable, not just local

A kubeconfig context can outlive its cluster — a kind cluster gets deleted or its Docker container stops (Docker/machine restart), but the context entry remains, pointing at a dead API-server port. The wizard checked the context looked local and handed it to helm, which failed with 'cluster unreachable'.

Add a clusterReachable() liveness probe: only offer the current context when it actually answers; if a local context is dead, fall through to the kind path. There, if kind still knows 'sim' but it's stopped, start its node containers and wait for the API; if it's gone, create fresh. Either way the user gets a working cluster instead of a cryptic helm failure.

* fix(helm): point appVersion at published image tags (v-prefixed, current)

The chart's appVersion was "0.6.73", but CI publishes GHCR tags with a v prefix (its release-commit regex captures v0.7.45). Since sim.image defaults every image tag to Chart.AppVersion, a default helm install requested ghcr.io/simstudioai/{simstudio,realtime,migrations}:0.6.73 — a tag that has never existed — so app and realtime sat in ImagePullBackOff and helm --wait failed with 'progress deadline exceeded'. Any self-hoster installing with default values hit this, not just the setup wizard.

Set appVersion to v0.7.45 (latest release on main; all three images verified present on ghcr) and bump the chart version to 1.1.1. Verified with helm lint, helm template (all images render as v0.7.45), and a live helm upgrade on a kind cluster where the new pods pull successfully while the old 0.6.73 pods remain in ImagePullBackOff.

* Revert "fix(helm): point appVersion at published image tags (v-prefixed, current)"

This reverts commit 28b6047d1d.

* chore(api-validation): rebaseline route count to 977 after staging merge

Staging moved the baseline to 975; this branch's two CLI-auth routes (approve, poll) make 977. The clean merge absorbed the earlier +2 adjustment.

* fix(settings): don't highlight a sibling nav item on nested settings pages

/account/settings/chat-keys is a real page but deliberately not a nav item, so the sidebar's parseSettingsPathSection fell through to defaultSection ('general') and highlighted General — the page read as though it lived inside General.

Resolve the sidebar's active item with a null default so an unmatched nested route highlights nothing, and widen SettingsSidebar's activeSection to string | null. The section feeding the title/description provider keeps its default (pages override title/description anyway), and /account/settings/billing/credit-usage still correctly highlights Billing.

* fix(setup,auth): manage explicitly-confirmed k8s contexts, fail loudly on reset, clear stale post-auth redirect

- lifecycle: detection is now factual — a sim-dev release either exists on the current context or it doesn't. Gating on locality stranded a release the user explicitly confirmed during setup (status/start/stop/down/reset all claimed no k8s install). Locality is recorded instead and surfaced through describeInstall, which every destructive confirm renders, so acting on a non-local cluster is named and defaulted to no rather than silently blocked or silently allowed.
- lifecycle: reset no longer discards helm uninstall's exit status. Env files are archived by that point, so claiming 'Reset complete' while the release still runs is the worst outcome — it now throws with retry/inspect commands.
- auth: signup clears POST_AUTH_REDIRECT_STORAGE_KEY when it has no callbackUrl, and the verification-disabled path consumes it, so a stale CLI/invite destination can't leak into a later flow in the same tab.
2026-07-25 04:24:36 -04:00

176 lines
6.0 KiB
TypeScript

import { existsSync, readFileSync, renameSync, writeFileSync } from 'node:fs'
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import { generateRandomHex } from '@sim/utils/random'
export const ROOT = path.resolve(fileURLToPath(new URL('.', import.meta.url)), '../..')
export type EnvTarget = 'sim' | 'realtime' | 'db' | 'root'
export const ENV_PATHS: Record<EnvTarget, string> = {
sim: path.join(ROOT, 'apps/sim/.env'),
realtime: path.join(ROOT, 'apps/realtime/.env'),
db: path.join(ROOT, 'packages/db/.env'),
root: path.join(ROOT, '.env'),
}
const EXAMPLE_PATHS: Partial<Record<EnvTarget, string>> = {
sim: path.join(ROOT, 'apps/sim/.env.example'),
realtime: path.join(ROOT, 'apps/realtime/.env.example'),
db: path.join(ROOT, 'packages/db/.env.example'),
}
/** Keys that must be byte-identical between apps/sim/.env and apps/realtime/.env. */
export const SHARED_KEYS = [
'DATABASE_URL',
'BETTER_AUTH_SECRET',
'INTERNAL_API_SECRET',
'BETTER_AUTH_URL',
'NEXT_PUBLIC_APP_URL',
] as const
export const SECRET_KEYS = [
'BETTER_AUTH_SECRET',
'ENCRYPTION_KEY',
'INTERNAL_API_SECRET',
'API_ENCRYPTION_KEY',
] as const
const PLACEHOLDER_VALUES = new Set([
'your_password',
'your_secret_key',
'your_encryption_key',
'your_internal_api_secret',
'your_api_encryption_key',
'your_better_auth_secret_min_32_chars',
'dev-secret-at-least-32-characters-long',
'dev-encryption-key-at-least-32-chars',
'dev-internal-api-secret-min-32-chars',
])
export interface EnvFile {
target: EnvTarget
path: string
exists: boolean
content: string
vars: Map<string, string>
}
const LINE_RE = /^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$/
function parseValue(raw: string): string {
const trimmed = raw.trim()
if (trimmed.startsWith('"') || trimmed.startsWith("'")) {
const quote = trimmed[0]
const end = trimmed.indexOf(quote, 1)
return end === -1 ? trimmed.slice(1) : trimmed.slice(1, end)
}
return trimmed.replace(/\s+#.*$/, '').trim()
}
export function parseEnv(content: string): Map<string, string> {
const vars = new Map<string, string>()
for (const line of content.split('\n')) {
const match = LINE_RE.exec(line)
if (match && !vars.has(match[1])) vars.set(match[1], parseValue(match[2]))
}
return vars
}
export function readEnvFile(target: EnvTarget): EnvFile {
const filePath = ENV_PATHS[target]
const exists = existsSync(filePath)
const content = exists ? readFileSync(filePath, 'utf8') : ''
return { target, path: filePath, exists, content, vars: parseEnv(content) }
}
/**
* Sets a key in env-file content: replaces the active line, uncomments a
* commented-out line, or appends. Returns the new content.
*/
export function upsertEnv(content: string, key: string, value: string): string {
const lines = content.split('\n')
const activeRe = new RegExp(`^\\s*(?:export\\s+)?${key}\\s*=`)
const commentedRe = new RegExp(`^#\\s*${key}\\s*=`)
const activeIdx = lines.findIndex((l) => activeRe.test(l))
const idx = activeIdx !== -1 ? activeIdx : lines.findIndex((l) => commentedRe.test(l))
const newLine = `${key}=${value}`
if (idx === -1) {
const trailing = lines.length > 0 && lines[lines.length - 1] === ''
if (trailing) lines.splice(lines.length - 1, 0, newLine)
else lines.push(newLine)
} else {
lines[idx] = newLine
}
return lines.join('\n')
}
/** Writes values into an env file, seeding a missing file from its .env.example. */
export function writeEnvValues(target: EnvTarget, values: Record<string, string>): void {
const filePath = ENV_PATHS[target]
let content: string
if (existsSync(filePath)) {
content = readFileSync(filePath, 'utf8')
} else {
const example = EXAMPLE_PATHS[target]
content = example && existsSync(example) ? readFileSync(example, 'utf8') : ''
}
for (const [key, value] of Object.entries(values)) {
content = upsertEnv(content, key, value)
}
writeFileSync(filePath, content)
}
export function archiveEnvFile(target: EnvTarget): string | null {
const filePath = ENV_PATHS[target]
if (!existsSync(filePath)) return null
const backup = `${filePath}.bak-${new Date().toISOString().replace(/[:.]/g, '-')}`
renameSync(filePath, backup)
return backup
}
export function generateSecret(): string {
return generateRandomHex(64)
}
/**
* `ENCRYPTION_KEY` and `API_ENCRYPTION_KEY` are read as raw AES-256 material,
* so the app requires exactly 64 hex characters and throws on anything else
* (`lib/core/security/encryption.ts`, `lib/api-key/crypto.ts`). A merely-long
* passphrase passes a length check and then fails every encryption path at
* runtime, so those two are validated on format rather than length.
*
* Lives here so setup (which replaces an unusable secret) and doctor (which
* reports one) apply the same rule — they disagreed while it was duplicated.
*/
const HEX_KEY_PATTERN = /^[0-9a-f]{64}$/i
const HEX_SECRET_KEYS = new Set<string>(['ENCRYPTION_KEY', 'API_ENCRYPTION_KEY'])
export function isUsableSecret(key: string, value: string): boolean {
if (isPlaceholder(value)) return false
return HEX_SECRET_KEYS.has(key) ? HEX_KEY_PATTERN.test(value) : value.length >= 32
}
/** Human-readable reason a secret is unusable, for doctor's finding message. */
export function secretRequirement(key: string): string {
return HEX_SECRET_KEYS.has(key)
? 'must be exactly 64 hex characters (32-byte AES key)'
: 'must be at least 32 characters'
}
export function isPlaceholder(value: string): boolean {
return PLACEHOLDER_VALUES.has(value) || value.startsWith('your_')
}
/**
* Mirrors the app's `isTruthy` (apps/sim/lib/core/config/env.ts:633) exactly —
* `true` or `1` only. The app's separate `envBoolean` additionally accepts
* `yes`/`on`, but feature flags read through `isTruthy`, so accepting the wider
* set here made the wizard and doctor report a flag as on that the app treats
* as off.
*/
export function isTruthy(value: string | undefined): boolean {
if (value === undefined) return false
return value.toLowerCase() === 'true' || value === '1'
}