From d48722a04edd1b1dbc189d031c7f49a84073aa79 Mon Sep 17 00:00:00 2001 From: Waleed Date: Thu, 23 Jul 2026 18:28:11 -0700 Subject: [PATCH] fix(helm): correct chart docs, examples, and dead config across the board (#5907) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(helm): correct chart docs, examples, and dead config across the board Audit-driven accuracy pass over the chart's entire documentation surface, verified by rendering every example against the templates: - migrations run as an init container on the app pod, not a Job — fix the README component list, troubleshooting commands, and sim-helm skill refs; drop the dead migrations-job NetworkPolicy ingress rule - referenced-but-never-created resources: document the GKE ManagedCertificate creation (values-gcp), comment out the key-file Secret mount that stuck all pods in ContainerCreating (values-gcp), enable certManager for the postgres TLS issuerRef (values-production), add the cert-manager cluster-issuer annotation nginx needs (values-azure) - values-external-db: networkPolicy.egress is a list, not a map (the map rendered an invalid manifest); fill schema-failing placeholder host/username - realtime >1 replica requires REDIS_URL (Socket.IO Redis adapter) — default examples to 1 replica with the scaling note, and warn where autoscaling HPAs override replicaCount - pod anti-affinity selectors matched nothing (simstudio vs sim name label) - kubernetes.mdx: install commands were missing required CRON_SECRET and postgresql password (failed at template time), wrong deployment name in port-forward, stale version requirements, unsupported key-remapping claim - remove unimplemented app.secrets.existingSecret.keys from values + schema; fix README PDB default, cronjob list, /metrics caveat, NOTES secret count, Azure-only StorageClass in generic examples, dead SOCKET_SERVER_URL and GOOGLE_CLOUD_* env, ESO apiVersion mismatch, and skill-reference drift - bump chart to 1.0.1 * fix(helm): review round 1 — scoped example egress, in-tab secret note, copilot Job wording - external-db example egress scopes to a placeholder database CIDR instead of to: [] (which allowed every destination on 5432, defeating the isolation the example teaches) - kubernetes.mdx cloud tabs state explicitly that they reuse the variables generated in the Installation block - Copilot migrations really do run as a Helm-hook Job — restore Job wording there (only the app migrations are an init container) * fix(helm): template-sweep fixes — telemetry validity, ESO rollout checksums, dead passwordKey knob - telemetry: memory_limiter gets the required check_interval (collector failed startup validation whenever telemetry.enabled=true); the jaeger exporter was removed from collector-contrib in v0.86 — export to Jaeger via its native OTLP endpoint instead (otlp/jaeger, default port 4317) - app/realtime rollout checksums now hash the ExternalSecret manifest too, mirroring the copilot pattern — with ESO enabled the inline Secret renders empty, so remoteRefs changes never rolled the pods - remove the unimplemented existingSecret.passwordKey knob (values, schema, README, dead helpers): nothing consumed it, and a non-default value silently produced a DATABASE_URL with an unexpandable placeholder; secrets must use the standard POSTGRES_PASSWORD / EXTERNAL_DB_PASSWORD keys - drop the orphaned sim.migrations.labels helper (its only consumer was the dead NetworkPolicy rule removed earlier) - helm test pod image resolves through sim.image so global.imageRegistry mirroring applies; NetworkPolicy realtime-ingress comment reflects actual traffic direction; smoke unittest suite loads the newly referenced external-secret template * chore(helm): bump chart to 1.1.0 with upgrade notes Removing (inert) documented values keys and changing the rollout-checksum inputs is a values-surface change — per SemVer chart conventions that is more than a patch. Adds an Upgrading section documenting the one-time pod roll, the removed no-op keys, and the Jaeger-over-OTLP change. * fix(helm): review round — external-db NP opt-in with real-CIDR-first flow, prod Jaeger OTLP endpoint - external-db example ships networkPolicy disabled so a verbatim install always reaches the database; the scoped egress rule stays as the documented opt-in (set your CIDR first, then enable) - values-production still pointed telemetry.jaeger at the legacy 14250 collector port — now Jaeger's OTLP gRPC endpoint to match the otlp/jaeger exporter * feat(helm): autoscaling.realtime.enabled toggle so examples can scale the app without unsafe realtime replicas Cursor correctly flagged that comment-level warnings didn't stop a verbatim production/external-db install from running the realtime HPA at minReplicas 2 without REDIS_URL (silent cross-pod event loss). Adds an opt-out toggle (default true — existing deployments unchanged): the realtime HPA renders only when autoscaling.realtime.enabled, and the realtime Deployment keeps spec.replicas under its control when the HPA is excluded. The three autoscaling examples set it false with the Redis rationale; README and upgrade notes document the toggle. * fix(helm): review round — whitelabeled realtime HPA opt-out, external-db isolation on with required CIDR in install flow - values-whitelabeled now actually sets autoscaling.realtime.enabled: false (the earlier batch aborted before reaching this file — Cursor caught it) - external-db keeps networkPolicy enabled (no isolation regression); the DB egress CIDR is marked REQUIRED and wired into both documented install commands via --set, so the copy-paste flow sets the real subnet in the same breath as the DB host * fix(helm): use a syntactically valid example CIDR in the external-db install commands An unreplaced literal fails Kubernetes CIDR validation and aborts the install; every sibling placeholder in the same command (host, username) installs fine and simply doesn't connect until replaced. The CIDR now behaves the same way: valid example value (10.20.0.0/24), explicitly marked as the operator's database subnet in both commands and the values comment. * fix(helm): remove text after continuation backslashes in external-db install commands A trailing comment after the line-continuation backslash (and equally a comment line spliced mid-command) breaks the shell command when copied — the remaining --set overrides run as separate commands and required-value validation fails. Both documented commands now reconstruct to bash-clean multi-line invocations (verified with bash -n); the CIDR guidance lives in the networkPolicy section comment. * fix(docs): cloud-tab installs are alternatives via helm upgrade --install Following Installation and then a cloud tab ran helm install twice for the same release and failed on the second. The tabs now state they replace the generic install and use helm upgrade --install, which is idempotent and also converts an existing generic install to the cloud values. * fix(docs): honest conversion caveat for cloud-tab upgrades over an existing install The cloud values rename the bundled Postgres database to simstudio, which Postgres only applies at first initialization — an in-place conversion of a generic install would point DATABASE_URL at a nonexistent database. Document the two safe paths: keep the original name via --set, or uninstall + delete PVCs and install fresh. * fix(helm): external-db header install command declares its secrets and includes CRON_SECRET The primary documented command failed the chart's required-value validation (missing app.env.CRON_SECRET with cronjobs default-on) and referenced an undeclared DB_PASSWORD. It now shows the export lines for every variable it uses and sets CRON_SECRET; both commands verified with bash -n. * chore(helm): restore values.schema.json formatting — surgical deletions only The earlier programmatic edit reformatted the whole file (~390 lines of whitespace churn hiding the 12 real deleted lines). Re-applied the removal of the dead keys/passwordKey properties as text-level deletions preserving the original style. * fix(helm+docs): explicit secret exports in external-db header; conversion must reuse original secrets - all five export lines are written out (three were only named in a trailing comment, so a verbatim copy passed empty required values) - the cloud-conversion caveat now leads with reusing the original secret values (helm get values) — a regenerated ENCRYPTION_KEY makes previously encrypted credentials undecryptable --- .../en/platform/self-hosting/kubernetes.mdx | 26 +++++-- helm/sim/.claude/skills/sim-helm/SKILL.md | 4 +- .../skills/sim-helm/references/secrets.md | 2 +- .../sim-helm/references/troubleshooting.md | 12 +-- .../sim-helm/references/values-model.md | 4 +- helm/sim/Chart.yaml | 2 +- helm/sim/README.md | 25 +++--- helm/sim/examples/values-aws.yaml | 19 +++-- helm/sim/examples/values-azure.yaml | 14 +++- helm/sim/examples/values-copilot.yaml | 2 +- helm/sim/examples/values-existing-secret.yaml | 8 +- helm/sim/examples/values-external-db.yaml | 48 +++++++++--- .../sim/examples/values-external-secrets.yaml | 12 ++- helm/sim/examples/values-gcp.yaml | 77 ++++++++++++------- helm/sim/examples/values-production.yaml | 31 ++++++-- helm/sim/examples/values-whitelabeled.yaml | 8 +- helm/sim/templates/NOTES.txt | 2 +- helm/sim/templates/_helpers.tpl | 32 -------- helm/sim/templates/deployment-app.yaml | 7 +- helm/sim/templates/deployment-realtime.yaml | 11 ++- helm/sim/templates/hpa.yaml | 2 +- helm/sim/templates/networkpolicy.yaml | 14 +--- helm/sim/templates/telemetry.yaml | 7 +- helm/sim/templates/tests/test-connection.yaml | 2 +- helm/sim/tests/smoke_test.yaml | 2 + helm/sim/values.schema.json | 12 --- helm/sim/values.yaml | 43 ++++++----- 27 files changed, 254 insertions(+), 174 deletions(-) diff --git a/apps/docs/content/docs/en/platform/self-hosting/kubernetes.mdx b/apps/docs/content/docs/en/platform/self-hosting/kubernetes.mdx index 33e2f1e7e1..ab15e327d4 100644 --- a/apps/docs/content/docs/en/platform/self-hosting/kubernetes.mdx +++ b/apps/docs/content/docs/en/platform/self-hosting/kubernetes.mdx @@ -9,8 +9,8 @@ import { FAQ } from '@/components/ui/faq' ## Prerequisites -- Kubernetes 1.19+ -- Helm 3.0+ +- Kubernetes 1.25+ +- Helm 3.8+ - PV provisioner support ## Installation @@ -23,47 +23,59 @@ git clone https://github.com/simstudioai/sim.git && cd sim BETTER_AUTH_SECRET=$(openssl rand -hex 32) ENCRYPTION_KEY=$(openssl rand -hex 32) INTERNAL_API_SECRET=$(openssl rand -hex 32) +CRON_SECRET=$(openssl rand -hex 32) +POSTGRES_PASSWORD=$(openssl rand -hex 24) # Install helm install sim ./helm/sim \ --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \ --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \ --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \ + --set app.env.CRON_SECRET="$CRON_SECRET" \ + --set postgresql.auth.password="$POSTGRES_PASSWORD" \ --namespace simstudio --create-namespace ``` ## Cloud-Specific Values +These are cloud-tuned **alternatives** to the generic install above — pick one path, don't run both. The commands reuse the `$BETTER_AUTH_SECRET`, `$ENCRYPTION_KEY`, `$INTERNAL_API_SECRET`, `$CRON_SECRET`, and `$POSTGRES_PASSWORD` variables generated in [Installation](#installation) above, so run that block's `openssl` lines first in the same shell. They use `helm upgrade --install`, so they work whether or not a release exists yet. Two caveats when converting an existing generic install rather than starting fresh: (1) **reuse the original secret values** — recover them with `helm get values sim -n simstudio` if your shell no longer has them; supplying a newly generated `ENCRYPTION_KEY` makes every previously encrypted credential (OAuth tokens, provider keys, environment variables) undecryptable. (2) The cloud values rename the bundled PostgreSQL database to `simstudio`, but Postgres only applies that setting on first initialization — add `--set postgresql.auth.database=sim` to keep your existing database. If you'd rather start clean, `helm uninstall sim -n simstudio`, delete its PVCs, and run the cloud command fresh. + ```bash -helm install sim ./helm/sim \ +helm upgrade --install sim ./helm/sim \ --values ./helm/sim/examples/values-aws.yaml \ --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \ --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \ --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \ + --set app.env.CRON_SECRET="$CRON_SECRET" \ + --set postgresql.auth.password="$POSTGRES_PASSWORD" \ --set app.env.NEXT_PUBLIC_APP_URL="https://sim.yourdomain.com" \ --namespace simstudio --create-namespace ``` ```bash -helm install sim ./helm/sim \ +helm upgrade --install sim ./helm/sim \ --values ./helm/sim/examples/values-azure.yaml \ --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \ --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \ --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \ + --set app.env.CRON_SECRET="$CRON_SECRET" \ + --set postgresql.auth.password="$POSTGRES_PASSWORD" \ --set app.env.NEXT_PUBLIC_APP_URL="https://sim.yourdomain.com" \ --namespace simstudio --create-namespace ``` ```bash -helm install sim ./helm/sim \ +helm upgrade --install sim ./helm/sim \ --values ./helm/sim/examples/values-gcp.yaml \ --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \ --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \ --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \ + --set app.env.CRON_SECRET="$CRON_SECRET" \ + --set postgresql.auth.password="$POSTGRES_PASSWORD" \ --set app.env.NEXT_PUBLIC_APP_URL="https://sim.yourdomain.com" \ --namespace simstudio --create-namespace ``` @@ -115,7 +127,7 @@ externalDatabase: ```bash # Port forward for local access -kubectl port-forward deployment/sim-sim-app 3000:3000 -n simstudio +kubectl port-forward deployment/sim-app 3000:3000 -n simstudio # View logs kubectl logs -l app.kubernetes.io/component=app -n simstudio --tail=100 @@ -130,7 +142,7 @@ helm uninstall sim --namespace simstudio get pods,events --sort-by='.lastTimestamp' kubectl --namespace logs deploy/sim-app --tail=200 kubectl --namespace logs deploy/sim-realtime --tail=200 kubectl --namespace logs sts/sim-postgresql --tail=200 -kubectl --namespace logs job/sim-migrations --tail=200 2>/dev/null +kubectl --namespace logs deploy/sim-app -c migrations --tail=200 2>/dev/null kubectl --namespace describe pod -l app.kubernetes.io/name=sim ``` diff --git a/helm/sim/.claude/skills/sim-helm/references/secrets.md b/helm/sim/.claude/skills/sim-helm/references/secrets.md index de6e241b70..e408b1cddd 100644 --- a/helm/sim/.claude/skills/sim-helm/references/secrets.md +++ b/helm/sim/.claude/skills/sim-helm/references/secrets.md @@ -24,7 +24,7 @@ export POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=') # if using ch | `INTERNAL_API_SECRET` | Shared auth between `sim-app` ↔ `sim-realtime` pods | 32 bytes = 64 hex chars | Both deployments must roll together — temporary realtime errors during the rollout | | `CRON_SECRET` | Authenticates scheduled CronJob pods to the app | 32 bytes = 64 hex chars | Rotating just needs `helm upgrade`; next cron run uses the new value | | `API_ENCRYPTION_KEY` (optional) | Encrypts user-stored API keys (OpenAI tokens, etc.) at rest in Postgres | **Exactly 64 hex chars** (the app rejects other lengths) | Without it, keys are stored plain. Once set, never rotate without a migration | -| `POSTGRES_PASSWORD` (chart-bundled Postgres only) | Postgres superuser password | Any length ≥ 12 chars matching `^[a-zA-Z0-9._-]+$` | Requires Postgres pod restart + app rollout | +| `POSTGRES_PASSWORD` (chart-bundled Postgres only) | Postgres superuser password | ≥ 8 chars (schema-enforced) matching `^[a-zA-Z0-9._-]+$`; 32-byte hex recommended | Requires Postgres pod restart + app rollout | The `^[a-zA-Z0-9._-]+$` constraint on the Postgres password exists because the chart embeds the password into `DATABASE_URL` without URL-encoding. The `tr -d '/+='` strips the three problematic characters from `openssl rand -base64` output. The chart enforces this regex at template time. diff --git a/helm/sim/.claude/skills/sim-helm/references/troubleshooting.md b/helm/sim/.claude/skills/sim-helm/references/troubleshooting.md index 1bc1c2df47..a4ce19fbe5 100644 --- a/helm/sim/.claude/skills/sim-helm/references/troubleshooting.md +++ b/helm/sim/.claude/skills/sim-helm/references/troubleshooting.md @@ -11,7 +11,7 @@ kubectl --namespace $NS describe pod -l app.kubernetes.io/instance=sim kubectl --namespace $NS logs deploy/sim-app --tail=200 kubectl --namespace $NS logs deploy/sim-realtime --tail=200 kubectl --namespace $NS logs sts/sim-postgresql --tail=200 -kubectl --namespace $NS logs job/sim-migrations --tail=200 2>/dev/null || true +kubectl --namespace $NS logs deploy/sim-app -c migrations --tail=200 2>/dev/null || true ``` --- @@ -63,9 +63,9 @@ Match the error: |---|---|---| | `Invalid env: ... NEXT_PUBLIC_APP_URL: Invalid url` | URL field set to empty string or invalid format | Set `app.env.NEXT_PUBLIC_APP_URL` to a valid URL — `https://sim.example.com` in prod, `http://localhost:3000` in dev | | `getaddrinfo ENOTFOUND ... -postgresql` / `connect ECONNREFUSED` | App can't reach Postgres | Check `kubectl get pod -l app.kubernetes.io/name=postgresql` is `Running`; check `postgresql.auth.password` matches the password in the Secret | -| `password authentication failed for user "sim"` | Postgres password rotated but app pod wasn't restarted, OR password contains URL-unsafe chars | `kubectl rollout restart deploy/sim-app -n sim`; regenerate password with `openssl rand -base64 24 \| tr -d '/+='` | +| `password authentication failed for user "postgres"` | Postgres password rotated but app pod wasn't restarted, OR password contains URL-unsafe chars | `kubectl rollout restart deploy/sim-app -n sim`; regenerate password with `openssl rand -base64 24 \| tr -d '/+='` | | `BETTER_AUTH_SECRET is missing` / `INTERNAL_API_SECRET is required` | Required env var not present in the Secret | Verify with `kubectl get secret sim-app-secrets -o jsonpath='{.data}' \| jq 'keys'`; if missing, fix your secret strategy | -| `Migration failed` or app starts before migration | Migration Job hasn't completed | `kubectl logs job/sim-migrations -n sim`; rerun with `kubectl delete job/sim-migrations && helm upgrade ...` | +| `Migration failed` | Migrations init container on the app pod failed | `kubectl logs deploy/sim-app -c migrations -n sim`; rerun with `kubectl rollout restart deploy/sim-app -n sim` (migrations run before each app pod starts, so the app can never start ahead of them) | --- @@ -112,7 +112,7 @@ kubectl describe ingress -n sim | No `ADDRESS` in `kubectl get ingress` | Ingress controller not installed — install `ingress-nginx`, AWS LBC, GCP LB controller, etc. | | `ingressClassName` doesn't match installed controller | `kubectl get ingressclass` to list installed classes, set `ingress.className` to match | | Address is set but DNS resolves to wrong IP | `dig ` — point DNS at the ingress controller's external IP / LoadBalancer / CNAME | -| TLS cert errors | If using cert-manager, check `kubectl describe certificate -n sim`; verify `ingress.tls.issuerRef` | +| TLS cert errors | If using cert-manager, check `kubectl describe certificate -n sim`; verify the `cert-manager.io/cluster-issuer` annotation on the ingress | | `503 Service Unavailable` | Ingress routing is fine but app pod isn't `Ready` — go back to the diagnostic block | --- @@ -153,7 +153,7 @@ Bump the relevant resource limit. Defaults: |---|---|---| | `app` | `1000m` CPU / `4Gi` memory | `2000m` CPU / `8Gi` memory | | `realtime` | `250m` CPU / `512Mi` memory | `500m` CPU / `1Gi` memory | -| `postgresql` | `250m` CPU / `512Mi` memory | `1000m` CPU / `2Gi` memory | +| `postgresql` | `500m` CPU / `1Gi` memory | `2Gi` memory (no CPU limit) | Override in values: @@ -172,7 +172,7 @@ app: ```bash helm version -kubectl version --short +kubectl version helm get values sim -n sim --revision $(helm history sim -n sim | tail -1 | awk '{print $1}') kubectl get all,pvc,ingress,externalsecret -n sim -o wide kubectl describe pods -n sim -l app.kubernetes.io/instance=sim | head -200 diff --git a/helm/sim/.claude/skills/sim-helm/references/values-model.md b/helm/sim/.claude/skills/sim-helm/references/values-model.md index 801dfd77b7..58cb68c96f 100644 --- a/helm/sim/.claude/skills/sim-helm/references/values-model.md +++ b/helm/sim/.claude/skills/sim-helm/references/values-model.md @@ -27,7 +27,7 @@ The Sim chart splits configuration across **four** layers. Understanding which l ▼ ┌───────────────────────────────────────────────────────────────────────────┐ │ Layer 3: chart-computed (inline env: on the Deployment) │ -│ → DATABASE_URL, SOCKET_SERVER_URL, OLLAMA_URL │ +│ → DATABASE_URL, SOCKET_SERVER_URL, OLLAMA_URL, PII_URL │ │ → Derived from postgresql.* / externalDatabase.* / service.* values │ │ → CANNOT be overridden via app.env — chart filters them out │ └───────────────────────────────────────────────────────────────────────────┘ @@ -63,7 +63,7 @@ The exhaustive list of keys per layer lives in `helm/sim/values.yaml`. Read the | Rate limits | 2 (envDefaults) | `RATE_LIMIT_WINDOW_MS`, `RATE_LIMIT_FREE_SYNC`, etc. | | Execution timeouts | 2 (envDefaults) | `EXECUTION_TIMEOUT_FREE`, `EXECUTION_TIMEOUT_PRO`, etc. | | IVM pool / quotas | 2 (envDefaults) | `IVM_POOL_SIZE`, `IVM_MAX_CONCURRENT`, `IVM_MAX_PER_WORKER`, etc. | -| Connection strings | 3 (chart-computed) | `DATABASE_URL`, `SOCKET_SERVER_URL`, `OLLAMA_URL` | +| Connection strings | 4 (chart-computed) | `DATABASE_URL`, `SOCKET_SERVER_URL`, `OLLAMA_URL`, `PII_URL` | | Custom downward API / configMapKeyRef | 4 (extraEnvVars) | anything that needs `valueFrom:` | ## Common authoring patterns diff --git a/helm/sim/Chart.yaml b/helm/sim/Chart.yaml index 1618461b10..c4c8b2ca7c 100644 --- a/helm/sim/Chart.yaml +++ b/helm/sim/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: sim description: A Helm chart for Sim - the open-source AI workspace where teams build, deploy, and manage AI agents type: application -version: 1.0.0 +version: 1.1.0 appVersion: "0.6.73" kubeVersion: ">=1.25.0-0" home: https://sim.ai diff --git a/helm/sim/README.md b/helm/sim/README.md index 908aa970ea..5b02a28856 100644 --- a/helm/sim/README.md +++ b/helm/sim/README.md @@ -40,8 +40,8 @@ This chart deploys the Sim platform on a Kubernetes cluster using the Helm packa * **`app`** — the Sim Next.js web application (Deployment). * **`realtime`** — the WebSocket service for live workflow updates (Deployment). * **`postgresql`** — an in-cluster `pgvector/pgvector` Postgres (StatefulSet, with a headless Service for stable per-pod DNS). -* **`migrations`** — a Job that applies database migrations on install/upgrade. -* **`cronjobs`** — scheduled jobs for workflow schedule execution, inbox/calendar/drive polling (Gmail, Outlook, Calendar, Drive, Sheets, IMAP, RSS), workspace event polling, subscription renewal, data drains, and connector syncs. +* **`migrations`** — an init container on the app Deployment that applies database migrations before each app pod starts. +* **`cronjobs`** — scheduled jobs for workflow schedule execution, inbox/calendar/drive polling (Gmail, Outlook, Calendar, Drive, Sheets, IMAP, RSS), workspace event and HubSpot webhook polling, outbox processing, subscription renewal, billing-seat and inbox-entitlement reconciliation, time-pause/resume polling, data drains, and connector syncs. * **`serviceaccount`** — a dedicated ServiceAccount with `automountServiceAccountToken: false`. Optional components (off by default): @@ -214,7 +214,7 @@ cat ./helm/sim/values.schema.json Before installing in production, confirm each of the following: -* **High availability** — scale `app.replicaCount > 1`. The chart auto-creates a `PodDisruptionBudget` with `minAvailable: 1`. Set `podDisruptionBudget.maxUnavailable: "25%"` for a more permissive policy or `minAvailable: "50%"` for a stricter one. +* **High availability** — scale `app.replicaCount > 1`. The chart auto-creates a `PodDisruptionBudget` with `maxUnavailable: "25%"`. Set `podDisruptionBudget.minAvailable` instead for a stricter policy. * **Pinned images** — override `image.tag` (or `image.digest`) with an explicit version. Do not rely on the chart's default tag in production. * **Secrets management** — provide secrets via External Secrets Operator (ESO) or pre-created Kubernetes Secrets. Never commit secrets to `values.yaml`. * **TLS / Ingress** — set the `cert-manager.io/cluster-issuer` annotation on the ingress and tune `proxy-body-size` / `proxy-read-timeout` for your workload. See commented examples in `values.yaml`. @@ -271,8 +271,7 @@ postgresql: auth: existingSecret: enabled: true - name: sim-postgres-secret - passwordKey: POSTGRES_PASSWORD + name: sim-postgres-secret # must contain the password under the key POSTGRES_PASSWORD ``` See `examples/values-existing-secret.yaml`. @@ -357,7 +356,7 @@ autoscaling: targetMemoryUtilizationPercentage: 80 ``` -When `autoscaling.enabled=true`, the chart omits `spec.replicas` from the Deployment so the HPA owns replica count. Requires `metrics-server` in the cluster. +When `autoscaling.enabled=true`, the chart omits `spec.replicas` from the Deployment so the HPA owns replica count. Requires `metrics-server` in the cluster. The realtime Deployment gets the same HPA unless `autoscaling.realtime.enabled=false` — scale realtime past one replica only with `REDIS_URL` set (Socket.IO Redis adapter), or cross-pod collaboration events are dropped. --- @@ -370,7 +369,7 @@ monitoring: interval: 30s ``` -Requires the Prometheus Operator CRDs. Scrapes `/metrics` on the app and realtime services. +Requires the Prometheus Operator CRDs. Scrapes `/metrics` on the app and realtime services — note the default images do not currently expose a `/metrics` endpoint, so enable this only with a build that does. --- @@ -427,7 +426,7 @@ Common causes: * `NEXT_PUBLIC_APP_URL` still set to `http://localhost:3000` in a clustered deploy → set it to your public origin. * `DATABASE_URL` not reachable → check the Postgres pod is running and `postgresql.auth.password` matches. -* Missing migration → check `kubectl logs job/sim-migrations`. +* Missing migration → check `kubectl logs deploy/sim-app -c migrations` (migrations run as an init container on the app pod). ### Image pull errors (`ErrImagePull` / `ImagePullBackOff`) @@ -463,11 +462,19 @@ kubectl describe ingress --namespace sim kubectl --namespace sim logs -f deployment/sim-app kubectl --namespace sim logs -f deployment/sim-realtime kubectl --namespace sim logs -f statefulset/sim-postgresql -kubectl --namespace sim logs job/sim-migrations +kubectl --namespace sim logs deploy/sim-app -c migrations ``` --- +## Upgrading to 1.1.0 + +No action is required for working configurations. Notes: + +* Pods for `app` and `realtime` roll once on upgrade (their rollout checksum now also covers the ExternalSecret manifest, fixing missed rollouts in ESO mode). +* Two values keys that were never consumed by any template were removed: `app.secrets.existingSecret.keys` and `*.existingSecret.passwordKey`. Existing secrets must use the standard key names (`BETTER_AUTH_SECRET`, ..., `POSTGRES_PASSWORD`, `EXTERNAL_DB_PASSWORD`); leftover keys in your values file are ignored, not rejected. +* `telemetry.jaeger` now exports over OTLP (`otlp/jaeger`) — point `telemetry.jaeger.endpoint` at Jaeger's OTLP gRPC port (4317). The previous `jaeger` exporter did not exist in the pinned collector image, so any prior jaeger-enabled config was already failing at collector startup. + ## Support * **Docs:** https://docs.sim.ai diff --git a/helm/sim/examples/values-aws.yaml b/helm/sim/examples/values-aws.yaml index a0837b2672..3be69b72be 100644 --- a/helm/sim/examples/values-aws.yaml +++ b/helm/sim/examples/values-aws.yaml @@ -8,7 +8,9 @@ # - EKS cluster (Kubernetes 1.25+) # - EBS CSI driver add-on (aws eks create-addon --addon-name aws-ebs-csi-driver) # - AWS Load Balancer Controller (for ALB ingress) -# - (recommended) cert-manager OR an ACM certificate ARN for the ingress +# - An ACM certificate for your domains (ALB auto-discovers it from the ingress +# hosts, or pin it with alb.ingress.kubernetes.io/certificate-arn). cert-manager +# does not work with ALB — ALB cannot serve Kubernetes TLS Secrets # # Install: # helm install sim ./helm/sim \ @@ -110,7 +112,11 @@ app: # Realtime service realtime: enabled: true - replicaCount: 2 + # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime + # shares the app secret): Socket.IO only bridges events across pods through + # its Redis adapter, so multi-replica realtime without Redis silently drops + # cross-pod collaboration updates. + replicaCount: 1 # Node selector for realtime pods # Uncomment and customize based on your EKS node labels: @@ -171,7 +177,7 @@ postgresql: # Persistent storage using AWS EBS volumes persistence: enabled: true - storageClass: "gp2" # Use gp2 (default) or create gp3 StorageClass + storageClass: "gp3" # Matches the global gp3 StorageClass above size: 50Gi accessModes: - ReadWriteOnce @@ -222,7 +228,7 @@ ollama: # High-performance storage for AI models persistence: enabled: true - storageClass: "gp2" # Use gp2 (default) or create gp3 StorageClass + storageClass: "gp3" # Matches the global gp3 StorageClass above size: 100Gi accessModes: - ReadWriteOnce @@ -239,7 +245,6 @@ ingress: className: alb annotations: - kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/ssl-redirect: "443" @@ -284,7 +289,7 @@ affinity: matchExpressions: - key: app.kubernetes.io/name operator: In - values: ["simstudio"] + values: ["sim"] topologyKey: kubernetes.io/hostname - weight: 50 podAffinityTerm: @@ -292,7 +297,7 @@ affinity: matchExpressions: - key: app.kubernetes.io/name operator: In - values: ["simstudio"] + values: ["sim"] topologyKey: topology.kubernetes.io/zone # Service Account with IAM roles for service account (IRSA) integration diff --git a/helm/sim/examples/values-azure.yaml b/helm/sim/examples/values-azure.yaml index 2404088b22..5333570a42 100644 --- a/helm/sim/examples/values-azure.yaml +++ b/helm/sim/examples/values-azure.yaml @@ -7,8 +7,8 @@ # Prerequisites: # - AKS cluster (Kubernetes 1.25+) # - NGINX ingress controller installed -# - (optional) cert-manager for TLS -# - (optional) GPU node pool labeled with role: datalake / accelerator: nvidia +# - cert-manager installed with a ClusterIssuer (issues the ingress TLS secret) +# - (optional) GPU node pool tainted with sku=gpu:NoSchedule for Ollama # # Install: # helm install sim ./helm/sim \ @@ -127,7 +127,11 @@ app: # Realtime service realtime: enabled: true - replicaCount: 2 + # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime + # shares the app secret): Socket.IO only bridges events across pods through + # its Redis adapter, so multi-replica realtime without Redis silently drops + # cross-pod collaboration updates. + replicaCount: 1 # Node selector for realtime pods # Uncomment and customize based on your AKS node labels: @@ -242,6 +246,10 @@ ingress: annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: "true" + # cert-manager issues the certificate into ingress.tls.secretName below — + # without this annotation (and cert-manager installed) nginx serves its + # self-signed fake certificate forever. + cert-manager.io/cluster-issuer: "letsencrypt-prod" # Main application app: diff --git a/helm/sim/examples/values-copilot.yaml b/helm/sim/examples/values-copilot.yaml index d95e1aa2e9..ff0a0b877e 100644 --- a/helm/sim/examples/values-copilot.yaml +++ b/helm/sim/examples/values-copilot.yaml @@ -1,6 +1,6 @@ # values-copilot.yaml # -# When to use: enable the Sim Copilot service (the chat / Mothership backend) +# When to use: enable the Sim Copilot service (the backend for Chat — the Sim agent) # alongside the main app. Provisions a dedicated Copilot Deployment, its own # Postgres StatefulSet, and a migration Job for the Copilot schema. # diff --git a/helm/sim/examples/values-existing-secret.yaml b/helm/sim/examples/values-existing-secret.yaml index a875e99aba..0ddf0621ac 100644 --- a/helm/sim/examples/values-existing-secret.yaml +++ b/helm/sim/examples/values-existing-secret.yaml @@ -7,7 +7,7 @@ # # Prerequisites: # Create the Secret objects before `helm install`. Example: -# kubectl create secret generic my-app-secrets --namespace sim \ +# kubectl create secret generic sim-app-secrets --namespace sim \ # --from-literal=BETTER_AUTH_SECRET=$(openssl rand -hex 32) \ # --from-literal=ENCRYPTION_KEY=$(openssl rand -hex 32) \ # --from-literal=INTERNAL_API_SECRET=$(openssl rand -hex 32) \ @@ -34,7 +34,11 @@ app: realtime: enabled: true - replicaCount: 2 + # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime + # shares the app secret): Socket.IO only bridges events across pods through + # its Redis adapter, so multi-replica realtime without Redis silently drops + # cross-pod collaboration updates. + replicaCount: 1 env: NEXT_PUBLIC_APP_URL: "https://sim.example.com" BETTER_AUTH_URL: "https://sim.example.com" diff --git a/helm/sim/examples/values-external-db.yaml b/helm/sim/examples/values-external-db.yaml index 630b2f49bb..9adfce02f6 100644 --- a/helm/sim/examples/values-external-db.yaml +++ b/helm/sim/examples/values-external-db.yaml @@ -11,7 +11,13 @@ # - Network reachability from the cluster (peering, private link, or public # endpoint with TLS) # -# Install: +# Install — generate each secret first in the same shell: +# export BETTER_AUTH_SECRET=$(openssl rand -hex 32) +# export ENCRYPTION_KEY=$(openssl rand -hex 32) +# export INTERNAL_API_SECRET=$(openssl rand -hex 32) +# export CRON_SECRET=$(openssl rand -hex 32) +# export DB_PASSWORD=$(openssl rand -hex 24) +# # helm install sim ./helm/sim \ # --namespace sim --create-namespace \ # --values ./helm/sim/examples/values-external-db.yaml \ @@ -19,9 +25,11 @@ # --set externalDatabase.username=simstudio_user \ # --set externalDatabase.password="$DB_PASSWORD" \ # --set externalDatabase.database=simstudio_prod \ +# --set "networkPolicy.egress[0].to[0].ipBlock.cidr=10.20.0.0/24" \ # --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \ # --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \ -# --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" +# --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \ +# --set app.env.CRON_SECRET="$CRON_SECRET" # Global configuration global: @@ -63,7 +71,12 @@ app: # Realtime service realtime: enabled: true - replicaCount: 2 + # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime + # shares the app secret): Socket.IO only bridges events across pods through + # its Redis adapter, so multi-replica realtime without Redis silently drops + # cross-pod collaboration updates. autoscaling.realtime.enabled is false + # below for the same reason — flip both once REDIS_URL is set. + replicaCount: 1 resources: limits: @@ -101,9 +114,9 @@ externalDatabase: enabled: true # Database connection details (REQUIRED - configure for your external database) - host: "" # Database hostname (e.g., "postgres.acme.com" or RDS endpoint) + host: "postgres.acme.com" # REQUIRED - your database hostname (RDS/Cloud SQL/Azure endpoint or IP) port: 5432 - username: "" # Database username (e.g., "simstudio_user") + username: "simstudio_user" # REQUIRED - your database username password: "" # Database password - set via --set flag or external secret database: "" # Database name (e.g., "simstudio_production") @@ -142,6 +155,11 @@ ingress: # Production-ready features (autoscaling, monitoring, etc.) autoscaling: enabled: true + # Keep realtime out of the HPA until REDIS_URL is set (Socket.IO Redis + # adapter) — multi-replica realtime without Redis silently drops cross-pod + # collaboration events. + realtime: + enabled: false minReplicas: 2 maxReplicas: 20 targetCPUUtilizationPercentage: 70 @@ -161,14 +179,19 @@ monitoring: networkPolicy: enabled: true - # Custom egress rules to allow database connectivity to an external DB. - # The chart's app/realtime NetworkPolicies already permit egress to - # `postgresql.service.targetPort` when `postgresql.enabled=true`. For an - # external DB on a different port (or off-cluster), append to extraRules. + # `networkPolicy.egress` is a LIST of NetworkPolicy egress rules. With the + # policy on, the chart's egress allowlist only covers the in-cluster + # Postgres, so this rule is what lets the pods reach your external database. + # The CIDR below is an example, exactly like `your-db.example.com` in the + # install command: REQUIRED to change — set your database's subnet (or /32 + # host) via the --set line in the install commands below, or edit it here. + # A wrong CIDR blocks migrations and every database connection; an empty + # `to:` would allow ANY host on 5432 and defeat the isolation. egress: - extraRules: - - to: [] - ports: + - to: + - ipBlock: + cidr: 10.0.0.0/16 # REQUIRED - your database subnet or /32 host CIDR + ports: - protocol: TCP port: 5432 @@ -179,6 +202,7 @@ networkPolicy: # --set externalDatabase.username="your-db-user" \ # --set externalDatabase.password="your-db-password" \ # --set externalDatabase.database="your-db-name" \ +# --set "networkPolicy.egress[0].to[0].ipBlock.cidr=10.20.0.0/24" \ # --set app.env.BETTER_AUTH_SECRET="$(openssl rand -hex 32)" \ # --set app.env.ENCRYPTION_KEY="$(openssl rand -hex 32)" \ # --set app.env.INTERNAL_API_SECRET="$(openssl rand -hex 32)" \ diff --git a/helm/sim/examples/values-external-secrets.yaml b/helm/sim/examples/values-external-secrets.yaml index 23aa6c5cfb..e9533fc168 100644 --- a/helm/sim/examples/values-external-secrets.yaml +++ b/helm/sim/examples/values-external-secrets.yaml @@ -63,7 +63,11 @@ app: realtime: enabled: true - replicaCount: 2 + # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime + # shares the app secret): Socket.IO only bridges events across pods through + # its Redis adapter, so multi-replica realtime without Redis silently drops + # cross-pod collaboration updates. + replicaCount: 1 envDefaults: NEXT_PUBLIC_APP_URL: "https://sim.example.com" BETTER_AUTH_URL: "https://sim.example.com" @@ -81,7 +85,7 @@ postgresql: # --- # Azure Key Vault (Workload Identity): -# apiVersion: external-secrets.io/v1beta1 +# apiVersion: external-secrets.io/v1 # kind: ClusterSecretStore # metadata: # name: sim-secret-store @@ -95,7 +99,7 @@ postgresql: # namespace: external-secrets # AWS Secrets Manager (IRSA): -# apiVersion: external-secrets.io/v1beta1 +# apiVersion: external-secrets.io/v1 # kind: ClusterSecretStore # metadata: # name: sim-secret-store @@ -107,7 +111,7 @@ postgresql: # role: arn:aws:iam::123456789012:role/external-secrets-role # HashiCorp Vault (Kubernetes Auth): -# apiVersion: external-secrets.io/v1beta1 +# apiVersion: external-secrets.io/v1 # kind: ClusterSecretStore # metadata: # name: sim-secret-store diff --git a/helm/sim/examples/values-gcp.yaml b/helm/sim/examples/values-gcp.yaml index 545d955ff9..fca78a7b01 100644 --- a/helm/sim/examples/values-gcp.yaml +++ b/helm/sim/examples/values-gcp.yaml @@ -7,7 +7,7 @@ # Prerequisites: # - GKE cluster (Kubernetes 1.25+) # - Workload Identity enabled (for IAM-bound ServiceAccount) -# - (optional) Managed certificate or cert-manager for TLS +# - A GKE ManagedCertificate named simstudio-ssl-cert (creation manifest in the ingress section below) — or cert-manager if you prefer # - (optional) GPU node pool with accelerator labels for Ollama # # Install: @@ -86,10 +86,6 @@ app: NODE_ENV: "production" NEXT_TELEMETRY_DISABLED: "1" - # GCP-specific environment variables - GOOGLE_CLOUD_PROJECT: "your-project-id" - GOOGLE_CLOUD_REGION: "us-central1" - # Google Cloud Storage Configuration (RECOMMENDED for production) # Create GCS buckets in your project and grant the Workload Identity # service account roles/storage.objectAdmin on them. Signed-URL generation @@ -117,7 +113,11 @@ app: # Realtime service realtime: enabled: true - replicaCount: 2 + # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime + # shares the app secret): Socket.IO only bridges events across pods through + # its Redis adapter, so multi-replica realtime without Redis silently drops + # cross-pod collaboration updates. + replicaCount: 1 # Node selector for realtime pods # Uncomment and customize based on your GKE node labels: @@ -266,10 +266,29 @@ ingress: - path: / pathType: Prefix - # TLS configuration + # TLS comes from the GKE ManagedCertificate referenced by the + # networking.gke.io/managed-certificates annotation above. The chart does NOT + # create that resource — create it once before installing (the certificate + # provisions automatically after DNS resolves; allow ~15-30 minutes): + # + # kubectl create namespace sim + # cat <<'EOF' | kubectl apply -f - + # apiVersion: networking.gke.io/v1 + # kind: ManagedCertificate + # metadata: + # name: simstudio-ssl-cert + # namespace: sim + # spec: + # domains: + # - simstudio.acme.com + # - simstudio-ws.acme.com + # EOF + # + # Keep the chart's secret-based TLS off with GCE managed certificates — an + # enabled spec.tls references a Secret nothing creates, and the GKE ingress + # controller reports sync errors for it. tls: - enabled: true - secretName: simstudio-tls-secret + enabled: false # Pod disruption budget for high availability podDisruptionBudget: @@ -291,7 +310,7 @@ affinity: matchExpressions: - key: app.kubernetes.io/name operator: In - values: ["simstudio"] + values: ["sim"] topologyKey: kubernetes.io/hostname - weight: 50 podAffinityTerm: @@ -299,7 +318,7 @@ affinity: matchExpressions: - key: app.kubernetes.io/name operator: In - values: ["simstudio"] + values: ["sim"] topologyKey: topology.gke.io/zone # Service Account with Workload Identity integration @@ -308,21 +327,27 @@ serviceAccount: annotations: iam.gke.io/gcp-service-account: "simstudio@your-project-id.iam.gserviceaccount.com" -# Additional environment variables for GCP service integration -extraEnvVars: - - name: GOOGLE_APPLICATION_CREDENTIALS - value: "/var/secrets/google/key.json" - -# Additional volumes for service account credentials -extraVolumes: - - name: google-cloud-key - secret: - secretName: google-service-account-key - -extraVolumeMounts: - - name: google-cloud-key - mountPath: /var/secrets/google - readOnly: true +# Key-file credentials — ONLY if NOT using Workload Identity (the serviceAccount +# annotation above is the recommended path and needs no key file; a mounted +# GOOGLE_APPLICATION_CREDENTIALS overrides Workload Identity/ADC). +# If you do use a key file, create the Secret first or every pod is stuck in +# ContainerCreating on the missing volume: +# kubectl -n sim create secret generic google-service-account-key \ +# --from-file=key.json=/path/to/service-account-key.json +# +# extraEnvVars: +# - name: GOOGLE_APPLICATION_CREDENTIALS +# value: "/var/secrets/google/key.json" +# +# extraVolumes: +# - name: google-cloud-key +# secret: +# secretName: google-service-account-key +# +# extraVolumeMounts: +# - name: google-cloud-key +# mountPath: /var/secrets/google +# readOnly: true # ----------------------------------------------------------------------------- # External Secrets Operator (ESO) — recommended for production on GCP # ----------------------------------------------------------------------------- diff --git a/helm/sim/examples/values-production.yaml b/helm/sim/examples/values-production.yaml index 6f529e59d1..39afb262ba 100644 --- a/helm/sim/examples/values-production.yaml +++ b/helm/sim/examples/values-production.yaml @@ -26,7 +26,9 @@ global: imageRegistry: "ghcr.io" # For production, use a StorageClass with reclaimPolicy: Retain - storageClass: "managed-csi-premium" + # Set to YOUR cluster's StorageClass: gp3 (EKS), standard-rwo (GKE), + # managed-csi-premium (AKS). Empty uses the cluster default. + storageClass: "" # Main application app: @@ -45,7 +47,6 @@ app: env: NEXT_PUBLIC_APP_URL: "https://sim.acme.ai" BETTER_AUTH_URL: "https://sim.acme.ai" - SOCKET_SERVER_URL: "https://sim-ws.acme.ai" NEXT_PUBLIC_SOCKET_URL: "https://sim-ws.acme.ai" # Security settings (REQUIRED - replace with your own secure secrets) @@ -72,7 +73,12 @@ app: # Realtime service realtime: enabled: true - replicaCount: 2 + # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime + # shares the app secret): Socket.IO only bridges events across pods through + # its Redis adapter, so multi-replica realtime without Redis silently drops + # cross-pod collaboration updates. autoscaling.realtime.enabled is false + # below for the same reason — flip both once REDIS_URL is set. + replicaCount: 1 resources: limits: @@ -121,10 +127,12 @@ postgresql: # Persistent storage configuration persistence: enabled: true - storageClass: "managed-csi-premium" + storageClass: "" # your cluster's StorageClass (empty = cluster default) size: 50Gi - # SSL/TLS configuration (recommended for production) + # SSL/TLS configuration (recommended for production). Requires + # certManager.enabled below — the Certificate's issuerRef (sim-ca-issuer) + # only exists when the chart creates its CA issuer. tls: enabled: true certificatesSecret: postgres-tls-secret @@ -168,9 +176,18 @@ ingress: enabled: true secretName: sim-tls-secret +# Chart-managed CA issuer — required for postgresql.tls above (issuerRef sim-ca-issuer) +certManager: + enabled: true + # Horizontal Pod Autoscaler (automatically scales pods based on CPU/memory usage) autoscaling: enabled: true + # Keep realtime out of the HPA until REDIS_URL is set (Socket.IO Redis + # adapter) — multi-replica realtime without Redis silently drops cross-pod + # collaboration events. + realtime: + enabled: false minReplicas: 2 maxReplicas: 20 targetCPUUtilizationPercentage: 70 @@ -242,4 +259,6 @@ telemetry: endpoint: "http://prometheus-server/api/v1/write" jaeger: enabled: true - endpoint: "http://jaeger-collector:14250" + # Jaeger's OTLP gRPC endpoint (the chart exports via otlp/jaeger; + # Jaeger >=1.35 ingests OTLP natively) + endpoint: "jaeger-collector:4317" diff --git a/helm/sim/examples/values-whitelabeled.yaml b/helm/sim/examples/values-whitelabeled.yaml index d7ec9c4a69..44d8b16c6e 100644 --- a/helm/sim/examples/values-whitelabeled.yaml +++ b/helm/sim/examples/values-whitelabeled.yaml @@ -19,7 +19,7 @@ # Global configuration global: imageRegistry: "ghcr.io" - storageClass: "managed-csi-premium" + storageClass: "" # your cluster's StorageClass (empty = cluster default) # Main application with custom branding app: @@ -31,7 +31,6 @@ app: # Application URLs (update with your domain) NEXT_PUBLIC_APP_URL: "https://sim.acme.ai" BETTER_AUTH_URL: "https://sim.acme.ai" - SOCKET_SERVER_URL: "https://sim-ws.acme.ai" NEXT_PUBLIC_SOCKET_URL: "https://sim-ws.acme.ai" # Security settings (REQUIRED) @@ -101,6 +100,11 @@ ingress: # Auto-scaling autoscaling: enabled: true + # Keep realtime out of the HPA until REDIS_URL is set (Socket.IO Redis + # adapter) — multi-replica realtime without Redis silently drops cross-pod + # collaboration events. + realtime: + enabled: false minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 70 diff --git a/helm/sim/templates/NOTES.txt b/helm/sim/templates/NOTES.txt index 250bdfd768..29be6c39bd 100644 --- a/helm/sim/templates/NOTES.txt +++ b/helm/sim/templates/NOTES.txt @@ -46,7 +46,7 @@ Your release is named {{ .Release.Name }} in namespace {{ .Release.Namespace }}. 3. Required secrets: - Sim requires three application secrets and a Postgres password to start. + Sim requires four application secrets and a Postgres password to start. {{- if .Values.externalSecrets.enabled }} Using External Secrets Operator. Verify the ExternalSecret has synced: diff --git a/helm/sim/templates/_helpers.tpl b/helm/sim/templates/_helpers.tpl index 32d917a33b..4ecaf0d263 100644 --- a/helm/sim/templates/_helpers.tpl +++ b/helm/sim/templates/_helpers.tpl @@ -133,14 +133,6 @@ PII (Presidio) selector labels app.kubernetes.io/component: pii {{- end }} -{{/* -Migrations specific labels -*/}} -{{- define "sim.migrations.labels" -}} -{{ include "sim.labels" . }} -app.kubernetes.io/component: migrations -{{- end }} - {{/* Copilot specific labels */}} @@ -393,18 +385,6 @@ Returns the name of the secret containing PostgreSQL password {{- end -}} {{- end }} -{{/* -Get the PostgreSQL password key name -Returns the key name in the secret that contains the password -*/}} -{{- define "sim.postgresqlPasswordKey" -}} -{{- if and .Values.postgresql.auth.existingSecret .Values.postgresql.auth.existingSecret.enabled -}} -{{- .Values.postgresql.auth.existingSecret.passwordKey | default "POSTGRES_PASSWORD" -}} -{{- else -}} -{{- print "POSTGRES_PASSWORD" -}} -{{- end -}} -{{- end }} - {{/* Get the external database secret name Returns the name of the secret containing external database password @@ -417,18 +397,6 @@ Returns the name of the secret containing external database password {{- end -}} {{- end }} -{{/* -Get the external database password key name -Returns the key name in the secret that contains the password -*/}} -{{- define "sim.externalDbPasswordKey" -}} -{{- if and .Values.externalDatabase.existingSecret .Values.externalDatabase.existingSecret.enabled -}} -{{- .Values.externalDatabase.existingSecret.passwordKey | default "EXTERNAL_DB_PASSWORD" -}} -{{- else -}} -{{- print "EXTERNAL_DB_PASSWORD" -}} -{{- end -}} -{{- end }} - {{/* Check if app secrets should be created by the chart Returns true if we should create the app secrets (not using existing or ESO) diff --git a/helm/sim/templates/deployment-app.yaml b/helm/sim/templates/deployment-app.yaml index 77a3d792e7..f3506edf53 100644 --- a/helm/sim/templates/deployment-app.yaml +++ b/helm/sim/templates/deployment-app.yaml @@ -17,7 +17,12 @@ spec: template: metadata: annotations: - checksum/secret: {{ include (print $.Template.BasePath "/secrets-app.yaml") . | sha256sum }} + {{- /* Hash the inline Secret AND the ExternalSecret manifest so spec + changes under externalSecrets.remoteRefs.app roll the pods (with ESO + enabled the inline Secret renders empty and would hash to a constant). + Rotated values inside the external store still can't be detected at + render time — ESO's own reconciliation handles those. */}} + checksum/secret: {{ printf "%s%s" (include (print $.Template.BasePath "/secrets-app.yaml") .) (include (print $.Template.BasePath "/external-secret-app.yaml") .) | sha256sum }} {{- if .Values.branding.enabled }} checksum/config: {{ include (print $.Template.BasePath "/configmap-branding.yaml") . | sha256sum }} {{- end }} diff --git a/helm/sim/templates/deployment-realtime.yaml b/helm/sim/templates/deployment-realtime.yaml index b46923d14c..1e3487164b 100644 --- a/helm/sim/templates/deployment-realtime.yaml +++ b/helm/sim/templates/deployment-realtime.yaml @@ -8,7 +8,9 @@ metadata: labels: {{- include "sim.realtime.labels" . | nindent 4 }} spec: - {{- if not .Values.autoscaling.enabled }} + {{- /* The HPA owns replicas only when the realtime HPA actually renders — + with autoscaling.realtime.enabled=false, replicaCount stays in charge. */}} + {{- if not (and .Values.autoscaling.enabled .Values.autoscaling.realtime.enabled) }} replicas: {{ .Values.realtime.replicaCount }} {{- end }} selector: @@ -17,7 +19,12 @@ spec: template: metadata: annotations: - checksum/secret: {{ include (print $.Template.BasePath "/secrets-app.yaml") . | sha256sum }} + {{- /* Hash the inline Secret AND the ExternalSecret manifest so spec + changes under externalSecrets.remoteRefs.app roll the pods (with ESO + enabled the inline Secret renders empty and would hash to a constant). + Rotated values inside the external store still can't be detected at + render time — ESO's own reconciliation handles those. */}} + checksum/secret: {{ printf "%s%s" (include (print $.Template.BasePath "/secrets-app.yaml") .) (include (print $.Template.BasePath "/external-secret-app.yaml") .) | sha256sum }} {{- with .Values.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} diff --git a/helm/sim/templates/hpa.yaml b/helm/sim/templates/hpa.yaml index 01c20058f8..4f1892d0b2 100644 --- a/helm/sim/templates/hpa.yaml +++ b/helm/sim/templates/hpa.yaml @@ -41,7 +41,7 @@ spec: {{- end }} {{- end }} -{{- if and .Values.autoscaling.enabled .Values.realtime.enabled }} +{{- if and .Values.autoscaling.enabled .Values.realtime.enabled .Values.autoscaling.realtime.enabled }} --- # HorizontalPodAutoscaler for realtime service apiVersion: autoscaling/v2 diff --git a/helm/sim/templates/networkpolicy.yaml b/helm/sim/templates/networkpolicy.yaml index 4488b0e22f..69854cebbe 100644 --- a/helm/sim/templates/networkpolicy.yaml +++ b/helm/sim/templates/networkpolicy.yaml @@ -16,7 +16,7 @@ spec: - Ingress - Egress ingress: - # Allow ingress from realtime service + # Allow ingress from realtime pods (defensive — current traffic flows app -> realtime only) {{- if .Values.realtime.enabled }} - from: - podSelector: @@ -237,7 +237,7 @@ spec: ports: - protocol: TCP port: {{ .Values.postgresql.service.targetPort }} - # Allow ingress from realtime service + # Allow ingress from realtime pods (defensive — current traffic flows app -> realtime only) {{- if .Values.realtime.enabled }} - from: - podSelector: @@ -247,16 +247,6 @@ spec: - protocol: TCP port: {{ .Values.postgresql.service.targetPort }} {{- end }} - # Allow ingress from migrations job - {{- if .Values.migrations.enabled }} - - from: - - podSelector: - matchLabels: - {{- include "sim.migrations.labels" . | nindent 10 }} - ports: - - protocol: TCP - port: {{ .Values.postgresql.service.targetPort }} - {{- end }} egress: # Allow minimal egress (for health checks, etc.) - to: [] diff --git a/helm/sim/templates/telemetry.yaml b/helm/sim/templates/telemetry.yaml index 71bfea320c..761c515578 100644 --- a/helm/sim/templates/telemetry.yaml +++ b/helm/sim/templates/telemetry.yaml @@ -33,11 +33,14 @@ data: timeout: 1s send_batch_size: 1024 memory_limiter: + check_interval: 1s limit_mib: 512 exporters: {{- if .Values.telemetry.jaeger.enabled }} - jaeger: + {{- /* The dedicated jaeger exporter was removed from collector-contrib in + v0.86; Jaeger >=1.35 ingests OTLP natively, so export via OTLP. */}} + otlp/jaeger: endpoint: {{ .Values.telemetry.jaeger.endpoint }} tls: insecure: {{ not .Values.telemetry.jaeger.tls.enabled }} @@ -74,7 +77,7 @@ data: exporters: - logging {{- if .Values.telemetry.jaeger.enabled }} - - jaeger + - otlp/jaeger {{- end }} {{- if .Values.telemetry.otlp.enabled }} - otlp diff --git a/helm/sim/templates/tests/test-connection.yaml b/helm/sim/templates/tests/test-connection.yaml index c6389ba4ed..f05c976ea4 100644 --- a/helm/sim/templates/tests/test-connection.yaml +++ b/helm/sim/templates/tests/test-connection.yaml @@ -25,7 +25,7 @@ spec: type: RuntimeDefault containers: - name: probe - image: "{{ .Values.tests.image.repository }}:{{ .Values.tests.image.tag }}" + image: {{ include "sim.image" (dict "imageRoot" .Values.tests.image "global" .Values.global "chartAppVersion" .Chart.AppVersion) }} imagePullPolicy: {{ .Values.tests.image.pullPolicy }} securityContext: allowPrivilegeEscalation: false diff --git a/helm/sim/tests/smoke_test.yaml b/helm/sim/tests/smoke_test.yaml index 0e50a2799b..4b95fd12f2 100644 --- a/helm/sim/tests/smoke_test.yaml +++ b/helm/sim/tests/smoke_test.yaml @@ -4,6 +4,8 @@ templates: - deployment-realtime.yaml - secrets-app.yaml - services.yaml + # referenced by the deployments' checksum/secret annotation + - external-secret-app.yaml release: name: t namespace: sim diff --git a/helm/sim/values.schema.json b/helm/sim/values.schema.json index f2dd2007c3..fc26f3c48c 100644 --- a/helm/sim/values.schema.json +++ b/helm/sim/values.schema.json @@ -94,10 +94,6 @@ "name": { "type": "string", "description": "Name of the existing Kubernetes secret" - }, - "keys": { - "type": "object", - "description": "Key name mappings in the existing secret" } } } @@ -577,10 +573,6 @@ "name": { "type": "string", "description": "Name of the existing Kubernetes secret" - }, - "passwordKey": { - "type": "string", - "description": "Key in the secret containing the password" } } } @@ -634,10 +626,6 @@ "name": { "type": "string", "description": "Name of the existing Kubernetes secret" - }, - "passwordKey": { - "type": "string", - "description": "Key in the secret containing the password" } } } diff --git a/helm/sim/values.yaml b/helm/sim/values.yaml index 543e24f1ff..314c4edb38 100644 --- a/helm/sim/values.yaml +++ b/helm/sim/values.yaml @@ -61,17 +61,11 @@ app: existingSecret: # Set to true to use an existing secret instead of creating one from values enabled: false - # Name of the existing Kubernetes secret containing app credentials + # Name of the existing Kubernetes secret containing app credentials. + # The secret is consumed wholesale via envFrom, so its keys must use the + # standard names (BETTER_AUTH_SECRET, ENCRYPTION_KEY, INTERNAL_API_SECRET, + # CRON_SECRET, ...) — key remapping is not supported. name: "" - # Key mappings - specify the key names in your existing secret - # Only needed if your secret uses different key names than the defaults - keys: - BETTER_AUTH_SECRET: "BETTER_AUTH_SECRET" - ENCRYPTION_KEY: "ENCRYPTION_KEY" - INTERNAL_API_SECRET: "INTERNAL_API_SECRET" - CRON_SECRET: "CRON_SECRET" - API_ENCRYPTION_KEY: "API_ENCRYPTION_KEY" - REDIS_URL: "REDIS_URL" # Environment variables env: @@ -542,9 +536,9 @@ realtime: extraVolumes: [] extraVolumeMounts: [] -# Database migrations job configuration +# Database migrations configuration (runs as an init container on the app pods) migrations: - # Enable/disable migrations job + # Enable/disable the migrations init container enabled: true # Image configuration @@ -592,8 +586,9 @@ postgresql: # This enables integration with External Secrets Operator, HashiCorp Vault, etc. existingSecret: enabled: false - name: "" # Name of existing Kubernetes secret - passwordKey: "POSTGRES_PASSWORD" # Key in the secret containing the password + name: "" # Name of existing Kubernetes secret. Must contain the + # password under the key POSTGRES_PASSWORD — key + # remapping is not supported. # Node selector for database pod scheduling (leave empty to allow scheduling on any node) nodeSelector: {} @@ -711,8 +706,9 @@ externalDatabase: # This enables integration with External Secrets Operator, HashiCorp Vault, etc. existingSecret: enabled: false - name: "" # Name of existing Kubernetes secret - passwordKey: "EXTERNAL_DB_PASSWORD" # Key in the secret containing the password + name: "" # Name of existing Kubernetes secret. Must contain the + # password under the key EXTERNAL_DB_PASSWORD — key + # remapping is not supported. # Ollama local AI models configuration ollama: @@ -1016,6 +1012,12 @@ serviceAccount: # if you use customMetrics) installed in the cluster. autoscaling: enabled: false + # Also scale the realtime Deployment with the same HPA settings. Multi-replica + # realtime REQUIRES REDIS_URL in app.env (Socket.IO Redis adapter) — without + # Redis, cross-pod collaboration events are silently dropped. Set false to + # keep realtime pinned at realtime.replicaCount while the app still scales. + realtime: + enabled: true minReplicas: 1 maxReplicas: 10 targetCPUUtilizationPercentage: 80 @@ -1435,7 +1437,9 @@ telemetry: # Jaeger tracing backend jaeger: enabled: false - endpoint: "http://jaeger-collector:14250" + # Jaeger's OTLP gRPC endpoint (Jaeger >=1.35 accepts OTLP natively; the + # legacy jaeger exporter was removed from the collector-contrib image) + endpoint: "jaeger-collector:4317" tls: enabled: false @@ -1680,9 +1684,10 @@ copilot: secretKey: DATABASE_URL url: "" - # Migration job configuration + # Migration job configuration (Copilot migrations run as a Helm-hook Job, + # unlike the app migrations which run as an init container) migrations: - # Enable/disable migrations job + # Enable/disable the Copilot migrations Job enabled: true # Image configuration (same as server)