fix(helm): correct chart docs, examples, and dead config across the board (#5907)

* fix(helm): correct chart docs, examples, and dead config across the board Audit-driven accuracy pass over the chart's entire documentation surface, verified by rendering every example against the templates: - migrations run as an init container on the app pod, not a Job — fix the README component list, troubleshooting commands, and sim-helm skill refs; drop the dead migrations-job NetworkPolicy ingress rule - referenced-but-never-created resources: document the GKE ManagedCertificate creation (values-gcp), comment out the key-file Secret mount that stuck all pods in ContainerCreating (values-gcp), enable certManager for the postgres TLS issuerRef (values-production), add the cert-manager cluster-issuer annotation nginx needs (values-azure) - values-external-db: networkPolicy.egress is a list, not a map (the map rendered an invalid manifest); fill schema-failing placeholder host/username - realtime >1 replica requires REDIS_URL (Socket.IO Redis adapter) — default examples to 1 replica with the scaling note, and warn where autoscaling HPAs override replicaCount - pod anti-affinity selectors matched nothing (simstudio vs sim name label) - kubernetes.mdx: install commands were missing required CRON_SECRET and postgresql password (failed at template time), wrong deployment name in port-forward, stale version requirements, unsupported key-remapping claim - remove unimplemented app.secrets.existingSecret.keys from values + schema; fix README PDB default, cronjob list, /metrics caveat, NOTES secret count, Azure-only StorageClass in generic examples, dead SOCKET_SERVER_URL and GOOGLE_CLOUD_* env, ESO apiVersion mismatch, and skill-reference drift - bump chart to 1.0.1 * fix(helm): review round 1 — scoped example egress, in-tab secret note, copilot Job wording - external-db example egress scopes to a placeholder database CIDR instead of to: [] (which allowed every destination on 5432, defeating the isolation the example teaches) - kubernetes.mdx cloud tabs state explicitly that they reuse the variables generated in the Installation block - Copilot migrations really do run as a Helm-hook Job — restore Job wording there (only the app migrations are an init container) * fix(helm): template-sweep fixes — telemetry validity, ESO rollout checksums, dead passwordKey knob - telemetry: memory_limiter gets the required check_interval (collector failed startup validation whenever telemetry.enabled=true); the jaeger exporter was removed from collector-contrib in v0.86 — export to Jaeger via its native OTLP endpoint instead (otlp/jaeger, default port 4317) - app/realtime rollout checksums now hash the ExternalSecret manifest too, mirroring the copilot pattern — with ESO enabled the inline Secret renders empty, so remoteRefs changes never rolled the pods - remove the unimplemented existingSecret.passwordKey knob (values, schema, README, dead helpers): nothing consumed it, and a non-default value silently produced a DATABASE_URL with an unexpandable placeholder; secrets must use the standard POSTGRES_PASSWORD / EXTERNAL_DB_PASSWORD keys - drop the orphaned sim.migrations.labels helper (its only consumer was the dead NetworkPolicy rule removed earlier) - helm test pod image resolves through sim.image so global.imageRegistry mirroring applies; NetworkPolicy realtime-ingress comment reflects actual traffic direction; smoke unittest suite loads the newly referenced external-secret template * chore(helm): bump chart to 1.1.0 with upgrade notes Removing (inert) documented values keys and changing the rollout-checksum inputs is a values-surface change — per SemVer chart conventions that is more than a patch. Adds an Upgrading section documenting the one-time pod roll, the removed no-op keys, and the Jaeger-over-OTLP change. * fix(helm): review round — external-db NP opt-in with real-CIDR-first flow, prod Jaeger OTLP endpoint - external-db example ships networkPolicy disabled so a verbatim install always reaches the database; the scoped egress rule stays as the documented opt-in (set your CIDR first, then enable) - values-production still pointed telemetry.jaeger at the legacy 14250 collector port — now Jaeger's OTLP gRPC endpoint to match the otlp/jaeger exporter * feat(helm): autoscaling.realtime.enabled toggle so examples can scale the app without unsafe realtime replicas Cursor correctly flagged that comment-level warnings didn't stop a verbatim production/external-db install from running the realtime HPA at minReplicas 2 without REDIS_URL (silent cross-pod event loss). Adds an opt-out toggle (default true — existing deployments unchanged): the realtime HPA renders only when autoscaling.realtime.enabled, and the realtime Deployment keeps spec.replicas under its control when the HPA is excluded. The three autoscaling examples set it false with the Redis rationale; README and upgrade notes document the toggle. * fix(helm): review round — whitelabeled realtime HPA opt-out, external-db isolation on with required CIDR in install flow - values-whitelabeled now actually sets autoscaling.realtime.enabled: false (the earlier batch aborted before reaching this file — Cursor caught it) - external-db keeps networkPolicy enabled (no isolation regression); the DB egress CIDR is marked REQUIRED and wired into both documented install commands via --set, so the copy-paste flow sets the real subnet in the same breath as the DB host * fix(helm): use a syntactically valid example CIDR in the external-db install commands An unreplaced <YOUR_DB_CIDR> literal fails Kubernetes CIDR validation and aborts the install; every sibling placeholder in the same command (host, username) installs fine and simply doesn't connect until replaced. The CIDR now behaves the same way: valid example value (10.20.0.0/24), explicitly marked as the operator's database subnet in both commands and the values comment. * fix(helm): remove text after continuation backslashes in external-db install commands A trailing comment after the line-continuation backslash (and equally a comment line spliced mid-command) breaks the shell command when copied — the remaining --set overrides run as separate commands and required-value validation fails. Both documented commands now reconstruct to bash-clean multi-line invocations (verified with bash -n); the CIDR guidance lives in the networkPolicy section comment. * fix(docs): cloud-tab installs are alternatives via helm upgrade --install Following Installation and then a cloud tab ran helm install twice for the same release and failed on the second. The tabs now state they replace the generic install and use helm upgrade --install, which is idempotent and also converts an existing generic install to the cloud values. * fix(docs): honest conversion caveat for cloud-tab upgrades over an existing install The cloud values rename the bundled Postgres database to simstudio, which Postgres only applies at first initialization — an in-place conversion of a generic install would point DATABASE_URL at a nonexistent database. Document the two safe paths: keep the original name via --set, or uninstall + delete PVCs and install fresh. * fix(helm): external-db header install command declares its secrets and includes CRON_SECRET The primary documented command failed the chart's required-value validation (missing app.env.CRON_SECRET with cronjobs default-on) and referenced an undeclared DB_PASSWORD. It now shows the export lines for every variable it uses and sets CRON_SECRET; both commands verified with bash -n. * chore(helm): restore values.schema.json formatting — surgical deletions only The earlier programmatic edit reformatted the whole file (~390 lines of whitespace churn hiding the 12 real deleted lines). Re-applied the removal of the dead keys/passwordKey properties as text-level deletions preserving the original style. * fix(helm+docs): explicit secret exports in external-db header; conversion must reuse original secrets - all five export lines are written out (three were only named in a trailing comment, so a verbatim copy passed empty required values) - the cloud-conversion caveat now leads with reusing the original secret values (helm get values) — a regenerated ENCRYPTION_KEY makes previously encrypted credentials undecryptable
2026-09-24 15:45:35 +08:00 · 2026-07-23 18:28:11 -07:00
parent ca2ea0066d
commit d48722a04e
27 changed files with 254 additions and 174 deletions
@@ -9,8 +9,8 @@ import { FAQ } from '@/components/ui/faq'

 ## Prerequisites

- Kubernetes 1.19+
- Helm 3.0+
+- Kubernetes 1.25+
+- Helm 3.8+
 - PV provisioner support

 ## Installation
@@ -23,47 +23,59 @@ git clone https://github.com/simstudioai/sim.git && cd sim
 BETTER_AUTH_SECRET=$(openssl rand -hex 32)
 ENCRYPTION_KEY=$(openssl rand -hex 32)
 INTERNAL_API_SECRET=$(openssl rand -hex 32)
+CRON_SECRET=$(openssl rand -hex 32)
+POSTGRES_PASSWORD=$(openssl rand -hex 24)

 # Install
 helm install sim ./helm/sim \
  --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
+  --set app.env.CRON_SECRET="$CRON_SECRET" \
+  --set postgresql.auth.password="$POSTGRES_PASSWORD" \
  --namespace simstudio --create-namespace
 ```

 ## Cloud-Specific Values

+These are cloud-tuned **alternatives** to the generic install above — pick one path, don't run both. The commands reuse the `$BETTER_AUTH_SECRET`, `$ENCRYPTION_KEY`, `$INTERNAL_API_SECRET`, `$CRON_SECRET`, and `$POSTGRES_PASSWORD` variables generated in [Installation](#installation) above, so run that block's `openssl` lines first in the same shell. They use `helm upgrade --install`, so they work whether or not a release exists yet. Two caveats when converting an existing generic install rather than starting fresh: (1) **reuse the original secret values** — recover them with `helm get values sim -n simstudio` if your shell no longer has them; supplying a newly generated `ENCRYPTION_KEY` makes every previously encrypted credential (OAuth tokens, provider keys, environment variables) undecryptable. (2) The cloud values rename the bundled PostgreSQL database to `simstudio`, but Postgres only applies that setting on first initialization — add `--set postgresql.auth.database=sim` to keep your existing database. If you'd rather start clean, `helm uninstall sim -n simstudio`, delete its PVCs, and run the cloud command fresh.
+
 <Tabs items={['AWS EKS', 'Azure AKS', 'GCP GKE']}>
  <Tab value="AWS EKS">
 ```bash
-helm install sim ./helm/sim \
+helm upgrade --install sim ./helm/sim \
  --values ./helm/sim/examples/values-aws.yaml \
  --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
+  --set app.env.CRON_SECRET="$CRON_SECRET" \
+  --set postgresql.auth.password="$POSTGRES_PASSWORD" \
  --set app.env.NEXT_PUBLIC_APP_URL="https://sim.yourdomain.com" \
  --namespace simstudio --create-namespace
 ```
  </Tab>
  <Tab value="Azure AKS">
 ```bash
-helm install sim ./helm/sim \
+helm upgrade --install sim ./helm/sim \
  --values ./helm/sim/examples/values-azure.yaml \
  --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
+  --set app.env.CRON_SECRET="$CRON_SECRET" \
+  --set postgresql.auth.password="$POSTGRES_PASSWORD" \
  --set app.env.NEXT_PUBLIC_APP_URL="https://sim.yourdomain.com" \
  --namespace simstudio --create-namespace
 ```
  </Tab>
  <Tab value="GCP GKE">
 ```bash
-helm install sim ./helm/sim \
+helm upgrade --install sim ./helm/sim \
  --values ./helm/sim/examples/values-gcp.yaml \
  --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
+  --set app.env.CRON_SECRET="$CRON_SECRET" \
+  --set postgresql.auth.password="$POSTGRES_PASSWORD" \
  --set app.env.NEXT_PUBLIC_APP_URL="https://sim.yourdomain.com" \
  --namespace simstudio --create-namespace
 ```
@@ -115,7 +127,7 @@ externalDatabase:

 ```bash
 # Port forward for local access
-kubectl port-forward deployment/sim-sim-app 3000:3000 -n simstudio
+kubectl port-forward deployment/sim-app 3000:3000 -n simstudio

 # View logs
 kubectl logs -l app.kubernetes.io/component=app -n simstudio --tail=100
@@ -130,7 +142,7 @@ helm uninstall sim --namespace simstudio
 <FAQ items={[
  { question: "What are the default resource limits for the Sim app pod?", answer: "The Helm chart defaults to 8 Gi memory limit / 4 Gi request and 2000m CPU limit / 1000m request per app pod. The pod runs as non-root (UID 1001) with fsGroup 1001 for security." },
  { question: "Can I use an external database instead of the bundled PostgreSQL?", answer: "Yes. Set postgresql.enabled to false and configure the externalDatabase section with your host, port, username, password, database name, and sslMode. The external database must have the pgvector extension installed." },
-  { question: "How do I manage secrets securely in Kubernetes?", answer: "The Helm chart supports pre-existing Kubernetes secrets via app.secrets.existingSecret. Set enabled to true and provide the secret name. This integrates with External Secrets Operator, HashiCorp Vault, Azure Key Vault, and similar tools. Key mappings can be customized if your secret uses different key names." },
+  { question: "How do I manage secrets securely in Kubernetes?", answer: "The Helm chart supports pre-existing Kubernetes secrets via app.secrets.existingSecret. Set enabled to true and provide the secret name. This integrates with External Secrets Operator, HashiCorp Vault, Azure Key Vault, and similar tools. The secret must use the standard key names (BETTER_AUTH_SECRET, ENCRYPTION_KEY, INTERNAL_API_SECRET, CRON_SECRET, ...) — it is consumed wholesale, so key remapping is not supported." },
  { question: "Can I scale the app to multiple replicas?", answer: "Yes. Set app.replicaCount in your values.yaml. The chart supports standard Kubernetes scaling. Ensure your database can handle the additional connections from multiple replicas." },
  { question: "Are there cloud-specific example configurations?", answer: "Yes. The Helm chart includes example values files for AWS EKS (values-aws.yaml), Azure AKS (values-azure.yaml), GCP GKE (values-gcp.yaml), as well as files for production, development, external databases, external secrets, and whitelabeled deployments." },
  { question: "What ingress options are supported?", answer: "The chart supports ingress with configurable className (e.g., nginx), TLS termination, and separate host configuration for the app. Enable it with ingress.enabled: true and set your host under ingress.app.host." },
@@ -121,7 +121,7 @@ These are non-negotiable. Violating any of these has burned users in the past.
 2. **Never set `image.tag: latest`.** The chart defaults to `Chart.AppVersion` for a reason — reproducible rollouts. If the user pinned `latest`, push back.
 3. **Never edit chart templates to work around a `fail` statement.** The validation exists because a misconfiguration would otherwise surface as a runtime CrashLoopBackOff with cryptic env errors.
 4. **Never drop `automountServiceAccountToken: false`** unless the workload genuinely needs in-cluster API access (Sim's app/realtime/postgres pods do not).
-5. **Never `kubectl delete sts` without `--cascade=orphan`** on a live Postgres. It deletes the pods and PVCs.
+5. **Never `kubectl delete sts` without `--cascade=orphan`** on a live Postgres. It deletes the pods (PVCs survive, but the database goes down immediately).
 6. **Never tell a user "the chart works on your cluster" without `helm lint` + `helm template` against their values.** Static reading is not validation.
 7. **Always confirm before `helm uninstall` in a shared namespace.** PVCs survive but other namespace resources may not.

@@ -136,7 +136,7 @@ kubectl --namespace <ns> get pods,events --sort-by='.lastTimestamp'
 kubectl --namespace <ns> logs deploy/sim-app --tail=200
 kubectl --namespace <ns> logs deploy/sim-realtime --tail=200
 kubectl --namespace <ns> logs sts/sim-postgresql --tail=200
-kubectl --namespace <ns> logs job/sim-migrations --tail=200 2>/dev/null
+kubectl --namespace <ns> logs deploy/sim-app -c migrations --tail=200 2>/dev/null
 kubectl --namespace <ns> describe pod -l app.kubernetes.io/name=sim
 ```

@@ -24,7 +24,7 @@ export POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=')  # if using ch
 | `INTERNAL_API_SECRET` | Shared auth between `sim-app` ↔ `sim-realtime` pods | 32 bytes = 64 hex chars | Both deployments must roll together — temporary realtime errors during the rollout |
 | `CRON_SECRET` | Authenticates scheduled CronJob pods to the app | 32 bytes = 64 hex chars | Rotating just needs `helm upgrade`; next cron run uses the new value |
 | `API_ENCRYPTION_KEY` (optional) | Encrypts user-stored API keys (OpenAI tokens, etc.) at rest in Postgres | **Exactly 64 hex chars** (the app rejects other lengths) | Without it, keys are stored plain. Once set, never rotate without a migration |
-| `POSTGRES_PASSWORD` (chart-bundled Postgres only) | Postgres superuser password | Any length ≥ 12 chars matching `^[a-zA-Z0-9._-]+$` | Requires Postgres pod restart + app rollout |
+| `POSTGRES_PASSWORD` (chart-bundled Postgres only) | Postgres superuser password | ≥ 8 chars (schema-enforced) matching `^[a-zA-Z0-9._-]+$`; 32-byte hex recommended | Requires Postgres pod restart + app rollout |

 The `^[a-zA-Z0-9._-]+$` constraint on the Postgres password exists because the chart embeds the password into `DATABASE_URL` without URL-encoding. The `tr -d '/+='` strips the three problematic characters from `openssl rand -base64` output. The chart enforces this regex at template time.

@@ -11,7 +11,7 @@ kubectl --namespace $NS describe pod -l app.kubernetes.io/instance=sim
 kubectl --namespace $NS logs deploy/sim-app --tail=200
 kubectl --namespace $NS logs deploy/sim-realtime --tail=200
 kubectl --namespace $NS logs sts/sim-postgresql --tail=200
-kubectl --namespace $NS logs job/sim-migrations --tail=200 2>/dev/null || true
+kubectl --namespace $NS logs deploy/sim-app -c migrations --tail=200 2>/dev/null || true
 ```

 ---
@@ -63,9 +63,9 @@ Match the error:
 |---|---|---|
 | `Invalid env: ... NEXT_PUBLIC_APP_URL: Invalid url` | URL field set to empty string or invalid format | Set `app.env.NEXT_PUBLIC_APP_URL` to a valid URL — `https://sim.example.com` in prod, `http://localhost:3000` in dev |
 | `getaddrinfo ENOTFOUND ... -postgresql` / `connect ECONNREFUSED` | App can't reach Postgres | Check `kubectl get pod -l app.kubernetes.io/name=postgresql` is `Running`; check `postgresql.auth.password` matches the password in the Secret |
-| `password authentication failed for user "sim"` | Postgres password rotated but app pod wasn't restarted, OR password contains URL-unsafe chars | `kubectl rollout restart deploy/sim-app -n sim`; regenerate password with `openssl rand -base64 24 \| tr -d '/+='` |
+| `password authentication failed for user "postgres"` | Postgres password rotated but app pod wasn't restarted, OR password contains URL-unsafe chars | `kubectl rollout restart deploy/sim-app -n sim`; regenerate password with `openssl rand -base64 24 \| tr -d '/+='` |
 | `BETTER_AUTH_SECRET is missing` / `INTERNAL_API_SECRET is required` | Required env var not present in the Secret | Verify with `kubectl get secret sim-app-secrets -o jsonpath='{.data}' \| jq 'keys'`; if missing, fix your secret strategy |
-| `Migration failed` or app starts before migration | Migration Job hasn't completed | `kubectl logs job/sim-migrations -n sim`; rerun with `kubectl delete job/sim-migrations && helm upgrade ...` |
+| `Migration failed` | Migrations init container on the app pod failed | `kubectl logs deploy/sim-app -c migrations -n sim`; rerun with `kubectl rollout restart deploy/sim-app -n sim` (migrations run before each app pod starts, so the app can never start ahead of them) |

 ---

@@ -112,7 +112,7 @@ kubectl describe ingress -n sim
 | No `ADDRESS` in `kubectl get ingress` | Ingress controller not installed — install `ingress-nginx`, AWS LBC, GCP LB controller, etc. |
 | `ingressClassName` doesn't match installed controller | `kubectl get ingressclass` to list installed classes, set `ingress.className` to match |
 | Address is set but DNS resolves to wrong IP | `dig <your-host>` — point DNS at the ingress controller's external IP / LoadBalancer / CNAME |
-| TLS cert errors | If using cert-manager, check `kubectl describe certificate -n sim`; verify `ingress.tls.issuerRef` |
+| TLS cert errors | If using cert-manager, check `kubectl describe certificate -n sim`; verify the `cert-manager.io/cluster-issuer` annotation on the ingress |
 | `503 Service Unavailable` | Ingress routing is fine but app pod isn't `Ready` — go back to the diagnostic block |

 ---
@@ -153,7 +153,7 @@ Bump the relevant resource limit. Defaults:
 |---|---|---|
 | `app` | `1000m` CPU / `4Gi` memory | `2000m` CPU / `8Gi` memory |
 | `realtime` | `250m` CPU / `512Mi` memory | `500m` CPU / `1Gi` memory |
-| `postgresql` | `250m` CPU / `512Mi` memory | `1000m` CPU / `2Gi` memory |
+| `postgresql` | `500m` CPU / `1Gi` memory | `2Gi` memory (no CPU limit) |

 Override in values:

@@ -172,7 +172,7 @@ app:

 ```bash
 helm version
-kubectl version --short
+kubectl version
 helm get values sim -n sim --revision $(helm history sim -n sim | tail -1 | awk '{print $1}')
 kubectl get all,pvc,ingress,externalsecret -n sim -o wide
 kubectl describe pods -n sim -l app.kubernetes.io/instance=sim | head -200
@@ -27,7 +27,7 @@ The Sim chart splits configuration across **four** layers. Understanding which l
                                  ▼
 ┌───────────────────────────────────────────────────────────────────────────┐
 │ Layer 3: chart-computed (inline env: on the Deployment)                    │
-│   → DATABASE_URL, SOCKET_SERVER_URL, OLLAMA_URL                           │
+│   → DATABASE_URL, SOCKET_SERVER_URL, OLLAMA_URL, PII_URL                           │
 │   → Derived from postgresql.* / externalDatabase.* / service.* values     │
 │   → CANNOT be overridden via app.env — chart filters them out             │
 └───────────────────────────────────────────────────────────────────────────┘
@@ -63,7 +63,7 @@ The exhaustive list of keys per layer lives in `helm/sim/values.yaml`. Read the
 | Rate limits | 2 (envDefaults) | `RATE_LIMIT_WINDOW_MS`, `RATE_LIMIT_FREE_SYNC`, etc. |
 | Execution timeouts | 2 (envDefaults) | `EXECUTION_TIMEOUT_FREE`, `EXECUTION_TIMEOUT_PRO`, etc. |
 | IVM pool / quotas | 2 (envDefaults) | `IVM_POOL_SIZE`, `IVM_MAX_CONCURRENT`, `IVM_MAX_PER_WORKER`, etc. |
-| Connection strings | 3 (chart-computed) | `DATABASE_URL`, `SOCKET_SERVER_URL`, `OLLAMA_URL` |
+| Connection strings | 4 (chart-computed) | `DATABASE_URL`, `SOCKET_SERVER_URL`, `OLLAMA_URL`, `PII_URL` |
 | Custom downward API / configMapKeyRef | 4 (extraEnvVars) | anything that needs `valueFrom:` |

 ## Common authoring patterns
@@ -2,7 +2,7 @@ apiVersion: v2
 name: sim
 description: A Helm chart for Sim - the open-source AI workspace where teams build, deploy, and manage AI agents
 type: application
-version: 1.0.0
+version: 1.1.0
 appVersion: "0.6.73"
 kubeVersion: ">=1.25.0-0"
 home: https://sim.ai
@@ -40,8 +40,8 @@ This chart deploys the Sim platform on a Kubernetes cluster using the Helm packa
 * **`app`** — the Sim Next.js web application (Deployment).
 * **`realtime`** — the WebSocket service for live workflow updates (Deployment).
 * **`postgresql`** — an in-cluster `pgvector/pgvector` Postgres (StatefulSet, with a headless Service for stable per-pod DNS).
-* **`migrations`** — a Job that applies database migrations on install/upgrade.
-* **`cronjobs`** — scheduled jobs for workflow schedule execution, inbox/calendar/drive polling (Gmail, Outlook, Calendar, Drive, Sheets, IMAP, RSS), workspace event polling, subscription renewal, data drains, and connector syncs.
+* **`migrations`** — an init container on the app Deployment that applies database migrations before each app pod starts.
+* **`cronjobs`** — scheduled jobs for workflow schedule execution, inbox/calendar/drive polling (Gmail, Outlook, Calendar, Drive, Sheets, IMAP, RSS), workspace event and HubSpot webhook polling, outbox processing, subscription renewal, billing-seat and inbox-entitlement reconciliation, time-pause/resume polling, data drains, and connector syncs.
 * **`serviceaccount`** — a dedicated ServiceAccount with `automountServiceAccountToken: false`.

 Optional components (off by default):
@@ -214,7 +214,7 @@ cat ./helm/sim/values.schema.json

 Before installing in production, confirm each of the following:

-* **High availability** — scale `app.replicaCount > 1`. The chart auto-creates a `PodDisruptionBudget` with `minAvailable: 1`. Set `podDisruptionBudget.maxUnavailable: "25%"` for a more permissive policy or `minAvailable: "50%"` for a stricter one.
+* **High availability** — scale `app.replicaCount > 1`. The chart auto-creates a `PodDisruptionBudget` with `maxUnavailable: "25%"`. Set `podDisruptionBudget.minAvailable` instead for a stricter policy.
 * **Pinned images** — override `image.tag` (or `image.digest`) with an explicit version. Do not rely on the chart's default tag in production.
 * **Secrets management** — provide secrets via External Secrets Operator (ESO) or pre-created Kubernetes Secrets. Never commit secrets to `values.yaml`.
 * **TLS / Ingress** — set the `cert-manager.io/cluster-issuer` annotation on the ingress and tune `proxy-body-size` / `proxy-read-timeout` for your workload. See commented examples in `values.yaml`.
@@ -271,8 +271,7 @@ postgresql:
  auth:
    existingSecret:
      enabled: true
-      name: sim-postgres-secret
-      passwordKey: POSTGRES_PASSWORD
+      name: sim-postgres-secret   # must contain the password under the key POSTGRES_PASSWORD
 ```

 See `examples/values-existing-secret.yaml`.
@@ -357,7 +356,7 @@ autoscaling:
  targetMemoryUtilizationPercentage: 80
 ```

-When `autoscaling.enabled=true`, the chart omits `spec.replicas` from the Deployment so the HPA owns replica count. Requires `metrics-server` in the cluster.
+When `autoscaling.enabled=true`, the chart omits `spec.replicas` from the Deployment so the HPA owns replica count. Requires `metrics-server` in the cluster. The realtime Deployment gets the same HPA unless `autoscaling.realtime.enabled=false` — scale realtime past one replica only with `REDIS_URL` set (Socket.IO Redis adapter), or cross-pod collaboration events are dropped.

 ---

@@ -370,7 +369,7 @@ monitoring:
    interval: 30s
 ```

-Requires the Prometheus Operator CRDs. Scrapes `/metrics` on the app and realtime services.
+Requires the Prometheus Operator CRDs. Scrapes `/metrics` on the app and realtime services — note the default images do not currently expose a `/metrics` endpoint, so enable this only with a build that does.

 ---

@@ -427,7 +426,7 @@ Common causes:

 * `NEXT_PUBLIC_APP_URL` still set to `http://localhost:3000` in a clustered deploy → set it to your public origin.
 * `DATABASE_URL` not reachable → check the Postgres pod is running and `postgresql.auth.password` matches.
-* Missing migration → check `kubectl logs job/sim-migrations`.
+* Missing migration → check `kubectl logs deploy/sim-app -c migrations` (migrations run as an init container on the app pod).

 ### Image pull errors (`ErrImagePull` / `ImagePullBackOff`)

@@ -463,11 +462,19 @@ kubectl describe ingress --namespace sim
 kubectl --namespace sim logs -f deployment/sim-app
 kubectl --namespace sim logs -f deployment/sim-realtime
 kubectl --namespace sim logs -f statefulset/sim-postgresql
-kubectl --namespace sim logs job/sim-migrations
+kubectl --namespace sim logs deploy/sim-app -c migrations
 ```

 ---

+## Upgrading to 1.1.0
+
+No action is required for working configurations. Notes:
+
+* Pods for `app` and `realtime` roll once on upgrade (their rollout checksum now also covers the ExternalSecret manifest, fixing missed rollouts in ESO mode).
+* Two values keys that were never consumed by any template were removed: `app.secrets.existingSecret.keys` and `*.existingSecret.passwordKey`. Existing secrets must use the standard key names (`BETTER_AUTH_SECRET`, ..., `POSTGRES_PASSWORD`, `EXTERNAL_DB_PASSWORD`); leftover keys in your values file are ignored, not rejected.
+* `telemetry.jaeger` now exports over OTLP (`otlp/jaeger`) — point `telemetry.jaeger.endpoint` at Jaeger's OTLP gRPC port (4317). The previous `jaeger` exporter did not exist in the pinned collector image, so any prior jaeger-enabled config was already failing at collector startup.
+
 ## Support

 * **Docs:** https://docs.sim.ai
@@ -8,7 +8,9 @@
 #   - EKS cluster (Kubernetes 1.25+)
 #   - EBS CSI driver add-on (aws eks create-addon --addon-name aws-ebs-csi-driver)
 #   - AWS Load Balancer Controller (for ALB ingress)
-#   - (recommended) cert-manager OR an ACM certificate ARN for the ingress
+#   - An ACM certificate for your domains (ALB auto-discovers it from the ingress
+#     hosts, or pin it with alb.ingress.kubernetes.io/certificate-arn). cert-manager
+#     does not work with ALB — ALB cannot serve Kubernetes TLS Secrets
 #
 # Install:
 #   helm install sim ./helm/sim \
@@ -110,7 +112,11 @@ app:
 # Realtime service
 realtime:
  enabled: true
-  replicaCount: 2
+  # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime
+  # shares the app secret): Socket.IO only bridges events across pods through
+  # its Redis adapter, so multi-replica realtime without Redis silently drops
+  # cross-pod collaboration updates.
+  replicaCount: 1

  # Node selector for realtime pods
  # Uncomment and customize based on your EKS node labels:
@@ -171,7 +177,7 @@ postgresql:
  # Persistent storage using AWS EBS volumes
  persistence:
    enabled: true
-    storageClass: "gp2"  # Use gp2 (default) or create gp3 StorageClass
+    storageClass: "gp3"  # Matches the global gp3 StorageClass above
    size: 50Gi
    accessModes:
      - ReadWriteOnce
@@ -222,7 +228,7 @@ ollama:
  # High-performance storage for AI models
  persistence:
    enabled: true
-    storageClass: "gp2"  # Use gp2 (default) or create gp3 StorageClass
+    storageClass: "gp3"  # Matches the global gp3 StorageClass above
    size: 100Gi
    accessModes:
      - ReadWriteOnce
@@ -239,7 +245,6 @@ ingress:
  className: alb
  
  annotations:
-    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/ssl-redirect: "443"
@@ -284,7 +289,7 @@ affinity:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
-                values: ["simstudio"]
+                values: ["sim"]
          topologyKey: kubernetes.io/hostname
      - weight: 50
        podAffinityTerm:
@@ -292,7 +297,7 @@ affinity:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
-                values: ["simstudio"]
+                values: ["sim"]
          topologyKey: topology.kubernetes.io/zone

 # Service Account with IAM roles for service account (IRSA) integration
@@ -7,8 +7,8 @@
 # Prerequisites:
 #   - AKS cluster (Kubernetes 1.25+)
 #   - NGINX ingress controller installed
-#   - (optional) cert-manager for TLS
-#   - (optional) GPU node pool labeled with role: datalake / accelerator: nvidia
+#   - cert-manager installed with a ClusterIssuer (issues the ingress TLS secret)
+#   - (optional) GPU node pool tainted with sku=gpu:NoSchedule for Ollama
 #
 # Install:
 #   helm install sim ./helm/sim \
@@ -127,7 +127,11 @@ app:
 # Realtime service
 realtime:
  enabled: true
-  replicaCount: 2
+  # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime
+  # shares the app secret): Socket.IO only bridges events across pods through
+  # its Redis adapter, so multi-replica realtime without Redis silently drops
+  # cross-pod collaboration updates.
+  replicaCount: 1

  # Node selector for realtime pods
  # Uncomment and customize based on your AKS node labels:
@@ -242,6 +246,10 @@ ingress:

  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+    # cert-manager issues the certificate into ingress.tls.secretName below —
+    # without this annotation (and cert-manager installed) nginx serves its
+    # self-signed fake certificate forever.
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"

  # Main application
  app:
@@ -1,6 +1,6 @@
 # values-copilot.yaml
 #
-# When to use: enable the Sim Copilot service (the chat / Mothership backend)
+# When to use: enable the Sim Copilot service (the backend for Chat — the Sim agent)
 # alongside the main app. Provisions a dedicated Copilot Deployment, its own
 # Postgres StatefulSet, and a migration Job for the Copilot schema.
 #
@@ -7,7 +7,7 @@
 #
 # Prerequisites:
 #   Create the Secret objects before `helm install`. Example:
-#     kubectl create secret generic my-app-secrets --namespace sim \
+#     kubectl create secret generic sim-app-secrets --namespace sim \
 #       --from-literal=BETTER_AUTH_SECRET=$(openssl rand -hex 32) \
 #       --from-literal=ENCRYPTION_KEY=$(openssl rand -hex 32) \
 #       --from-literal=INTERNAL_API_SECRET=$(openssl rand -hex 32) \
@@ -34,7 +34,11 @@ app:

 realtime:
  enabled: true
-  replicaCount: 2
+  # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime
+  # shares the app secret): Socket.IO only bridges events across pods through
+  # its Redis adapter, so multi-replica realtime without Redis silently drops
+  # cross-pod collaboration updates.
+  replicaCount: 1
  env:
    NEXT_PUBLIC_APP_URL: "https://sim.example.com"
    BETTER_AUTH_URL: "https://sim.example.com"
@@ -11,7 +11,13 @@
 #   - Network reachability from the cluster (peering, private link, or public
 #     endpoint with TLS)
 #
-# Install:
+# Install — generate each secret first in the same shell:
+#   export BETTER_AUTH_SECRET=$(openssl rand -hex 32)
+#   export ENCRYPTION_KEY=$(openssl rand -hex 32)
+#   export INTERNAL_API_SECRET=$(openssl rand -hex 32)
+#   export CRON_SECRET=$(openssl rand -hex 32)
+#   export DB_PASSWORD=$(openssl rand -hex 24)
+#
 #   helm install sim ./helm/sim \
 #     --namespace sim --create-namespace \
 #     --values ./helm/sim/examples/values-external-db.yaml \
@@ -19,9 +25,11 @@
 #     --set externalDatabase.username=simstudio_user \
 #     --set externalDatabase.password="$DB_PASSWORD" \
 #     --set externalDatabase.database=simstudio_prod \
+#     --set "networkPolicy.egress[0].to[0].ipBlock.cidr=10.20.0.0/24" \
 #     --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
 #     --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
-#     --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET"
+#     --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
+#     --set app.env.CRON_SECRET="$CRON_SECRET"

 # Global configuration
 global:
@@ -63,7 +71,12 @@ app:
 # Realtime service
 realtime:
  enabled: true
-  replicaCount: 2
+  # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime
+  # shares the app secret): Socket.IO only bridges events across pods through
+  # its Redis adapter, so multi-replica realtime without Redis silently drops
+  # cross-pod collaboration updates. autoscaling.realtime.enabled is false
+  # below for the same reason — flip both once REDIS_URL is set.
+  replicaCount: 1
  
  resources:
    limits:
@@ -101,9 +114,9 @@ externalDatabase:
  enabled: true
  
  # Database connection details (REQUIRED - configure for your external database)
-  host: ""                    # Database hostname (e.g., "postgres.acme.com" or RDS endpoint)
+  host: "postgres.acme.com"   # REQUIRED - your database hostname (RDS/Cloud SQL/Azure endpoint or IP)
  port: 5432
-  username: ""                # Database username (e.g., "simstudio_user")
+  username: "simstudio_user"  # REQUIRED - your database username
  password: ""                # Database password - set via --set flag or external secret
  database: ""                # Database name (e.g., "simstudio_production")
  
@@ -142,6 +155,11 @@ ingress:
 # Production-ready features (autoscaling, monitoring, etc.)
 autoscaling:
  enabled: true
+  # Keep realtime out of the HPA until REDIS_URL is set (Socket.IO Redis
+  # adapter) — multi-replica realtime without Redis silently drops cross-pod
+  # collaboration events.
+  realtime:
+    enabled: false
  minReplicas: 2
  maxReplicas: 20
  targetCPUUtilizationPercentage: 70
@@ -161,14 +179,19 @@ monitoring:

 networkPolicy:
  enabled: true
-  # Custom egress rules to allow database connectivity to an external DB.
-  # The chart's app/realtime NetworkPolicies already permit egress to
-  # `postgresql.service.targetPort` when `postgresql.enabled=true`. For an
-  # external DB on a different port (or off-cluster), append to extraRules.
+  # `networkPolicy.egress` is a LIST of NetworkPolicy egress rules. With the
+  # policy on, the chart's egress allowlist only covers the in-cluster
+  # Postgres, so this rule is what lets the pods reach your external database.
+  # The CIDR below is an example, exactly like `your-db.example.com` in the
+  # install command: REQUIRED to change — set your database's subnet (or /32
+  # host) via the --set line in the install commands below, or edit it here.
+  # A wrong CIDR blocks migrations and every database connection; an empty
+  # `to:` would allow ANY host on 5432 and defeat the isolation.
  egress:
-    extraRules:
-      - to: []
-        ports:
+    - to:
+        - ipBlock:
+            cidr: 10.0.0.0/16 # REQUIRED - your database subnet or /32 host CIDR
+      ports:
        - protocol: TCP
          port: 5432

@@ -179,6 +202,7 @@ networkPolicy:
 #   --set externalDatabase.username="your-db-user" \
 #   --set externalDatabase.password="your-db-password" \
 #   --set externalDatabase.database="your-db-name" \
+#   --set "networkPolicy.egress[0].to[0].ipBlock.cidr=10.20.0.0/24" \
 #   --set app.env.BETTER_AUTH_SECRET="$(openssl rand -hex 32)" \
 #   --set app.env.ENCRYPTION_KEY="$(openssl rand -hex 32)" \
 #   --set app.env.INTERNAL_API_SECRET="$(openssl rand -hex 32)" \
@@ -63,7 +63,11 @@ app:

 realtime:
  enabled: true
-  replicaCount: 2
+  # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime
+  # shares the app secret): Socket.IO only bridges events across pods through
+  # its Redis adapter, so multi-replica realtime without Redis silently drops
+  # cross-pod collaboration updates.
+  replicaCount: 1
  envDefaults:
    NEXT_PUBLIC_APP_URL: "https://sim.example.com"
    BETTER_AUTH_URL: "https://sim.example.com"
@@ -81,7 +85,7 @@ postgresql:
 # ---

 # Azure Key Vault (Workload Identity):
-# apiVersion: external-secrets.io/v1beta1
+# apiVersion: external-secrets.io/v1
 # kind: ClusterSecretStore
 # metadata:
 #   name: sim-secret-store
@@ -95,7 +99,7 @@ postgresql:
 #         namespace: external-secrets

 # AWS Secrets Manager (IRSA):
-# apiVersion: external-secrets.io/v1beta1
+# apiVersion: external-secrets.io/v1
 # kind: ClusterSecretStore
 # metadata:
 #   name: sim-secret-store
@@ -107,7 +111,7 @@ postgresql:
 #       role: arn:aws:iam::123456789012:role/external-secrets-role

 # HashiCorp Vault (Kubernetes Auth):
-# apiVersion: external-secrets.io/v1beta1
+# apiVersion: external-secrets.io/v1
 # kind: ClusterSecretStore
 # metadata:
 #   name: sim-secret-store
@@ -7,7 +7,7 @@
 # Prerequisites:
 #   - GKE cluster (Kubernetes 1.25+)
 #   - Workload Identity enabled (for IAM-bound ServiceAccount)
-#   - (optional) Managed certificate or cert-manager for TLS
+#   - A GKE ManagedCertificate named simstudio-ssl-cert (creation manifest in the ingress section below) — or cert-manager if you prefer
 #   - (optional) GPU node pool with accelerator labels for Ollama
 #
 # Install:
@@ -86,10 +86,6 @@ app:
    NODE_ENV: "production"
    NEXT_TELEMETRY_DISABLED: "1"
    
-    # GCP-specific environment variables
-    GOOGLE_CLOUD_PROJECT: "your-project-id"
-    GOOGLE_CLOUD_REGION: "us-central1"
-
    # Google Cloud Storage Configuration (RECOMMENDED for production)
    # Create GCS buckets in your project and grant the Workload Identity
    # service account roles/storage.objectAdmin on them. Signed-URL generation
@@ -117,7 +113,11 @@ app:
 # Realtime service
 realtime:
  enabled: true
-  replicaCount: 2
+  # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime
+  # shares the app secret): Socket.IO only bridges events across pods through
+  # its Redis adapter, so multi-replica realtime without Redis silently drops
+  # cross-pod collaboration updates.
+  replicaCount: 1

  # Node selector for realtime pods
  # Uncomment and customize based on your GKE node labels:
@@ -266,10 +266,29 @@ ingress:
      - path: /
        pathType: Prefix
  
-  # TLS configuration
+  # TLS comes from the GKE ManagedCertificate referenced by the
+  # networking.gke.io/managed-certificates annotation above. The chart does NOT
+  # create that resource — create it once before installing (the certificate
+  # provisions automatically after DNS resolves; allow ~15-30 minutes):
+  #
+  #   kubectl create namespace sim
+  #   cat <<'EOF' | kubectl apply -f -
+  #   apiVersion: networking.gke.io/v1
+  #   kind: ManagedCertificate
+  #   metadata:
+  #     name: simstudio-ssl-cert
+  #     namespace: sim
+  #   spec:
+  #     domains:
+  #       - simstudio.acme.com
+  #       - simstudio-ws.acme.com
+  #   EOF
+  #
+  # Keep the chart's secret-based TLS off with GCE managed certificates — an
+  # enabled spec.tls references a Secret nothing creates, and the GKE ingress
+  # controller reports sync errors for it.
  tls:
-    enabled: true
-    secretName: simstudio-tls-secret
+    enabled: false

 # Pod disruption budget for high availability
 podDisruptionBudget:
@@ -291,7 +310,7 @@ affinity:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
-                values: ["simstudio"]
+                values: ["sim"]
          topologyKey: kubernetes.io/hostname
      - weight: 50
        podAffinityTerm:
@@ -299,7 +318,7 @@ affinity:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
-                values: ["simstudio"]
+                values: ["sim"]
          topologyKey: topology.gke.io/zone

 # Service Account with Workload Identity integration
@@ -308,21 +327,27 @@ serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account: "simstudio@your-project-id.iam.gserviceaccount.com"

-# Additional environment variables for GCP service integration
-extraEnvVars:
-  - name: GOOGLE_APPLICATION_CREDENTIALS
-    value: "/var/secrets/google/key.json"
-
-# Additional volumes for service account credentials
-extraVolumes:
-  - name: google-cloud-key
-    secret:
-      secretName: google-service-account-key
-
-extraVolumeMounts:
-  - name: google-cloud-key
-    mountPath: /var/secrets/google
-    readOnly: true
+# Key-file credentials — ONLY if NOT using Workload Identity (the serviceAccount
+# annotation above is the recommended path and needs no key file; a mounted
+# GOOGLE_APPLICATION_CREDENTIALS overrides Workload Identity/ADC).
+# If you do use a key file, create the Secret first or every pod is stuck in
+# ContainerCreating on the missing volume:
+#   kubectl -n sim create secret generic google-service-account-key \
+#     --from-file=key.json=/path/to/service-account-key.json
+#
+# extraEnvVars:
+#   - name: GOOGLE_APPLICATION_CREDENTIALS
+#     value: "/var/secrets/google/key.json"
+#
+# extraVolumes:
+#   - name: google-cloud-key
+#     secret:
+#       secretName: google-service-account-key
+#
+# extraVolumeMounts:
+#   - name: google-cloud-key
+#     mountPath: /var/secrets/google
+#     readOnly: true
 # -----------------------------------------------------------------------------
 # External Secrets Operator (ESO) — recommended for production on GCP
 # -----------------------------------------------------------------------------
@@ -26,7 +26,9 @@
 global:
  imageRegistry: "ghcr.io"
  # For production, use a StorageClass with reclaimPolicy: Retain
-  storageClass: "managed-csi-premium"
+  # Set to YOUR cluster's StorageClass: gp3 (EKS), standard-rwo (GKE),
+  # managed-csi-premium (AKS). Empty uses the cluster default.
+  storageClass: ""

 # Main application
 app:
@@ -45,7 +47,6 @@ app:
  env:
    NEXT_PUBLIC_APP_URL: "https://sim.acme.ai"
    BETTER_AUTH_URL: "https://sim.acme.ai"
-    SOCKET_SERVER_URL: "https://sim-ws.acme.ai"
    NEXT_PUBLIC_SOCKET_URL: "https://sim-ws.acme.ai"
    
    # Security settings (REQUIRED - replace with your own secure secrets)
@@ -72,7 +73,12 @@ app:
 # Realtime service
 realtime:
  enabled: true
-  replicaCount: 2
+  # Keep realtime at 1 replica unless REDIS_URL is set (in app.env — realtime
+  # shares the app secret): Socket.IO only bridges events across pods through
+  # its Redis adapter, so multi-replica realtime without Redis silently drops
+  # cross-pod collaboration updates. autoscaling.realtime.enabled is false
+  # below for the same reason — flip both once REDIS_URL is set.
+  replicaCount: 1

  resources:
    limits:
@@ -121,10 +127,12 @@ postgresql:
  # Persistent storage configuration
  persistence:
    enabled: true
-    storageClass: "managed-csi-premium"
+    storageClass: ""  # your cluster's StorageClass (empty = cluster default)
    size: 50Gi
  
-  # SSL/TLS configuration (recommended for production)
+  # SSL/TLS configuration (recommended for production). Requires
+  # certManager.enabled below — the Certificate's issuerRef (sim-ca-issuer)
+  # only exists when the chart creates its CA issuer.
  tls:
    enabled: true
    certificatesSecret: postgres-tls-secret
@@ -168,9 +176,18 @@ ingress:
    enabled: true
    secretName: sim-tls-secret

+# Chart-managed CA issuer — required for postgresql.tls above (issuerRef sim-ca-issuer)
+certManager:
+  enabled: true
+
 # Horizontal Pod Autoscaler (automatically scales pods based on CPU/memory usage)
 autoscaling:
  enabled: true
+  # Keep realtime out of the HPA until REDIS_URL is set (Socket.IO Redis
+  # adapter) — multi-replica realtime without Redis silently drops cross-pod
+  # collaboration events.
+  realtime:
+    enabled: false
  minReplicas: 2
  maxReplicas: 20
  targetCPUUtilizationPercentage: 70
@@ -242,4 +259,6 @@ telemetry:
    endpoint: "http://prometheus-server/api/v1/write"
  jaeger:
    enabled: true
-    endpoint: "http://jaeger-collector:14250"
+    # Jaeger's OTLP gRPC endpoint (the chart exports via otlp/jaeger;
+    # Jaeger >=1.35 ingests OTLP natively)
+    endpoint: "jaeger-collector:4317"
@@ -19,7 +19,7 @@
 # Global configuration
 global:
  imageRegistry: "ghcr.io"
-  storageClass: "managed-csi-premium"
+  storageClass: ""  # your cluster's StorageClass (empty = cluster default)

 # Main application with custom branding
 app:
@@ -31,7 +31,6 @@ app:
    # Application URLs (update with your domain)
    NEXT_PUBLIC_APP_URL: "https://sim.acme.ai"
    BETTER_AUTH_URL: "https://sim.acme.ai"
-    SOCKET_SERVER_URL: "https://sim-ws.acme.ai"
    NEXT_PUBLIC_SOCKET_URL: "https://sim-ws.acme.ai"
    
    # Security settings (REQUIRED)
@@ -101,6 +100,11 @@ ingress:
 # Auto-scaling
 autoscaling:
  enabled: true
+  # Keep realtime out of the HPA until REDIS_URL is set (Socket.IO Redis
+  # adapter) — multi-replica realtime without Redis silently drops cross-pod
+  # collaboration events.
+  realtime:
+    enabled: false
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
@@ -46,7 +46,7 @@ Your release is named {{ .Release.Name }} in namespace {{ .Release.Namespace }}.

 3. Required secrets:

-   Sim requires three application secrets and a Postgres password to start.
+   Sim requires four application secrets and a Postgres password to start.
   {{- if .Values.externalSecrets.enabled }}

   Using External Secrets Operator. Verify the ExternalSecret has synced:
@@ -133,14 +133,6 @@ PII (Presidio) selector labels
 app.kubernetes.io/component: pii
 {{- end }}

-{{/*
-Migrations specific labels
-*/}}
-{{- define "sim.migrations.labels" -}}
-{{ include "sim.labels" . }}
-app.kubernetes.io/component: migrations
-{{- end }}
-
 {{/*
 Copilot specific labels
 */}}
@@ -393,18 +385,6 @@ Returns the name of the secret containing PostgreSQL password
 {{- end -}}
 {{- end }}

-{{/*
-Get the PostgreSQL password key name
-Returns the key name in the secret that contains the password
-*/}}
-{{- define "sim.postgresqlPasswordKey" -}}
-{{- if and .Values.postgresql.auth.existingSecret .Values.postgresql.auth.existingSecret.enabled -}}
-{{- .Values.postgresql.auth.existingSecret.passwordKey | default "POSTGRES_PASSWORD" -}}
-{{- else -}}
-{{- print "POSTGRES_PASSWORD" -}}
-{{- end -}}
-{{- end }}
-
 {{/*
 Get the external database secret name
 Returns the name of the secret containing external database password
@@ -417,18 +397,6 @@ Returns the name of the secret containing external database password
 {{- end -}}
 {{- end }}

-{{/*
-Get the external database password key name
-Returns the key name in the secret that contains the password
-*/}}
-{{- define "sim.externalDbPasswordKey" -}}
-{{- if and .Values.externalDatabase.existingSecret .Values.externalDatabase.existingSecret.enabled -}}
-{{- .Values.externalDatabase.existingSecret.passwordKey | default "EXTERNAL_DB_PASSWORD" -}}
-{{- else -}}
-{{- print "EXTERNAL_DB_PASSWORD" -}}
-{{- end -}}
-{{- end }}
-
 {{/*
 Check if app secrets should be created by the chart
 Returns true if we should create the app secrets (not using existing or ESO)
@@ -17,7 +17,12 @@ spec:
  template:
    metadata:
      annotations:
-        checksum/secret: {{ include (print $.Template.BasePath "/secrets-app.yaml") . | sha256sum }}
+        {{- /* Hash the inline Secret AND the ExternalSecret manifest so spec
+          changes under externalSecrets.remoteRefs.app roll the pods (with ESO
+          enabled the inline Secret renders empty and would hash to a constant).
+          Rotated values inside the external store still can't be detected at
+          render time — ESO's own reconciliation handles those. */}}
+        checksum/secret: {{ printf "%s%s" (include (print $.Template.BasePath "/secrets-app.yaml") .) (include (print $.Template.BasePath "/external-secret-app.yaml") .) | sha256sum }}
        {{- if .Values.branding.enabled }}
        checksum/config: {{ include (print $.Template.BasePath "/configmap-branding.yaml") . | sha256sum }}
        {{- end }}
@@ -8,7 +8,9 @@ metadata:
  labels:
    {{- include "sim.realtime.labels" . | nindent 4 }}
 spec:
-  {{- if not .Values.autoscaling.enabled }}
+  {{- /* The HPA owns replicas only when the realtime HPA actually renders —
+      with autoscaling.realtime.enabled=false, replicaCount stays in charge. */}}
+  {{- if not (and .Values.autoscaling.enabled .Values.autoscaling.realtime.enabled) }}
  replicas: {{ .Values.realtime.replicaCount }}
  {{- end }}
  selector:
@@ -17,7 +19,12 @@ spec:
  template:
    metadata:
      annotations:
-        checksum/secret: {{ include (print $.Template.BasePath "/secrets-app.yaml") . | sha256sum }}
+        {{- /* Hash the inline Secret AND the ExternalSecret manifest so spec
+          changes under externalSecrets.remoteRefs.app roll the pods (with ESO
+          enabled the inline Secret renders empty and would hash to a constant).
+          Rotated values inside the external store still can't be detected at
+          render time — ESO's own reconciliation handles those. */}}
+        checksum/secret: {{ printf "%s%s" (include (print $.Template.BasePath "/secrets-app.yaml") .) (include (print $.Template.BasePath "/external-secret-app.yaml") .) | sha256sum }}
        {{- with .Values.podAnnotations }}
        {{- toYaml . | nindent 8 }}
        {{- end }}
@@ -41,7 +41,7 @@ spec:
  {{- end }}
 {{- end }}

-{{- if and .Values.autoscaling.enabled .Values.realtime.enabled }}
+{{- if and .Values.autoscaling.enabled .Values.realtime.enabled .Values.autoscaling.realtime.enabled }}
 ---
 # HorizontalPodAutoscaler for realtime service
 apiVersion: autoscaling/v2
@@ -16,7 +16,7 @@ spec:
  - Ingress
  - Egress
  ingress:
-  # Allow ingress from realtime service
+  # Allow ingress from realtime pods (defensive — current traffic flows app -> realtime only)
  {{- if .Values.realtime.enabled }}
  - from:
    - podSelector:
@@ -237,7 +237,7 @@ spec:
    ports:
    - protocol: TCP
      port: {{ .Values.postgresql.service.targetPort }}
-  # Allow ingress from realtime service
+  # Allow ingress from realtime pods (defensive — current traffic flows app -> realtime only)
  {{- if .Values.realtime.enabled }}
  - from:
    - podSelector:
@@ -247,16 +247,6 @@ spec:
    - protocol: TCP
      port: {{ .Values.postgresql.service.targetPort }}
  {{- end }}
-  # Allow ingress from migrations job
-  {{- if .Values.migrations.enabled }}
-  - from:
-    - podSelector:
-        matchLabels:
-          {{- include "sim.migrations.labels" . | nindent 10 }}
-    ports:
-    - protocol: TCP
-      port: {{ .Values.postgresql.service.targetPort }}
-  {{- end }}
  egress:
  # Allow minimal egress (for health checks, etc.)
  - to: []
@@ -33,11 +33,14 @@ data:
        timeout: 1s
        send_batch_size: 1024
      memory_limiter:
+        check_interval: 1s
        limit_mib: 512
      
    exporters:
      {{- if .Values.telemetry.jaeger.enabled }}
-      jaeger:
+      {{- /* The dedicated jaeger exporter was removed from collector-contrib in
+           v0.86; Jaeger >=1.35 ingests OTLP natively, so export via OTLP. */}}
+      otlp/jaeger:
        endpoint: {{ .Values.telemetry.jaeger.endpoint }}
        tls:
          insecure: {{ not .Values.telemetry.jaeger.tls.enabled }}
@@ -74,7 +77,7 @@ data:
          exporters:
            - logging
            {{- if .Values.telemetry.jaeger.enabled }}
-            - jaeger
+            - otlp/jaeger
            {{- end }}
            {{- if .Values.telemetry.otlp.enabled }}
            - otlp
@@ -25,7 +25,7 @@ spec:
      type: RuntimeDefault
  containers:
    - name: probe
-      image: "{{ .Values.tests.image.repository }}:{{ .Values.tests.image.tag }}"
+      image: {{ include "sim.image" (dict "imageRoot" .Values.tests.image "global" .Values.global "chartAppVersion" .Chart.AppVersion) }}
      imagePullPolicy: {{ .Values.tests.image.pullPolicy }}
      securityContext:
        allowPrivilegeEscalation: false
@@ -4,6 +4,8 @@ templates:
  - deployment-realtime.yaml
  - secrets-app.yaml
  - services.yaml
+  # referenced by the deployments' checksum/secret annotation
+  - external-secret-app.yaml
 release:
  name: t
  namespace: sim
@@ -94,10 +94,6 @@
                "name": {
                  "type": "string",
                  "description": "Name of the existing Kubernetes secret"
-                },
-                "keys": {
-                  "type": "object",
-                  "description": "Key name mappings in the existing secret"
                }
              }
            }
@@ -577,10 +573,6 @@
                "name": {
                  "type": "string",
                  "description": "Name of the existing Kubernetes secret"
-                },
-                "passwordKey": {
-                  "type": "string",
-                  "description": "Key in the secret containing the password"
                }
              }
            }
@@ -634,10 +626,6 @@
            "name": {
              "type": "string",
              "description": "Name of the existing Kubernetes secret"
-            },
-            "passwordKey": {
-              "type": "string",
-              "description": "Key in the secret containing the password"
            }
          }
        }
@@ -61,17 +61,11 @@ app:
    existingSecret:
      # Set to true to use an existing secret instead of creating one from values
      enabled: false
-      # Name of the existing Kubernetes secret containing app credentials
+      # Name of the existing Kubernetes secret containing app credentials.
+      # The secret is consumed wholesale via envFrom, so its keys must use the
+      # standard names (BETTER_AUTH_SECRET, ENCRYPTION_KEY, INTERNAL_API_SECRET,
+      # CRON_SECRET, ...) — key remapping is not supported.
      name: ""
-      # Key mappings - specify the key names in your existing secret
-      # Only needed if your secret uses different key names than the defaults
-      keys:
-        BETTER_AUTH_SECRET: "BETTER_AUTH_SECRET"
-        ENCRYPTION_KEY: "ENCRYPTION_KEY"
-        INTERNAL_API_SECRET: "INTERNAL_API_SECRET"
-        CRON_SECRET: "CRON_SECRET"
-        API_ENCRYPTION_KEY: "API_ENCRYPTION_KEY"
-        REDIS_URL: "REDIS_URL"

  # Environment variables
  env:
@@ -542,9 +536,9 @@ realtime:
  extraVolumes: []
  extraVolumeMounts: []

-# Database migrations job configuration
+# Database migrations configuration (runs as an init container on the app pods)
 migrations:
-  # Enable/disable migrations job
+  # Enable/disable the migrations init container
  enabled: true
  
  # Image configuration
@@ -592,8 +586,9 @@ postgresql:
    # This enables integration with External Secrets Operator, HashiCorp Vault, etc.
    existingSecret:
      enabled: false
-      name: ""           # Name of existing Kubernetes secret
-      passwordKey: "POSTGRES_PASSWORD"  # Key in the secret containing the password
+      name: ""           # Name of existing Kubernetes secret. Must contain the
+                         # password under the key POSTGRES_PASSWORD — key
+                         # remapping is not supported.
  
  # Node selector for database pod scheduling (leave empty to allow scheduling on any node)
  nodeSelector: {}
@@ -711,8 +706,9 @@ externalDatabase:
  # This enables integration with External Secrets Operator, HashiCorp Vault, etc.
  existingSecret:
    enabled: false
-    name: ""           # Name of existing Kubernetes secret
-    passwordKey: "EXTERNAL_DB_PASSWORD"  # Key in the secret containing the password
+    name: ""           # Name of existing Kubernetes secret. Must contain the
+                       # password under the key EXTERNAL_DB_PASSWORD — key
+                       # remapping is not supported.

 # Ollama local AI models configuration
 ollama:
@@ -1016,6 +1012,12 @@ serviceAccount:
 # if you use customMetrics) installed in the cluster.
 autoscaling:
  enabled: false
+  # Also scale the realtime Deployment with the same HPA settings. Multi-replica
+  # realtime REQUIRES REDIS_URL in app.env (Socket.IO Redis adapter) — without
+  # Redis, cross-pod collaboration events are silently dropped. Set false to
+  # keep realtime pinned at realtime.replicaCount while the app still scales.
+  realtime:
+    enabled: true
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80
@@ -1435,7 +1437,9 @@ telemetry:
  # Jaeger tracing backend
  jaeger:
    enabled: false
-    endpoint: "http://jaeger-collector:14250"
+    # Jaeger's OTLP gRPC endpoint (Jaeger >=1.35 accepts OTLP natively; the
+    # legacy jaeger exporter was removed from the collector-contrib image)
+    endpoint: "jaeger-collector:4317"
    tls:
      enabled: false
  
@@ -1680,9 +1684,10 @@ copilot:
    secretKey: DATABASE_URL
    url: ""
  
-  # Migration job configuration
+  # Migration job configuration (Copilot migrations run as a Helm-hook Job,
+  # unlike the app migrations which run as an init container)
  migrations:
-    # Enable/disable migrations job
+    # Enable/disable the Copilot migrations Job
    enabled: true
    
    # Image configuration (same as server)