Commit Graph
34 Commits
Author SHA1 Message Date
Waleed c49751b32e perf(prefetch): stop calling our own API over the wire during server render (#6657)
* perf(prefetch): read the data layer instead of calling our own API over the wire

Four server-render prefetches went out over HTTP to our own routes. With
INTERNAL_API_BASE_URL unset in prod, getInternalApiBaseUrl() falls back to the
public base URL, so each was RSC -> public HTTPS -> load balancer -> back into
the app, awaited inside the render with a second round of auth.

- /home fetched the workflow folder list that the workspace layout had already
  fetched, under the identical query key. Since getQueryClient() builds a new
  client per call on the server, the two never deduped: same data, twice a
  request, once directly and once over the wire. Dropped; the layout's entry
  already hydrates it.
- /home cached raw route JSON under workspaceFilesKeys.list, while
  files/prefetch.ts seeds that same key from listWorkspaceFilesWithShares. The
  contract declares the date fields z.coerce.date(), so consumers hold Dates —
  a file record's type depended on which page the viewer landed on. Now reads
  the same function files/prefetch.ts does.
- tables and knowledge folder reads now call listFoldersForWorkspace, matching
  the sidebar prefetch.

These reads carry no authorization of their own, so each surface proves the
viewer through getWorkspaceHostContextForViewer first and caches nothing when
it fails, leaving the client fetch to reach the route for the real 403. Both it
and getSession are cache()d and already resolved by the layout, so the proof
costs no extra queries.

Left on the wire, deliberately: the tables and knowledge lists, whose cached
shape is the serialized wire shape, and pinned items and members, which have no
exported data-layer function.

* improvement(prefetch): skip the viewer proof when there is no session

Passing an empty-string userId ran a real permission query that could only
return null. Take an optional userId instead and skip straight to the
unauthorized path, matching how the home prefetch is called.

* perf(prefetch): finish removing self-HTTP prefetches and delete the legacy helper

Converts the last four server-render prefetches that called our own API over
HTTP, and deletes prefetch-internal-fetch.ts now that nothing imports it.

- knowledge bases: runs the route's own listInternalKnowledgeBases use case
  with a principal from the same internalSessionAuth policy the route declares,
  then projects through the same presenter and contract. Not a bypass of the
  application boundary — the same path, called in-process.
- tables: extracts the route's list projection into lib/table/wire.ts as
  toTableListItem, which the route and the prefetch now both call. This matters
  because listTablesContract's response schema is a passthrough z.custom, so a
  client fetch caches the route's JSON verbatim. Seeding listTables() directly
  would have put Date objects and the server-only metadata field under a key the
  hook never sees them on.
- pinned items: extracts the route's inline query into lib/pinned-items/queries.ts
  as listPinnedItemsForUser, which the route now calls too.
- workspace members: getWorkspaceMemberProfiles already existed; the prefetch
  calls it directly.

normalizeColumn moves from app/api/table/utils.ts to lib/table/wire.ts with ten
importers repointed. That also removes a pre-existing lib/* -> app/api/* boundary
violation in lib/table/import-runner.ts. No response shape changes: the v1/v2
edits are import-path moves only.

Every converted read proves the viewer first and caches nothing when that fails,
so an unauthorized viewer's client fetch still reaches the route for the real
403. Authorization equivalence was checked by unfolding both paths to
checkWorkspaceAccess rather than assumed.

* improvement(prefetch): collapse the duplicated folder prefetch and unify the call shape

- Extract prefetchResourceFolders. The same eight-line folder prefetch was
  written three times, varying only by resourceType, with the key, stale time
  and mapper kept in sync by hand.
- Adopting it removes the conditional spread from the tables and knowledge
  prefetches. Tables can now early-return, matching prefetchFilesBrowser:
  prefetchResourceListChrome already self-guards on the same cached host
  context, so a null context meant the function did nothing either way.
- Take userId as string | undefined everywhere and guard inside, so every
  prefetch module has one calling convention rather than two.
- Export toWireTimestamp and use it for the create-table response's own copy of
  the same idiom, and drop a cast that the extraction made dead: the parameter
  is already TableDefinition, whose schema is TableSchema.
- Read params and the session concurrently on the tables and knowledge pages,
  matching the files page, and drop TSDoc that restated each prefetch's own.

* fix(prefetch): keep the tables list on its route and cut the executor edge

Reading listTables from a page prefetch put the executable tool registry into
the Tables page server graph — ~4,700 modules, which check:tool-registry-boundary
rejects. lib/table/service reaches workflow-columns by several independent
paths (directly, and through jobs/service and rows/service), so severing one
edge is not enough; untangling that belongs in its own change.

- The tables list goes back through GET /api/table, with the reason recorded so
  the next person does not repeat the attempt. Folders and chrome on that page
  stay on the data layer.
- stripGroupDeps moves to its own leaf module. It is a pure projection over a
  WorkflowGroup, but living beside the group runtime meant every importer of
  lib/table/service paid for the executor to get it.

Net effect on the Tables page graph: 2,186 modules to 1,742.

* perf(prefetch): finish the migration, delete the legacy helper, ratchet page graphs

Answers the question the previous commit left open: the tables list did not have
to stay on HTTP. lib/table/service reached the executor through
jobs/service -> rows/service -> workflow-columns, for one symbol.
pendingDeleteMask is a delete-visibility SQL clause with no executor
involvement, so it moves to its own leaf and that chain is cut. The tables
prefetch now reads the data layer like every other one, and
prefetch-internal-fetch.ts is deleted: nothing in the app calls its own API over
HTTP during a server render any more.

stripGroupDeps likewise moves to a leaf rather than being re-exported through
workflow-columns, so its importers no longer pull the executor to get a pure
projection.

React Query mechanism fixes, all found by audit:
- settings/[section] fired two prefetches without awaiting them. Only a settled
  query is dehydrated, so those were shipped mid-flight; a rejection hydrated
  into an error state retryOnMount: false never retries, leaving the panel
  broken for the session. Awaited now, and the pending-dehydration opt-in is
  removed since nothing streams.
- The viewer profile was prefetched by both the layout and the settings page.
  Separate server QueryClients mean that was a real second read per request.
- prefetchSubscriptionData was dead, and hand-rolled an unannotated raw fetch.
- retry is scoped to the browser. Query core defaults it to 0 on the server;
  stating one value for both opted awaited prefetches into a retry backoff. The
  gcTime default is dropped entirely — 5 minutes is already the browser default,
  and setting it explicitly overrode the server's Infinity, leaving a live timer
  and payload per request.

check:tool-registry-boundary now also ratchets per-page module counts against a
committed baseline, attributing a regression to the import that caused it via a
dominator tree. It caught a +444 regression in this branch by hand; it would
have caught it in CI. Its import regex also missed bare side-effect imports,
so `import '@/tools/registry'` could have slipped past it entirely.

Prefetch guidance added to .claude/rules/sim-queries.md.

* fix(prefetch): correct the extracted module's db imports and stale rationale

Audit findings from the migration.

- pending-delete-mask imported its schema tables from @sim/db rather than
  @sim/db/schema, which the module it came from was careful to split. The
  global test mocks are bound per-entrypoint and only the schema mock exports
  tables, so every suite that reaches pendingDeleteMask would have failed on a
  missing mock export. Restored to the original convention, and the same split
  applied to the new pinned-items queries module before it grows a test.
- The settings prefetch and page justified awaiting with a mechanism this
  branch removed — pending queries being shipped with their promise. Only a
  settled query is dehydrated now, so an unawaited prefetch is dropped from the
  payload entirely. Same conclusion, correct reason, and no longer contradicting
  the rule this branch added.
- Removed the doc block left orphaned above validateSchema when stripGroupDeps
  moved out of workflow-columns.

Skill projections regenerated after trimming the boundary skill.

* chore(table): drop a section separator comment

Separators like these are non-TSDoc decoration that CLAUDE.md already rules
out. This is the only one in a file this branch touches; the rest of the repo
is swept separately.

* chore: remove section separator comments

CLAUDE.md already rules these out ("No ==== separators. No non-TSDoc
comments"), but 546 of them had accumulated across 48 files. They decorate
rather than explain, and they drift: a separator says "Validation" while the
code beneath it moved elsewhere, as one in workflow-columns already had.

Pure deletion — no source line was touched, and lines inside template
literals were skipped so nothing in a generated string changed.
2026-08-13 00:12:45 -07:00
Vikhyath Mondreti 5cf1f9be84 improvement(provenance): cleanup secrets boundary (#6374)
* fix(secrets): preserve raw outputs with durable provenance

* improvement(provenance): cleanup boundary

* fix copy resources

* fix fork copies to work with provenance

* address comments

* fix
2026-08-07 17:12:17 -07:00
Theodore Li e369997eac fix(db): encode Date binds in raw sql, document the fetch_types contract (#6040)
* fix(db): encode Date binds in raw sql, document the fetch_types contract

* fix(testing): drop template placeholders from mock error message for biome
2026-07-29 13:06:34 -04:00
Waleed 513292f17b feat(sso): DNS domain verification gating org SSO registration (#5909)
* feat(sso): DNS domain verification gating org SSO registration

Add org-scoped domain ownership verification (DNS TXT challenge) as the
security precondition for configuring SSO. Closes the first-come domain-claim
vuln where any org could wire another company's domain to its own IdP.

- New sso_domain table + migration 0266; existing org SSO domains are
  grandfathered as verified so live tenants are unaffected
- Verified-domains settings UI (enterprise-gated) with add/verify/remove
- Register route now requires a verified domain for org-scoped registration;
  personal SSO and already-grandfathered domains are unaffected
- Self-host register script writes the verified sso_domain row directly, so
  script-driven registration stays backwards compatible

* fix(sso): harden domain verification against concurrency + fix CI lint

Addresses review findings on state invariants under concurrent/failed writes:
- Add unique index on (organization_id, domain) so concurrent claims can't
  create duplicate pending rows; POST re-reads and stays idempotent on conflict
- Verify flips the row only if it's still the exact pending challenge checked
  (guards deletion/token-rotation mid-DNS-lookup) and maps the partial unique
  index violation to 409 instead of an unhandled 500
- Wrap the self-host script's provider write + verified-domain upsert in a
  transaction so a failed ownership write can't leave a provider committed
- Format 0266 snapshot/journal with biome (fixes @sim/db lint:check)

* fix(sso): re-check domain verification before provider write (TOCTOU)

The register gate checked the verified sso_domain row only at handler entry,
then ran OIDC discovery before writing the provider. A verified row removed
during that window could still complete registration. Extract the check into a
closure and call it both as an entry fast-fail and authoritatively right before
registerSSOProvider, alongside the existing domain-conflict re-check.

* fix(sso): stop rotating verification token on idempotent re-add

Re-adding a pending domain rotated its verification token, which invalidated a
TXT record the admin may have already published and — under two concurrent
re-adds — could return a token the racing write had already superseded, so the
admin's DNS record would never verify. Return the existing row unchanged
instead; the pending token is always shown in the UI, so it is never lost.

* fix(sso): close register TOCTOU with compensating delete + harden edges

Audit-driven hardening:
- Close the residual register TOCTOU: registerSSOProvider is create-only (throws
  if the providerId exists), so a compensating delete after the write is provably
  safe — it can only remove the just-created row. If verification was revoked
  during the write, roll the provider back and 403.
- Verify is now idempotent under concurrency: a same-org row already flipped to
  verified by a racing request returns 200, not a confusing 409.
- Grandfather backfill + self-host script now match normalizeSSODomain's dominant
  transforms (lower + trim + strip leading wildcard) so a non-canonical legacy
  domain can't miss the runtime gate's lookup. Prod backfill result is unchanged.
- Cleanup: drop dead default export, align card radius to sibling convention.

* fix(sso): redact domain tokens from non-admins + fix script stale-update

Round-5 review findings:
- GET /domains redacted the pending TXT verification token (a management
  secret) to any org member. Now only owner/admins read it; members see the
  list and status without it. Non-Enterprise orgs get an empty list (entitlement
  flag only), never the domains/tokens.
- Self-host script decided update-vs-insert from a read taken OUTSIDE the
  transaction; a provider deleted mid-flight made the UPDATE match zero rows
  silently while the verified-domain upsert still committed (orphaned domain).
  The decision now happens inside the transaction from the UPDATE's row count.

* docs(sso): drop unshipped enforce-SSO / auto-join copy from verified domains

Verified domains currently only gate SSO configuration. Remove the
forward-looking references to enforcing SSO and auto-joining members (deferred
to a later release) from the docs, settings copy, nav description, and schema
comment so we don't promise unshipped features.

* fix(sso): guard rollback to new providers only + Enterprise-gate domain removal

Round-6 review findings:
- The compensating provider rollback now only fires when the provider did not
  exist before this request (providerExistedBefore). registerSSOProvider is
  create-only today so reaching the rollback already implies a fresh create, but
  this makes the safety local and future-proof: if Better Auth ever allowed
  updating an existing provider, a revoked-verification rollback must not delete
  that pre-existing row.
- DELETE /domains now requires an Enterprise plan like add/list/verify, so all
  domain mutations share one entitlement (the UI already hides removal from
  non-Enterprise orgs). Adds a delete-route test.

* fix(sso): roll back the SSO provider by row id, not logical keys

The compensating rollback deleted by (providerId, orgId). providerId is unique,
so if this request's row were deleted and recreated by a concurrent registration
in the narrow window before the rollback, the logical-key delete would remove
that other request's provider. Delete by the primary-key id registerSSOProvider
returns instead, so only the exact row this request created is ever removed.

* chore(sso): final-review polish — trim script read, unify copy, doc migration edge

Cosmetic cleanup from a final 4-track adversarial review (no bugs found in the
new logic):
- Self-host script: narrow the pre-transaction existence read to select({ id })
  instead of SELECT * (it only feeds a log line now).
- Unify invalid-domain copy ("for example acme.com") and the verified-elsewhere
  409 wording ("is already verified by another organization") across routes.
- p-3 shorthand on the domain row card.
- Document the migration's rare two-orgs-share-a-domain grandfather behavior
  (login unaffected; validated no such duplicates in prod).

* fix(sso): apply attribute mapping + make SSO edit work; drop dead guard

Two pre-existing SSO bugs the final review surfaced (prod has one SSO org, RVW,
script-registered with the default mapping, so neither change affects it):

- Attribute mapping was passed at the top level of the register payload, which
  Better Auth ignores — it reads oidcConfig.mapping / samlConfig.mapping. Nest it
  so custom mappings actually apply. (Default mapping is unchanged, so existing
  logins are unaffected.)
- Editing an SSO provider was broken: registerSSOProvider is create-only and
  threw on the existing providerId → generic 500. Route now detects a provider
  the caller already owns and updates it via Better Auth's updateSSOProvider, and
  surfaces Better Auth's own error status/message instead of a blanket 500.

Also drops the now-unnecessary providerExistedBefore guard (the rollback deletes
by the created row's primary-key id and register is create-only) and the earlier
final-review polish (script read, unified copy, migration edge note).

Smoke-test SSO login + edit on staging before merge (auth-path change).

* fix(sso): require null org on personal-mode provider lookups (gate bypass)

The personal branch of both provider-ownership lookups keyed on
(providerId, userId) without requiring organizationId IS NULL. Because org
providers store userId = their creator and providerId is globally unique, an org
admin could send a personal-mode request (no orgId) — which skips the membership
check and the domain-verification gate — yet still match, and then via the new
update path move, their org's provider to an unverified domain. Add
isNull(organizationId) to the personal branch of both clauses so it can only
match a genuinely personal provider, matching the route's own isOwnedByCaller.

Found by an adversarial review of the update path added in 394bda9f7.

* fix(sso): script updates the observed provider by id, not providerId

Inside the registration transaction the script updated WHERE providerId — the
logical key. If the observed provider was deregistered and a replacement created
with the same providerId before the transaction ran, that update would clobber
the replacement's config and ownership. Update the specific observed row by its
primary-key id instead; if it's gone we insert, which fails cleanly on the
providerId unique constraint rather than overwriting the replacement.

* fix(sso): script upserts provider via delete-then-insert (no unique constraint)

sso_provider.provider_id is a plain (non-unique) index and prod holds legitimate
duplicates, so the previous "update by id, else insert" could create a duplicate
provider when the observed row was deregistered and replaced before the
transaction — the fallback insert would succeed. Delete every row for the
providerId then insert exactly one, inside the transaction, so the providerId
ends up as exactly this config atomically. Linked accounts key on the providerId
string (not the row id), so existing logins are unaffected.

* fix(sso): guard compensating-delete row id so rollback can't silently no-op

* chore(sso): regenerate migration as 0268 after merging staging

Staging landed migrations 0266/0267, colliding with our 0266. Removed our
migration, merged staging, and regenerated cleanly with drizzle-kit as
0268_sso_domain_verification (identical sso_domain table + indexes), then
re-appended the grandfather backfill. api-validation baseline reconciled to 973
(staging 970 + our 3 domain routes). Also make the register-route test's
registerSSOProvider mock return an id so the guarded compensating delete runs.

* refactor(sso): share normalizeSSODomain via @sim/utils so script matches gate

The self-host script canonicalized SSO domains with a minimal inline transform
(lower+trim+wildcard) that diverged from the app's full normalizeSSODomain
(protocol, port, path, trailing dot, email local part) — equivalent spellings
could store a different ownership key than the runtime gate looks up. Move
normalizeSSODomain into @sim/utils/sso-domain (a pure function) so the register
route, the domain-claim route, and the script all use the identical canonicalizer.
The script now skips the verified-domain record when SSO_DOMAIN isn't a valid
registrable domain instead of storing a malformed key.
2026-07-24 00:34:16 -07:00
Theodore Li de934b9979 feat(script): migrate agent/router/evaluator LLM keys to BYOK by model prefix (#5607)
* feat(byok): migrate agent/router/evaluator LLM keys to BYOK by model prefix

* fix(script): address review — cover translate/guardrails, trim provider ids, reject mixed mapping, add key-prefix guard

* fix(script): include pi block in LLM-family set, trim key-prefix value

* fix(script): add router_v2 to LLM-family set, canonicalize provider ids to lowercase

* fix(script): resolve overlapping model prefixes deterministically (longest match wins)
2026-07-13 18:28:12 -04:00
Vikhyath Mondreti ef7c8e24b2 feat(platform): settings permissions, admin, billing attribution (#5545)
* fix(invites): preserve active organization for external access

Keep organization activation server-owned so failed membership checks cannot clear a valid session context.

* feat(admin, billing, settings): cleanup settings visibility, billing actor resolution, new admin routes

* address comments

* chore(db): reset pending migrations before staging merge

Remove locally generated migrations so they can be regenerated against the latest staging schema without preserving stale snapshots or numbering.

* regen migrations

* address comments

* chore(db): reset generated migrations before staging merge

Remove this branch's generated migrations so they can be regenerated against the latest staging schema with fresh numbering.

* upgrade global work

* fix lint

* address comments

* legacy callbacks correctness

* address comments

* update

* guardrail attribution
2026-07-13 00:53:49 -07:00
Theodore Li 12fb4a9db1 feat(db): auto-apply tracked script data migrations in db:migrate (#5497)
* feat(db): auto-apply tracked script data migrations in db:migrate

* fix(db): reset session lock_timeout before script migrations, guard journal insert

* improvement(db): re-verify advisory-lock session before script migrations
2026-07-07 20:55:42 -04:00
Waleed e1b8200660 fix(sso): support skipping the OIDC UserInfo endpoint at registration (#5386)
* fix(sso): support skipping the OIDC UserInfo endpoint at registration

* fix(sso): cap OIDC discovery fetch at 10s to avoid stalling registration

* test(sso): default-mock discovery fetch so intent is explicit

* fix(sso): prefer client_secret_post and surface discovery failure reasons

* fix(sso): always resolve token auth method and skip SSRF-checking a discarded userInfoEndpoint
2026-07-03 12:09:22 -07:00
Waleed 2fa3dd65bc fix(db): retry the migration connection on transient slot exhaustion (#5226)
* ci(migrations): skip db:migrate on merges that change no migration files

Every push to main/staging ran db:migrate against the production/staging
database even when the merge changed no schema, so a no-op migration would dial
the DB and fail whenever it was at its connection limit (53300, slots reserved
for SUPERUSER) — red-X'ing UI-only merges.

Add a detect-migrations job (dorny/paths-filter on packages/db/migrations/**)
and pass the result into the reusable migrations workflow, which now skips the
apply step when no migration files changed. The migrate job still runs so
downstream build/deploy jobs that need it are never skipped, and the flag
defaults to 'true' so manual dispatch and any unknown value always apply
migrations — the gate only ever skips a provably-empty change.

* fix(db): retry the migration connection on transient slot exhaustion

The migration opens its session on the first query (the advisory-lock
acquire). When the deploy database briefly exhausts every non-superuser
connection slot at peak, that connect fails with 53300 ("remaining connection
slots are reserved for roles with the SUPERUSER attribute") and the whole
deploy's migrate step errors out — even when the spike clears within seconds.

Add a bounded connectWithRetry() before acquiring the lock that retries 53300,
the 08xxx connection_exception class, and the driver's transport errors with
backoff (10 attempts, ~90s ceiling). Non-transient errors (auth, bad config)
still fail fast. The migration is a single short-lived session, so waiting out
a transient spike is far safer than failing the deploy.

* ci: drop the migration paths-filter gate (out of scope)

Revert the detect-migrations gate carried over from the closed CI PR; we are
fixing the connection failure at its source (migrate.ts connection retry)
rather than gating db:migrate, which the reviewers correctly noted could leave
a previously-merged migration unapplied after a failed deploy.
2026-06-26 12:21:11 -07:00
Theodore Li a68d38ae75 feat(db): attribute Postgres connections by runtime via application_name (#5211)
* feat(db): attribute Postgres connections by runtime via application_name

* improvement(db): label migration-runner connection sim-migrate; trim DB_APP_NAME comment

* fix(db): label realtime's shared @sim/db connections sim-realtime too

The realtime process uses both its own socketDb pool and the shared @sim/db
client (handlers, preflight, permissions). Only socketDb was labeled, so the
shared client defaulted to sim-app, mislabeling much of realtime's DB traffic.
Set DB_APP_NAME=sim-realtime at the process level (bootstrap before the dynamic
@/index import for prod; dev/start scripts for local) so both clients report it.
2026-06-25 17:11:23 -04:00
Waleed 9efb7b45f4 feat(byok): support multiple keys per provider with round-robin rotation (#4963)
* feat(byok): support multiple keys per provider with round-robin rotation

* fix(byok): address review feedback — serialize cap check, defer encryption, explicit delete guard

* fix(byok): drop ON CONFLICT on removed unique index in legacy key migration script

* improvement(byok): guard double-submit, Enter-to-save on key field, cap hint in manage modal
2026-06-11 10:00:58 -07:00
Waleed 284edf0b85 improvement(db): opt-in read-replica client + migration runner hardening (#4955)
* improvement(db): add opt-in read-replica client and harden migration runner

* fix(db): detect wrapped lock-timeout errors, jittered retries, direct migration DSN support

* fix(db): audit fixes — unparameterized SET, primary reads for authz scoping, shared error helper

* fix(db): pin migration session — disable max_lifetime recycling, guard backend pid across retries

* fix(db): gate pid session guard to direct migration connections (pooler pids legitimately vary)

* improvement(billing): route display-only usage aggregations to the read replica via executor threading

* chore(db): drop call-site justification comments

* chore(db): tighten doc comments

* ci(migrations): map optional direct-connection DSN secrets

* fix(data-drains): add stability window to time cursors so late-visible rows are never skipped

* fix(billing): resolve pagination cursor on primary; replica for member ledger display

* fix(billing): usage-limits returns cost and limit from one computation
2026-06-10 18:33:58 -07:00
Waleed b2a485e164 fix(db): serialize concurrent migrations with a Postgres advisory lock (#4939)
* fix(db): serialize concurrent migrations with a Postgres advisory lock

Deployments start N app replicas at once, each with a migration sidecar.
drizzle migrate() has no cross-process lock, so all N read
__drizzle_migrations, all see the same migration pending, and all apply it
concurrently — one wins, the losers run the same DDL against already-mutated
state and exit 1 (e.g. DROP TABLE "form" -> table does not exist /
TaskFailedToStart). Wrap migrate() in a session-level pg_advisory_lock so
runners serialize: the winner migrates, the losers block, then re-read and
find nothing pending. Session locks auto-release on disconnect, so a crashed
runner never wedges the lock.

* fix(db): guard pg_advisory_unlock so it cannot mask a successful migration

If the explicit unlock throws (e.g. connection drops in the window after
migrate() commits), the exception bubbled to the outer catch and exited 1 —
falsely reporting a failed migration to the deploy orchestrator. The session
lock auto-releases on disconnect anyway, so swallow and log instead.

* refactor(db): move unlock-guard rationale to TSDoc helper
2026-06-09 21:54:27 -07:00
Vikhyath MondretiandCursor 49673055be improvement(logs): object storage backed tracespans (#4787)
* improvement(logs): obj storage backed tracespans

* fix storage write context

* fix tests

* address comments

* address comments

* chore(db): remove migration 0219 to regenerate after staging merge

Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the
trace-spans/cost schema migration can be regenerated on top of the latest
staging migration chain (avoids a number collision with staging's migrations).

Co-authored-by: Cursor <cursoragent@cursor.com>

* improvement(billing): accurate per-member usage via shared ledger helper

Per-member/per-user usage in the org-member routes now adds the usage_log
ledger to the currentPeriodCost baseline (which is no longer incremented),
via a shared getOrgMemberLedgerByUser helper to avoid repeating the
subscription→period→ledger lookup across the admin and member-facing routes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* regen migrations

* update migration

* address comments

* more code cleanup

* incorrect type cast

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-29 13:47:32 -07:00
Theodore Li 786c6f0607 fix(db): disable statement_timeout for migrations (#4714)
* fix(db): disable statement_timeout for migrations

* fix(ci): route migration workflow through guarded migrate.ts
2026-05-22 14:23:06 -04:00
Waleed 8d7bbbc670 chore(utils): migrate to shared random/ID utilities and add enforcement linting (#4623)
* chore(utils): migrate to shared random/ID utilities and add enforcement linting

- Replace all Math.random(), crypto.randomUUID(), crypto.randomBytes(), nanoid, and uuid usages with shared @sim/utils/random and @sim/utils/id helpers across 72 files
- Add new @sim/utils exports: deepClone, omit, filterUndefined (object), truncate (string), backoffWithJitter, parseRetryAfter (retry), getErrorMessage (errors)
- Sweep all getErrorMessage, sleep, deepClone callsites across 500+ files to use shared utilities
- Add Biome noRestrictedImports rule to catch nanoid, uuid, and crypto named imports at lint time
- Add scripts/check-utils-enforcement.ts to catch Math.random and crypto.* global property access
- Add check:utils script to package.json

* chore(utils): replace deepClone wrapper with structuredClone built-in

deepClone() was a one-line wrapper around structuredClone(), which is
universally available in Node 17+ and all modern browsers. Removing the
abstraction reduces indirection and means contributors don't need to
learn a project-specific name for a well-known built-in.

- Remove deepClone from packages/utils/src/object.ts and index.ts
- Replace all 17 call sites with structuredClone() directly
- Update check:utils script suggestion text
- Update CLAUDE.md and global.md docs

* fix(utils): add missing biome noRestrictedImports rule and correct truncate docs

- Add noRestrictedImports to biome.json under style — bans nanoid and uuid
  package imports at lint time (crypto.randomUUID/randomBytes are caught by
  the check:utils grep script which handles global property access)
- Correct truncate() TSDoc and parameter name: sliceLength makes it clear
  that total output length is sliceLength + suffix.length, matching the
  behavior all callers were already written to expect

* fix(utils): add missing getErrorMessage imports at 4 call sites

The sweep agents added getErrorMessage calls without the corresponding
import in 4 files, causing test failures. Added the missing imports.

* fix(utils): fix build errors from getErrorMessage sweep and retry.ts Turbopack issue

- Fix retry.ts cross-file import: Turbopack cannot resolve './random.js' for
  internal package imports; inline the jitter crypto call directly
- Add missing getErrorMessage imports to 32 files where the sweep added calls
  without the corresponding import (caught by type-check and test runs)
- Remove accidental getErrorMessage import from crowdstrike/query/route.ts
  which has its own domain-specific getErrorMessage for parsing CrowdStrike's
  JSON error format
- Fix use-sub-block-value.ts type error from structuredClone narrowing:
  add 'as T' cast at emitValue callsite (safe — valueCopy is always a
  structural copy of newValue)

* fix(tools): use toError in crowdstrike catch block instead of local getErrorMessage

The catch block was calling the local getErrorMessage function which
parses CrowdStrike API JSON responses, not JavaScript Error objects.
Use toError(error).message to correctly extract the message from a
caught value in this context.
2026-05-15 17:31:27 -07:00
Waleed 0c25fc4ee1 fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies (#4369)
* fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies

- auth client now uses browser origin first, falling back to NEXT_PUBLIC_APP_URL
- socket client falls back to page origin when served from non-localhost (assumes /socket.io is proxied)
- add TRUSTED_ORIGINS env var to extend Better Auth trustedOrigins (apex+www, alias hostnames)
- warn at startup when NEXT_PUBLIC_APP_URL is localhost in production
- preprocess empty NEXT_PUBLIC_SOCKET_URL so docker-compose ${VAR:-} works
- migrate remaining uuid/nanoid/randomUUID usages to @sim/utils generateId/generateShortId
- extend generateShortId with optional alphabet param (rejection sampling)
- document TRUSTED_ORIGINS in .env.example, docker-compose.prod.yml, and helm values.yaml

Fixes simstudioai/sim#1243

* fix(auth): address PR review comments

* chore(env): drop unnecessary NEXT_PUBLIC_SOCKET_URL preprocess (skipValidation is true)

* fix(docker): include @sim/utils in migrations image

Migration scripts now import generateId from @sim/utils/id; without copying packages/utils into the image, bun install fails to resolve the workspace dep at build time and the import fails at runtime.

* fix(helm): remove unused NEXT_PUBLIC_SOCKET_URL from realtime sections

The realtime service never reads NEXT_PUBLIC_SOCKET_URL — its env schema
only includes BETTER_AUTH_URL, NEXT_PUBLIC_APP_URL, ALLOWED_ORIGINS,
BETTER_AUTH_SECRET, INTERNAL_API_SECRET, DATABASE_URL, and REDIS_URL.
Remove the dead config from all helm values files and the values schema.

* fix(helm): allow empty NEXT_PUBLIC_SOCKET_URL in values schema

The default in values.yaml is now "" (empty string), which falls back to
the page origin at runtime. The schema previously required a valid URI,
which would reject the default. Mirror the INTERNAL_API_BASE_URL pattern
using anyOf with const "". Also add TRUSTED_ORIGINS to the schema.

* docs(self-hosting): mark NEXT_PUBLIC_SOCKET_URL as optional

The page-origin fallback in getSocketUrl() means self-hosters no longer
need to set NEXT_PUBLIC_SOCKET_URL when realtime is on the same origin
as the app. Update docs to reflect this:

- Remove NEXT_PUBLIC_SOCKET_URL from .env scaffolding examples in
  docker.mdx, platforms.mdx, environment-variables.mdx
- Mark the variable as Optional in the env vars table with the new
  default behavior described
- Update troubleshooting to point at reverse-proxy /socket.io routing
  rather than the env var
- Flip dev docker-compose defaults (local, ollama, devcontainer) from
  http://localhost:3002 to empty for consistency with prod.yml; the
  in-code localhost fallback handles the dev case identically

Applied across all 6 documentation languages (en/fr/de/ja/es/zh).

* chore: untrack and ignore .claude/scheduled_tasks.lock
2026-04-30 19:26:19 -07:00
Theodore Li 8ce56fe1f2 fix(auth): add api key auth via sha256 hash lookup (#4266)
* fix(auth): add api key auth via sha256 hash lookup

* Remove promise all logic

* Restore feature flag

* fix feature flag

* Combine auth and hash gate
2026-04-22 18:30:37 -04:00
Vikhyath Mondreti 41a1b50ace improvement(migrations): log better errors (#4260) 2026-04-21 22:06:05 -07:00
Waleed 0cd14f4ac9 improvement(sso): fix provider lookup, migrate UI to emcn, add enterprise SSO docs (#4238)
* improvement(sso): fix provider lookup, migrate UI to emcn, add enterprise SSO docs

* fix(sso): add org membership guard on providers route, fix idpMetadata round-trip

* fix(sso): add org membership guard on register route, fix SP entityID, remove fullError leak

* fix(sso): fix SAML script callbackUrl and SP entityID to use app base URL

* fix(sso): correct SAML callback URL path in script header comment

* fix(sso): restrict SSO provider read/write to org owners and admins

* docs(sso): restructure page, fix provider guide accuracy, add external doc links

* fix(sso): correct SAML callback path and generate idpMetadata from cert+entryPoint

* fix(sso): always require NEXT_PUBLIC_APP_URL for SAML SP metadata entityID

* fix(sso): scope provider query to org only when organizationId is provided

* fix(sso): escape XML special chars in script idpMetadata generation

* fix(sso): final audit corrections — saml mapping, xml escaping, self-hosted org guard

* fix(sso): redact oidc client secret in providers response, add self-hosted org admin guard

* fix(sso): scope redacted-secret lookup to caller's org or userId

* fix(sso): null out oidcConfig on parse failure to prevent unredacted secret leak

* fix(sso): use issuer as entityID in auto-generated idp metadata xml
2026-04-20 16:45:37 -07:00
mini f46886e6cf fix(sso): default tokenEndpointAuthentication to client_secret_post (#3627)
* fix(sso): default tokenEndpointAuthentication to client_secret_post

better-auth's SSO plugin does not URL-encode credentials before Base64
encoding in client_secret_basic mode (RFC 6749 §2.3.1). When the client
secret contains special characters (+, =, /), OIDC providers decode them
incorrectly, causing invalid_client errors.

Default to client_secret_post when tokenEndpointAuthentication is not
explicitly set to avoid this upstream encoding issue.

Fixes #3626

* fix(sso): use nullish coalescing and add env var for tokenEndpointAuthentication

- Use ?? instead of || for semantic correctness
- Add SSO_OIDC_TOKEN_ENDPOINT_AUTH env var so users can explicitly
  set client_secret_basic when their provider requires it

* docs(sso): add SSO_OIDC_TOKEN_ENDPOINT_AUTH to script usage comment

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>

* fix(sso): validate SSO_OIDC_TOKEN_ENDPOINT_AUTH env var value

Replace unsafe `as` type cast with runtime validation to ensure only
'client_secret_post' or 'client_secret_basic' are accepted. Invalid
values (typos, empty strings) now fall back to undefined, letting the
downstream ?? fallback apply correctly.

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>

---------

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>
2026-04-07 08:46:18 -07:00
abhinavDhulipala 7971a64e63 fix(setup): db migrate hard fail and correct ini env (#3946) 2026-04-04 16:22:19 -07:00
42fb434354 fix(encryption): specify authTagLength on all AES-GCM cipher/decipher calls (#3883)
* fix: specify authTagLength in AES-GCM decipheriv calls

Fixes missing authTagLength parameter in createDecipheriv calls using
AES-256-GCM mode. Without explicit tag length specification, the
application may be tricked into accepting shorter authentication tags,
potentially allowing ciphertext spoofing.

CWE-310: Cryptographic Issues (gcm-no-tag-length)

* fix: specify authTagLength on createCipheriv calls for AES-GCM consistency

Complements #3881 by adding explicit authTagLength: 16 to the encrypt
side as well, ensuring both cipher and decipher specify the tag length.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: clean up crypto modules

- Fix error: any → error: unknown with proper type guard in encryption.ts
- Eliminate duplicate iv.toString('hex') calls in both encrypt functions
- Remove redundant string split in decryptApiKey (was splitting twice)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* new turborepo version

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Lakee Sivaraya <71339072+lakeesiv@users.noreply.github.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com>
Co-authored-by: NLmejiro <kuroda.k1021@gmail.com>
2026-04-01 10:46:58 -07:00
Theodore LiandTheodore Li 5ba3118495 feat(byok-migration) byok migration script (#3584)
* Add byok migration script

* Fix lint

* Add skipping if byok already provided

* Fix lint

---------

Co-authored-by: Theodore Li <theo@sim.ai>
2026-03-14 16:11:21 -04:00
e24c824c9a feat(tables): added tables (#2867)
* updates

* required

* trashy table viewer

* updates

* updates

* filtering ui

* updates

* updates

* updates

* one input mode

* format

* fix lints

* improved errors

* updates

* updates

* chages

* doc strings

* breaking down file

* update comments with ai

* updates

* comments

* changes

* revert

* updates

* dedupe

* updates

* updates

* updates

* refactoring

* renames & refactors

* refactoring

* updates

* undo

* update db

* wand

* updates

* fix comments

* fixes

* simplify comments

* u[dates

* renames

* better comments

* validation

* updates

* updates

* updates

* fix sorting

* fix appearnce

* updating prompt to make it user sort

* rm

* updates

* rename

* comments

* clean comments

* simplicifcaiton

* updates

* updates

* refactor

* reduced type confusion

* undo

* rename

* undo changes

* undo

* simplify

* updates

* updates

* revert

* updates

* db updates

* type fix

* fix

* fix error handling

* updates

* docs

* docs

* updates

* rename

* dedupe

* revert

* uncook

* updates

* fix

* fix

* fix

* fix

* prepare merge

* readd migrations

* add back missed code

* migrate enrichment logic to general abstraction

* address bugbot concerns

* adhere to size limits for tables

* remove conflicting migration

* add back migrations

* fix tables auth

* fix permissive auth

* fix lint

* reran migrations

* migrate to use tanstack query for all server state

* update table-selector

* update names

* added tables to permission groups, updated subblock types

---------

Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: waleed <walif6@gmail.com>
2026-02-19 13:11:35 -08:00
Waleed ac991d4b54 fix(sso): removed provider specific OIDC logic from SSO registration & deregistration scripts (#2896)
* fix(sso): updated registration & deregistration script for explicit support for Entra ID

* cleanup

* ack PR comment

* ack PR comment

* tested edge cases, ack'd PR comments

* remove trailing slash
2026-01-19 19:23:50 -08:00
Waleed 9502227fd4 fix(sso): add missing deps to db container for running script (#2746) 2026-01-09 09:42:13 -08:00
Waleed 10692b5e5a fix(db): remove overly complex db connection logic (#1538) 2025-10-02 19:54:32 -07:00
Waleed fa9c97816b fix(db): add more options for SSL connection, add envvar for base64 db cert (#1533) 2025-10-02 15:53:45 -07:00
Waleed ace83ebcae feat(cmdk): added knowledgebases to the cmdk modal (#1530) 2025-10-01 21:21:42 -07:00
Waleed 980a6d8347 improvement(db): enforce SSL everywhere where a DB connection is established (#1522)
* improvement(db): enforce SSL everywhere where a DB connection is established

* remove extraneous comment
2025-10-01 19:09:08 -07:00
Waleedandwaleed 6e63eafb79 improvement(db): remove vercel, remove railway, remove crons, improve DB connection config (#1519)
* improvement(db): remove vercel, remove railway, remove crons, improve DB connection config

* remove NEXT_PUBLIC_VERCEL_URL

* remove db url fallbacks

* remove railway & more vercel stuff

---------

Co-authored-by: waleed <waleed>
2025-10-01 16:37:13 -07:00
bf1c178282 feat(sso): add support for login with SAML/SSO (#1489)
* feat(sso): added login with SAML/SSO

* restore env

* fixed login styling

* upgrade deps, update UI

* more styling improvements

* reran migrations, tested with script

* improvement(auth): created SSO page

* improvement(auth): remove email option for SSO if not enabled

* cleanup

* cleaned up, added documentation for SSO/SAML config + tested registering either one with script and UI form

* cleanup

* ack PR comments

* move sso known providers to consts

---------

Co-authored-by: waleed <waleed>
Co-authored-by: Emir Karabeg <emirkarabeg@berkeley.edu>
2025-09-29 20:05:26 -07:00
Vikhyath Mondreti 9de7a00373 improvement(code-structure): move db into separate package (#1364)
* improvement(code-structure): move db into separate package

* make db separate package

* remake bun lock

* update imports to not maintain two separate ones

* fix CI for tests by adding dummy url

* vercel build fix attempt

* update bun lock

* regenerate bun lock

* fix mocks

* remove db commands from apps/sim package json
2025-09-17 15:41:13 -07:00