Commit Graph
21 Commits
Author SHA1 Message Date
Vikhyath MondretiandCursor 49673055be improvement(logs): object storage backed tracespans (#4787)
* improvement(logs): obj storage backed tracespans

* fix storage write context

* fix tests

* address comments

* address comments

* chore(db): remove migration 0219 to regenerate after staging merge

Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the
trace-spans/cost schema migration can be regenerated on top of the latest
staging migration chain (avoids a number collision with staging's migrations).

Co-authored-by: Cursor <cursoragent@cursor.com>

* improvement(billing): accurate per-member usage via shared ledger helper

Per-member/per-user usage in the org-member routes now adds the usage_log
ledger to the currentPeriodCost baseline (which is no longer incremented),
via a shared getOrgMemberLedgerByUser helper to avoid repeating the
subscription→period→ledger lookup across the admin and member-facing routes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* regen migrations

* update migration

* address comments

* more code cleanup

* incorrect type cast

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-29 13:47:32 -07:00
Theodore Li 786c6f0607 fix(db): disable statement_timeout for migrations (#4714)
* fix(db): disable statement_timeout for migrations

* fix(ci): route migration workflow through guarded migrate.ts
2026-05-22 14:23:06 -04:00
Waleed 8d7bbbc670 chore(utils): migrate to shared random/ID utilities and add enforcement linting (#4623)
* chore(utils): migrate to shared random/ID utilities and add enforcement linting

- Replace all Math.random(), crypto.randomUUID(), crypto.randomBytes(), nanoid, and uuid usages with shared @sim/utils/random and @sim/utils/id helpers across 72 files
- Add new @sim/utils exports: deepClone, omit, filterUndefined (object), truncate (string), backoffWithJitter, parseRetryAfter (retry), getErrorMessage (errors)
- Sweep all getErrorMessage, sleep, deepClone callsites across 500+ files to use shared utilities
- Add Biome noRestrictedImports rule to catch nanoid, uuid, and crypto named imports at lint time
- Add scripts/check-utils-enforcement.ts to catch Math.random and crypto.* global property access
- Add check:utils script to package.json

* chore(utils): replace deepClone wrapper with structuredClone built-in

deepClone() was a one-line wrapper around structuredClone(), which is
universally available in Node 17+ and all modern browsers. Removing the
abstraction reduces indirection and means contributors don't need to
learn a project-specific name for a well-known built-in.

- Remove deepClone from packages/utils/src/object.ts and index.ts
- Replace all 17 call sites with structuredClone() directly
- Update check:utils script suggestion text
- Update CLAUDE.md and global.md docs

* fix(utils): add missing biome noRestrictedImports rule and correct truncate docs

- Add noRestrictedImports to biome.json under style — bans nanoid and uuid
  package imports at lint time (crypto.randomUUID/randomBytes are caught by
  the check:utils grep script which handles global property access)
- Correct truncate() TSDoc and parameter name: sliceLength makes it clear
  that total output length is sliceLength + suffix.length, matching the
  behavior all callers were already written to expect

* fix(utils): add missing getErrorMessage imports at 4 call sites

The sweep agents added getErrorMessage calls without the corresponding
import in 4 files, causing test failures. Added the missing imports.

* fix(utils): fix build errors from getErrorMessage sweep and retry.ts Turbopack issue

- Fix retry.ts cross-file import: Turbopack cannot resolve './random.js' for
  internal package imports; inline the jitter crypto call directly
- Add missing getErrorMessage imports to 32 files where the sweep added calls
  without the corresponding import (caught by type-check and test runs)
- Remove accidental getErrorMessage import from crowdstrike/query/route.ts
  which has its own domain-specific getErrorMessage for parsing CrowdStrike's
  JSON error format
- Fix use-sub-block-value.ts type error from structuredClone narrowing:
  add 'as T' cast at emitValue callsite (safe — valueCopy is always a
  structural copy of newValue)

* fix(tools): use toError in crowdstrike catch block instead of local getErrorMessage

The catch block was calling the local getErrorMessage function which
parses CrowdStrike API JSON responses, not JavaScript Error objects.
Use toError(error).message to correctly extract the message from a
caught value in this context.
2026-05-15 17:31:27 -07:00
Waleed 0c25fc4ee1 fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies (#4369)
* fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies

- auth client now uses browser origin first, falling back to NEXT_PUBLIC_APP_URL
- socket client falls back to page origin when served from non-localhost (assumes /socket.io is proxied)
- add TRUSTED_ORIGINS env var to extend Better Auth trustedOrigins (apex+www, alias hostnames)
- warn at startup when NEXT_PUBLIC_APP_URL is localhost in production
- preprocess empty NEXT_PUBLIC_SOCKET_URL so docker-compose ${VAR:-} works
- migrate remaining uuid/nanoid/randomUUID usages to @sim/utils generateId/generateShortId
- extend generateShortId with optional alphabet param (rejection sampling)
- document TRUSTED_ORIGINS in .env.example, docker-compose.prod.yml, and helm values.yaml

Fixes simstudioai/sim#1243

* fix(auth): address PR review comments

* chore(env): drop unnecessary NEXT_PUBLIC_SOCKET_URL preprocess (skipValidation is true)

* fix(docker): include @sim/utils in migrations image

Migration scripts now import generateId from @sim/utils/id; without copying packages/utils into the image, bun install fails to resolve the workspace dep at build time and the import fails at runtime.

* fix(helm): remove unused NEXT_PUBLIC_SOCKET_URL from realtime sections

The realtime service never reads NEXT_PUBLIC_SOCKET_URL — its env schema
only includes BETTER_AUTH_URL, NEXT_PUBLIC_APP_URL, ALLOWED_ORIGINS,
BETTER_AUTH_SECRET, INTERNAL_API_SECRET, DATABASE_URL, and REDIS_URL.
Remove the dead config from all helm values files and the values schema.

* fix(helm): allow empty NEXT_PUBLIC_SOCKET_URL in values schema

The default in values.yaml is now "" (empty string), which falls back to
the page origin at runtime. The schema previously required a valid URI,
which would reject the default. Mirror the INTERNAL_API_BASE_URL pattern
using anyOf with const "". Also add TRUSTED_ORIGINS to the schema.

* docs(self-hosting): mark NEXT_PUBLIC_SOCKET_URL as optional

The page-origin fallback in getSocketUrl() means self-hosters no longer
need to set NEXT_PUBLIC_SOCKET_URL when realtime is on the same origin
as the app. Update docs to reflect this:

- Remove NEXT_PUBLIC_SOCKET_URL from .env scaffolding examples in
  docker.mdx, platforms.mdx, environment-variables.mdx
- Mark the variable as Optional in the env vars table with the new
  default behavior described
- Update troubleshooting to point at reverse-proxy /socket.io routing
  rather than the env var
- Flip dev docker-compose defaults (local, ollama, devcontainer) from
  http://localhost:3002 to empty for consistency with prod.yml; the
  in-code localhost fallback handles the dev case identically

Applied across all 6 documentation languages (en/fr/de/ja/es/zh).

* chore: untrack and ignore .claude/scheduled_tasks.lock
2026-04-30 19:26:19 -07:00
Theodore Li 8ce56fe1f2 fix(auth): add api key auth via sha256 hash lookup (#4266)
* fix(auth): add api key auth via sha256 hash lookup

* Remove promise all logic

* Restore feature flag

* fix feature flag

* Combine auth and hash gate
2026-04-22 18:30:37 -04:00
Vikhyath Mondreti 41a1b50ace improvement(migrations): log better errors (#4260) 2026-04-21 22:06:05 -07:00
Waleed 0cd14f4ac9 improvement(sso): fix provider lookup, migrate UI to emcn, add enterprise SSO docs (#4238)
* improvement(sso): fix provider lookup, migrate UI to emcn, add enterprise SSO docs

* fix(sso): add org membership guard on providers route, fix idpMetadata round-trip

* fix(sso): add org membership guard on register route, fix SP entityID, remove fullError leak

* fix(sso): fix SAML script callbackUrl and SP entityID to use app base URL

* fix(sso): correct SAML callback URL path in script header comment

* fix(sso): restrict SSO provider read/write to org owners and admins

* docs(sso): restructure page, fix provider guide accuracy, add external doc links

* fix(sso): correct SAML callback path and generate idpMetadata from cert+entryPoint

* fix(sso): always require NEXT_PUBLIC_APP_URL for SAML SP metadata entityID

* fix(sso): scope provider query to org only when organizationId is provided

* fix(sso): escape XML special chars in script idpMetadata generation

* fix(sso): final audit corrections — saml mapping, xml escaping, self-hosted org guard

* fix(sso): redact oidc client secret in providers response, add self-hosted org admin guard

* fix(sso): scope redacted-secret lookup to caller's org or userId

* fix(sso): null out oidcConfig on parse failure to prevent unredacted secret leak

* fix(sso): use issuer as entityID in auto-generated idp metadata xml
2026-04-20 16:45:37 -07:00
mini f46886e6cf fix(sso): default tokenEndpointAuthentication to client_secret_post (#3627)
* fix(sso): default tokenEndpointAuthentication to client_secret_post

better-auth's SSO plugin does not URL-encode credentials before Base64
encoding in client_secret_basic mode (RFC 6749 §2.3.1). When the client
secret contains special characters (+, =, /), OIDC providers decode them
incorrectly, causing invalid_client errors.

Default to client_secret_post when tokenEndpointAuthentication is not
explicitly set to avoid this upstream encoding issue.

Fixes #3626

* fix(sso): use nullish coalescing and add env var for tokenEndpointAuthentication

- Use ?? instead of || for semantic correctness
- Add SSO_OIDC_TOKEN_ENDPOINT_AUTH env var so users can explicitly
  set client_secret_basic when their provider requires it

* docs(sso): add SSO_OIDC_TOKEN_ENDPOINT_AUTH to script usage comment

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>

* fix(sso): validate SSO_OIDC_TOKEN_ENDPOINT_AUTH env var value

Replace unsafe `as` type cast with runtime validation to ensure only
'client_secret_post' or 'client_secret_basic' are accepted. Invalid
values (typos, empty strings) now fall back to undefined, letting the
downstream ?? fallback apply correctly.

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>

---------

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>
2026-04-07 08:46:18 -07:00
abhinavDhulipala 7971a64e63 fix(setup): db migrate hard fail and correct ini env (#3946) 2026-04-04 16:22:19 -07:00
42fb434354 fix(encryption): specify authTagLength on all AES-GCM cipher/decipher calls (#3883)
* fix: specify authTagLength in AES-GCM decipheriv calls

Fixes missing authTagLength parameter in createDecipheriv calls using
AES-256-GCM mode. Without explicit tag length specification, the
application may be tricked into accepting shorter authentication tags,
potentially allowing ciphertext spoofing.

CWE-310: Cryptographic Issues (gcm-no-tag-length)

* fix: specify authTagLength on createCipheriv calls for AES-GCM consistency

Complements #3881 by adding explicit authTagLength: 16 to the encrypt
side as well, ensuring both cipher and decipher specify the tag length.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: clean up crypto modules

- Fix error: any → error: unknown with proper type guard in encryption.ts
- Eliminate duplicate iv.toString('hex') calls in both encrypt functions
- Remove redundant string split in decryptApiKey (was splitting twice)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* new turborepo version

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Lakee Sivaraya <71339072+lakeesiv@users.noreply.github.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com>
Co-authored-by: NLmejiro <kuroda.k1021@gmail.com>
2026-04-01 10:46:58 -07:00
Theodore LiandTheodore Li 5ba3118495 feat(byok-migration) byok migration script (#3584)
* Add byok migration script

* Fix lint

* Add skipping if byok already provided

* Fix lint

---------

Co-authored-by: Theodore Li <theo@sim.ai>
2026-03-14 16:11:21 -04:00
e24c824c9a feat(tables): added tables (#2867)
* updates

* required

* trashy table viewer

* updates

* updates

* filtering ui

* updates

* updates

* updates

* one input mode

* format

* fix lints

* improved errors

* updates

* updates

* chages

* doc strings

* breaking down file

* update comments with ai

* updates

* comments

* changes

* revert

* updates

* dedupe

* updates

* updates

* updates

* refactoring

* renames & refactors

* refactoring

* updates

* undo

* update db

* wand

* updates

* fix comments

* fixes

* simplify comments

* u[dates

* renames

* better comments

* validation

* updates

* updates

* updates

* fix sorting

* fix appearnce

* updating prompt to make it user sort

* rm

* updates

* rename

* comments

* clean comments

* simplicifcaiton

* updates

* updates

* refactor

* reduced type confusion

* undo

* rename

* undo changes

* undo

* simplify

* updates

* updates

* revert

* updates

* db updates

* type fix

* fix

* fix error handling

* updates

* docs

* docs

* updates

* rename

* dedupe

* revert

* uncook

* updates

* fix

* fix

* fix

* fix

* prepare merge

* readd migrations

* add back missed code

* migrate enrichment logic to general abstraction

* address bugbot concerns

* adhere to size limits for tables

* remove conflicting migration

* add back migrations

* fix tables auth

* fix permissive auth

* fix lint

* reran migrations

* migrate to use tanstack query for all server state

* update table-selector

* update names

* added tables to permission groups, updated subblock types

---------

Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: waleed <walif6@gmail.com>
2026-02-19 13:11:35 -08:00
Waleed ac991d4b54 fix(sso): removed provider specific OIDC logic from SSO registration & deregistration scripts (#2896)
* fix(sso): updated registration & deregistration script for explicit support for Entra ID

* cleanup

* ack PR comment

* ack PR comment

* tested edge cases, ack'd PR comments

* remove trailing slash
2026-01-19 19:23:50 -08:00
Waleed 9502227fd4 fix(sso): add missing deps to db container for running script (#2746) 2026-01-09 09:42:13 -08:00
Waleed 10692b5e5a fix(db): remove overly complex db connection logic (#1538) 2025-10-02 19:54:32 -07:00
Waleed fa9c97816b fix(db): add more options for SSL connection, add envvar for base64 db cert (#1533) 2025-10-02 15:53:45 -07:00
Waleed ace83ebcae feat(cmdk): added knowledgebases to the cmdk modal (#1530) 2025-10-01 21:21:42 -07:00
Waleed 980a6d8347 improvement(db): enforce SSL everywhere where a DB connection is established (#1522)
* improvement(db): enforce SSL everywhere where a DB connection is established

* remove extraneous comment
2025-10-01 19:09:08 -07:00
Waleedandwaleed 6e63eafb79 improvement(db): remove vercel, remove railway, remove crons, improve DB connection config (#1519)
* improvement(db): remove vercel, remove railway, remove crons, improve DB connection config

* remove NEXT_PUBLIC_VERCEL_URL

* remove db url fallbacks

* remove railway & more vercel stuff

---------

Co-authored-by: waleed <waleed>
2025-10-01 16:37:13 -07:00
bf1c178282 feat(sso): add support for login with SAML/SSO (#1489)
* feat(sso): added login with SAML/SSO

* restore env

* fixed login styling

* upgrade deps, update UI

* more styling improvements

* reran migrations, tested with script

* improvement(auth): created SSO page

* improvement(auth): remove email option for SSO if not enabled

* cleanup

* cleaned up, added documentation for SSO/SAML config + tested registering either one with script and UI form

* cleanup

* ack PR comments

* move sso known providers to consts

---------

Co-authored-by: waleed <waleed>
Co-authored-by: Emir Karabeg <emirkarabeg@berkeley.edu>
2025-09-29 20:05:26 -07:00
Vikhyath Mondreti 9de7a00373 improvement(code-structure): move db into separate package (#1364)
* improvement(code-structure): move db into separate package

* make db separate package

* remake bun lock

* update imports to not maintain two separate ones

* fix CI for tests by adding dummy url

* vercel build fix attempt

* update bun lock

* regenerate bun lock

* fix mocks

* remove db commands from apps/sim package json
2025-09-17 15:41:13 -07:00