mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
efe1de315eb592bcb2ced4fc84c18eac65e87b68
5606
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Waleed
|
efe1de315e |
fix(tables): sniff CSV delimiter and capture the real header row on import (#5888)
* fix(tables): sniff CSV delimiter and capture the real header row on import Table CSV import derived the delimiter purely from the file extension, so semicolon-delimited exports (European-locale Excel) parsed as a single column. Header derivation also read Object.keys(rows[0]), so with relax_column_count a ragged/sparse first data row dropped its trailing columns from schema inference. - Add detectCsvDelimiter: trial-parse the head against ,/;/tab/| and pick the candidate with the most columns (tie-break on row-width consistency), quote-aware so separators inside quoted cells don't skew the guess; extension is now only the fallback - Add sniffCsvDelimiterFromStream: memory-bounded (64KB) peek-then-replay for the streaming import paths so multi-GB files sniff without buffering - Capture the true header row via the csv-parse columns callback on every path, fixing dropped columns when the first row is short - Wire into the create route, append/replace route, background runner, client dialog preview, and parseFileRows (copilot) * fix(tables): rank CSV delimiter by row consistency and bound the peek buffer Addresses review findings on the delimiter sniffer: - Rank candidates by row-width consistency first (modal column count), then column count. Ranking on column count alone let a comma that appears in a semicolon file's unquoted header win over the real separator; the wide-but-ragged split now loses to the uniform one. - Move the partial-trailing-line trim into detectCsvDelimiter so every path (stream, client preview, parseFileRows) prepares the sniff sample identically and can't disagree on the delimiter. - Cap the stream peeker's detection copy at the sniff window via Buffer.concat's length arg and replay the buffered chunks by reference, so an oversized source chunk can't break the bounded-memory contract. * fix(tables): score CSV delimiter by width*consistency and dedupe headers Addresses round-2 review findings: - Score each delimiter candidate by modalWidth * consistency instead of consistency alone. Consistency-only let a separator that appears uniformly inside values (e.g. a pipe once per row) beat a real delimiter whose rows are legitimately ragged; the product rewards a split that is both wide and uniform, with width breaking ties. Genuinely symmetric files (two valid columns under either separator) fall back to the global-default candidate order. - Dedupe the captured header row to match the parser's record keys. csv-parse collapses duplicate column names onto one key (last value wins), so reporting the raw duplicates made schema inference invent a phantom empty column and could fail mapping validation. dedupeHeaders keeps first-occurrence order, case-sensitively, matching the emitted keys. * fix(tables): keep the final row when sniffing a complete CSV detectCsvDelimiter always trimmed to the last newline, so a complete file with no trailing newline lost its final data row from scoring — leaving the header alone and letting a header-widening separator beat the real one. It now takes a `complete` flag: the trim runs only for truncated prefixes, where a mid-record cut must be dropped. The stream sniffer passes it from `exhausted`, and the buffered callers (client preview, parseFileRows) pass it when the sample covers the whole file. * fix(tables): keep the double-cast annotation adjacent to its cast Hoist the csv-parse options out of the multi-line parse() call so the `// double-cast-allowed` annotation sits directly above the `as unknown as` token — the strict boundary audit only matches an annotation within three lines of the cast. Also simplify the stream sniffer's completeness check to `exhausted` (the loop only ends early on end-of-stream, so it already means the whole file fit the sniff window). * fix(tables): observe EOF at the exact sniff-window boundary The stream sniffer's read loop stopped as soon as the buffered size reached the sniff window, so a file whose size is exactly the window never triggered the extra read that observes end-of-stream — it was judged a truncated prefix and dropped its final newline-less row, disagreeing with the buffered callers (which mark that size complete). Read while size <= window so the boundary case sees EOF and `exhausted` (hence `complete`) is set correctly. Regression test added. * test(tables): use sleep helper instead of raw setTimeout promise |
||
|
Waleed
|
44749b85b3 |
improvement(credentials): actionable Shopify admin-token rejection message (#5891)
Shopify custom-app token verification faithfully surfaces a real 401 from Shopify, but the generic 'double-check it in Shopify' copy didn't tell users what to check. The #1 real cause is pasting the wrong secret (API key / API secret key) instead of the shpat_ Admin API access token, or using a token bound to a different store. - Add an optional per-provider invalidCredentialsHelp override on the token service-account descriptor; set it for Shopify to name the exact fix. - Move the error-code to message mapping out of the shared connect modal into the descriptor module (getTokenServiceAccountErrorMessage) so provider copy is inherited from the definition rather than hard-coded in the modal. - Add unit tests for the mapper (override, fallback, all codes). |
||
|
Waleed
|
78fb2c0679 |
chore(deps): bump next to 16.2.11 to clear security advisories (#5890)
* chore(deps): bump next to 16.2.11 to clear security advisories Patches SSRF, cache confusion, DoS, and middleware-bypass advisories (GHSA-89xv-2m56-2m9x et al.) affecting next < 16.2.11 across apps/sim, apps/docs, and packages/emcn. Excludes next/@next/env from the minimum-release-age gate until the 7-day window elapses on 2026-07-28. * chore(deps): drop aged-out typescript entries from release-age excludes typescript and @typescript/typescript6 passed the 7-day minimum-release-age gate (aged out 2026-07-15 and 2026-07-13), so their exclusions are no longer needed. Keeps @typescript/native-preview (permanent nightly builds) and the Pi packages (age out 2026-07-24). |
||
|
Waleed
|
3914d2ec24 |
improvement(landing): keyword-forward SEO copy on the home and enterprise pages (#5887)
* improvement(landing): keyword-forward SEO copy on the home and enterprise pages * fix(landing): sync homepage JSON-LD title and restore visible open-source claim |
||
|
Theodore Li
|
0199508706 | fix(custom-blocks): stop image icons blowing up on class-sized surfaces (#5878) | ||
|
Vikhyath MondretiandSim Pi Agent
|
4856ef396c |
feat(library): Best AI Agents for Customer Support Automation (#5879)
Co-authored-by: Sim Pi Agent <pi@sim.ai> |
||
|
Waleed
|
387fa378a5 |
fix(knowledge): purge trashed Google Sheets tabs on a normal sync (#5883)
* fix(knowledge): purge trashed Google Sheets tabs on a normal sync Trashing a spreadsheet made listDocuments return an empty listing, but the sync engine's zero-document guard skips deletion reconciliation whenever a listing comes back empty and documents already exist — it can't tell a genuinely empty source apart from a provider outage. For a single-spreadsheet connector, trashing its one source item empties the entire listing, so the guard always fired and the stale tabs never got cleaned up on a normal sync, contradicting the documented behavior. Add shouldSkipEmptyListing, mirroring shouldReconcileDeletions: a connector can now set syncContext.sourceConfirmedEmpty when it has positively confirmed the empty result against the source (not merely inferred it from an empty listing page), letting reconciliation proceed. The Google Sheets connector sets this flag when it confirms the spreadsheet is trashed via a direct Drive metadata lookup. No other connector sets it, so this doesn't change behavior anywhere else. * fix(knowledge): let sourceConfirmedEmpty also bypass the mass-deletion safety threshold The zero-document guard bypass alone wasn't enough: for a trashed spreadsheet with more than 5 tabs, reconciliation would proceed but the separate mass-deletion ratio guard (>50% deleted, >5 docs) still blocked the actual delete on a normal sync, requiring a forced full resync anyway. Extracted the ratio guard into exceedsDeletionSafetyThreshold, mirroring shouldSkipEmptyListing, so a connector's positive source confirmation bypasses both guards consistently. |
||
|
Waleed
|
874e742a47 |
fix(connectors): purge archived/deleted source items in KB connectors (#5880)
* fix(confluence): exclude archived pages from KB connector listings so reconciliation purges them
* fix(connectors): purge archived/deleted source items across seven more KB connectors
The sync engine only purges a knowledge-base document when its source item is
absent from a full-sync listing, so any connector that keeps listing
archived/trashed/canceled items never drops them. An audit of all 51 connectors
found seven with this bug:
- asana: list only non-archived projects (the API returns both when `archived`
is omitted), so tasks under archived projects stop being re-listed
- google-sheets: skip a spreadsheet Drive reports as trashed, which stays
readable by id for 30 days before the Sheets call starts 404ing
- incidentio: exclude canceled incidents by default (cancelling is incident.io's
documented stand-in for deletion), with an explicit opt-in to sync them
- outlook: exclude Deleted Items from the all-mail listing, which Graph
otherwise includes
- servicenow: drop retired knowledge articles, which the Table API returns with
no implicit state filter
- webflow: drop archived CMS items, which the staged items endpoint always
returns and offers no way to filter
- youtube: drop playlist entries whose video was deleted or made private, which
the API keeps returning as placeholder items
Every exclusion keys off an explicit non-current signal and fails open on a
missing field or a failed metadata read, since wrongly excluding a live item
would hard-delete it. Explicit user filter selections are still honoured
verbatim; the new defaults apply only when nothing is configured.
Also flag truncated listings as capped in asana, outlook, and servicenow. All
three silently cut a listing short at their configured item cap without setting
`syncContext.listingCapped`, so reconciliation read the untraversed tail as
deleted at the source and hard-deleted it.
* fix(asana): honour the pinned-project exception on the task rehydrate path
listDocuments deliberately keeps syncing a project the user pinned via the
`project` config field even once it is archived, but getDocument ignored
sourceConfig and applied the all-parents-archived exclusion unconditionally.
For a pinned archived project the listing kept emitting its tasks while every
hydration returned null, so new tasks were dropped as empty and already-indexed
ones were frozen at their last content.
isTaskUnderActiveProject now takes the pinned project gid and keeps any task
reachable through it, matching the listing exactly. The unpinned path is
unchanged and still fails open on missing/non-boolean archived values.
* fix(connectors): key removal on explicit source signals, never on absence
Follow-up to the connector purge fixes, from an independent audit.
YouTube inferred deletion from absence: a playlist entry whose id was missing
from a `videos.list` response was dropped, so a well-formed 200 that returned 49
of 50 requested ids hard-deleted the 50th. Playlist items instead carry a
documented `status.privacyStatus`, available as a free part on a call the
connector already makes, so the extra `videos.list` request is gone along with
its quota-failure and pagination-wedge risks. An item is now excluded only on an
explicit `private`; missing, empty, or unrecognized values keep it.
ServiceNow read every record through a guard requiring a string `sys_id`, but
the listing requests `sysparm_display_value=all`, under which every field —
`sys_id` included — comes back as `{display_value, value}`. The guard rejected
every record, so the retired-article filter was unreachable and the sys_id
object would have leaked into `externalId` and `title` had it not been. Records
are now read through the existing `rawValue` normalizer, which accepts both wire
shapes, and the fixtures use the shape the API actually returns.
Also: resolve the ServiceNow cap ambiguity with `X-Total-Count` so a table that
ends exactly on a page boundary is not read as truncated; stop the Google Sheets
comment claiming a purge the engine's zero-document guard prevents; and assert
the Outlook junk-mail invariant instead of comparing a constant to itself.
Document the behavior change: content archived, retired, or trashed at the
source is now removed from the knowledge base, and restoring it re-ingests it.
|
||
|
Waleed
|
26e4c30888 |
fix(ci): unblock the deploy chain and remove the promotion-dedup gate (#5881)
* fix(ci): unblock the deploy chain — explicit need results on jobs downstream of test-build test-build's needs chain now contains dedup-promotion, which is skipped on every push event. A skipped transitive ancestor fails the implicit success() on downstream jobs, so migrate, promote-images, create-ghcr-manifests, process-docs, and create-release all cascade-skipped on push runs — blocking staging and main deploys (no ECR tag push, no CodeDeploy). Each of those jobs now uses !cancelled() plus explicit needs.<job>.result == 'success' checks, preserving their original semantics while ignoring the skipped ancestor. * chore(ci): remove the promotion-dedup gate — savings don't justify deploy-graph complexity The bespoke dedup job hand-rolled what content-addressed caching solves idiomatically, saved only ~$50-90/mo, and its needs edge just caused the deploy-chain skip incident. test-build returns to its original shape; the explicit need-result conditions on the deploy chain stay as hygiene. |
||
|
Waleed
|
227cd65f41 |
fix(mcp): bound and retry OAuth start so a transient stall recovers instead of a blank popup (#5874)
* fix(mcp): bound and retry OAuth start so a transient stall recovers instead of a blank popup Empirically root-caused a blank/stuck authorize popup: the provider (planetscale) and our guarded OAuth fetch are both fast (120/120 legs clean from staging), and /oauth/start uses local AES encryption with no Redis lock in its path — so the intermittent hang is the same transient headers-then-stalled-body class we've documented for CDN-fronted MCP hosts (a per-connection stall a fresh attempt dodges), which /oauth/start had no server-side bound against. - Bound every /oauth/start step with the shared timedStep helper (extracted from the callback route, now used by both) + an entry log, so a stalled step surfaces as a labeled error instead of hanging the request (and the browser popup) to the client's 30s timeout. - Retry mcpAuthGuarded once on a bounded 12s timeout: a fresh attempt gets a fresh connection and recovers from the transient stall automatically (two 12s attempts stay under the client's 30s deadline). McpOauthRedirectRequired (the success signal) and DCR-unsupported errors are never retried. Adds OauthStepTimeoutError + makeTimedStep to the shared oauth barrel and test mocks. * fix(mcp): drop the unsafe OAuth-start retry; fail fast without error-logging success Review fixes on the bound+retry change: - Removed the mcpAuthGuarded auto-retry. timedStep can't cancel the loser, so a lingering first attempt shares this server's OAuth row and could overwrite the retry's PKCE verifier / state after the client already got the second authorize URL, breaking the callback. Recovery is now fail-fast (504) → the user re-clicks, which is a clean fresh flow (fresh connection dodges the transient stall) with no shared-state race. - Catch McpOauthRedirectRequired (the success signal) INSIDE the bounded step and return it as a value, so a successful authorize is no longer error-logged as 'OAuth step failed'. - Tighten step budgets (5s DB x3 + 12s auth = 27s) to stay under the client's 30s /oauth/start deadline. * fix(mcp): route all bounded-step timeouts to the 504 handler Move OauthStepTimeoutError handling to the outer catch so a DB-step timeout (loadServer/getOrCreateOauthRow/loadPreregisteredClient) returns the same fast 504 'try again' as the auth step, not a generic 500. Documents that the fresh retry is race-safe: the callback correlates on the state nonce, so a lingering timed-out attempt overwriting the row's state only yields a clean invalid_state on the user's fresh authorize URL — never silent corruption. * fix(mcp): bound the setOauthRowUser write too so no step escapes the budget The user-stamp write was the one DB op left unbounded on the start path; wrap it in timedStep(DB_STEP_MS) so every step stays inside the sub-30s budget and its timeout routes to the same 504. * fix(mcp): shrink OAuth-start step budgets to fit the 30s client deadline with 4 DB steps Bounding setOauthRowUser added a fourth possible DB step, so 4x5+12=32s exceeded the client's 30s /oauth/start abort. Lower DB steps to 4s and auth to 10s: 4x4+10=26s worst case, leaving margin for middleware/network. Comment corrected. |
||
|
Waleed
|
dda602a367 |
improvement(tests+ci): phase 3 — shared-mock convergence completion and CI runner-minute cuts (#5875)
* chore(ci): cut redundant runner minutes — dedup promotion-PR test runs, companion-pr-check concurrency, right-size trivial jobs
- ci.yml: new dedup-promotion gate skips the pull_request test-build on
staging/main-headed promotion PRs only when the merge tree provably equals
the head tree (empty base delta over the merge base) AND the push-event run
at the same sha passed its test jobs (polled). Fail-open on any error/
timeout/failure, job-level skip only (skipped job reports Success); verified
no required status checks are configured on main/staging rulesets.
Measured 39 duplicate PR runs / 5.15 days (~227/mo) at ~7.1 min each on
8vcpu (~57 vcpu-min), probe costs ~9 vcpu-min worst case on 2vcpu.
- companion-pr-check.yml: per-PR concurrency group with cancel-in-progress so
superseded synchronize/edit runs stop; no paths filter (check depends on PR
body + cross-repo state, not changed files).
- detect-version and check-docs-changes: 4vcpu -> 2vcpu Blacksmith runners
(pure shell / depth-2 checkout + path filter only).
* improvement(testing): complete stateful shared mocks for env, urls, redis-config, environment-utils
Shared mock infrastructure for vitest isolate:false convergence:
- packages/testing/src/mocks/env.mock.ts: stateful envMock (live env proxy, setEnv/resetEnvMock, process.env fallback)
- packages/testing/src/mocks/urls.mock.ts: complete urlsMock with real-behavior default impls + resetUrlsMock
- packages/testing/src/mocks/redis-config.mock.ts: adds getRedisConnectionDefaults + resetRedisConfigMock
- packages/testing/src/mocks/environment-utils.mock.ts: new environmentUtilsMock + fns + reset
- contract tests: env.mock.test.ts, urls.mock.test.ts, redis-config.mock.test.ts, environment-utils.mock.test.ts
- packages/testing/src/mocks/index.ts: barrel exports
- apps/sim/vitest.setup.ts: global installs for env, urls, redis, environment/utils
- real-module tests unmocked: lib/core/config/env.test.ts, lib/core/config/redis.test.ts, lib/core/utils/urls.test.ts, tools/index.test.ts (urls)
- stubEnv/process.env fallout migrated to setEnv: lib/webhooks/providers/{revenuecat,rootly,instantly}.test.ts, app/api/auth/oauth2/authorize/route.test.ts
* improvement(tests): drop redundant local mocks in executor/tools/providers and misc dirs (shared-worker readiness)
* improvement(tests): drop redundant local mocks in app routes (shared-worker readiness)
* improvement(tests): drop redundant local mocks in lib (shared-worker readiness)
* fix(ci+testing): live base-tip recheck before dedup skip; prod-aware urls mock fallbacks
- the dedup gate re-verifies merge-tree equivalence against the LIVE base
tip at decision time, closing the window where the base branch gains real
commits during the poll (frozen BASE_SHA check alone was stale)
- the urls mock's getBaseUrl protocol prefix and getBaseDomain parse
fallback now follow the shared isProd flag, mirroring the real module
* fix(ci+testing): fail-closed nojobs fallback in dedup gate; TLS-aware redis defaults mock
- the dedup gate no longer infers coverage from overall run conclusion when
no 'Test and Build /' jobs match — a renamed or skipped test job now runs
the tests instead of skipping them
- the shared getRedisConnectionDefaults mock mirrors the real TLS resolution
(rediss:// to a raw IP requires REDIS_TLS_SERVERNAME and yields
tls.servername)
* fix(ci): keep polling while nested test jobs have not appeared yet
An in-progress push run lists its reusable-workflow jobs only after the
caller starts; nojobs is now terminal (fail closed) only once the run has
completed without them.
|
||
|
Waleed
|
9d8e14ce9f |
improvement(tests): converge env-flags mocks onto a complete shared mock (#5871)
* improvement(tests): converge env-flags mocks onto a complete shared mock
* fix(tests): drop the repo's only bare vi.mock automock
A bare vi.mock('drizzle-orm') automock colliding with factory mocks of the
same module in a shared worker corrupts vitest's mock registry (upstream
vitest-dev/vitest#10290 / #10145, reproduced in isolation). The global
factory mock already covers this suite.
* chore(tests): drop defensive resets in non-mutating suites, merge sequential setEnvFlags calls
* fix(tests): run providers/utils cases sequentially over shared env-flags state
|
||
|
Waleed
|
02bc8f1828 |
feat(ci): warm Next.js builds via Turbopack persistent cache on a sticky disk (#5869)
* feat(ci): warm Next.js builds via Turbopack persistent cache on a sticky disk - enable experimental.turbopackFileSystemCacheForBuild behind NEXT_TURBOPACK_BUILD_CACHE so only the CI check build opts in; production image builds stay on the default cold path until the feature stabilizes - mount ./apps/sim/.next/cache as a Blacksmith sticky disk (cache-mount) instead of actions/cache: the turbopack cache is ~5 GB, which a sticky disk mounts in ~1s while an actions/cache round-trip would eat the win - measured locally: 105s cold compile vs 22s warm (4.8x) * chore(ci): drop the superseded actions/cache comment and restore trailing newline |
||
|
Waleed
|
2b5a92a3c8 |
feat(auth): org session policies — lifetime/idle limits, org-wide revocation (#5862)
* feat(auth): org session policies — lifetime/idle limits, org-wide revocation, cookie-cache versioning * refactor(auth): consolidate session-policy clamp semantics, shared security-policy version module, canonical bounds, docs * polish(session-policy): cleanup pass — muted field labels, spinner reset, state tracker, response-seeded baseline, comment trims * fix(session-policy): govern member sessions by membership (closes revoke cookie-cache hole), normalize createdAt, remount on org switch, sync audit mock * fix(session-policy): clamp pre-join sessions on invite acceptance, normalize expiresAt, sync unified nav test * fix(session-policy): invalidate membership cache on removal/transfer, spare impersonator sessions in revoke-all, raise idle floor to 2x cookie window * fix(session-policy): resolve governing org by membership only — activeOrganizationId goes stale across transfer/leave * fix(session-policy): atomic policy save + eager clamp, asymmetric membership TTL, admin-add cache invalidation * fix(session-policy): org-scoped cookie version string, atomic revoke delete+bump * fix(session-policy): plan-gate effective policy so downgraded orgs stop enforcing automatically * chore(session-policy): drop dead bumpSecurityPolicyVersion helper — call sites bump transactionally * fix(session-policy): unify join paths on applySessionPolicyToNewMember; final audit polish (dead exports, response bound, test name) |
||
|
WaleedandMarcus Chandra
|
8cce661a37 |
feat(api): proxyUrl for residential/custom proxy egress on the API block (#5867)
* feat(api): add proxyUrl for residential/custom proxy egress on the API block The HTTP/API block egresses from the app runtime's fixed datacenter IPs via secureFetchWithPinnedIP, so targets behind Cloudflare/WAF that block datacenter IPs (e.g. state .gov license portals) return 403/429 even when the identical request works from a browser. There was no way to route a request through a residential/custom proxy. Add an optional `proxyUrl` field (Advanced) to the API block. When set, the request routes through the given http:// proxy so it egresses from that proxy's IP. Security: - validateAndPinProxyUrl resolves the proxy host's DNS and blocks private/reserved/loopback IPs (same SSRF guard as target URLs), then pins the connection by rewriting the host to the resolved IP (creds/port preserved), closing the DNS-rebinding window. - Restricted to the http: proxy scheme (https/socks rejected) so host pinning is safe without breaking TLS-to-proxy SNI. - Target-IP pinning is intentionally bypassed when a proxy is active (the proxy resolves the target); target URL validation still runs. Threaded block field -> http tool param -> formatRequestParams -> executeToolRequest (validate + pin) -> secureFetchWithPinnedIP, which swaps its pinned Node agent for HttpsProxyAgent/HttpProxyAgent (keyed off target protocol) when proxyUrl is set. * docs(api): document the Proxy URL advanced field and steer proxy credentials to env vars * fix(api): reject loopback/private proxy hosts unconditionally, closing the self-hosted rebinding gap * chore(api): tighten proxy-path inline comments --------- Co-authored-by: Marcus Chandra <mzxchandra@gmail.com> |
||
|
Waleed
|
21ac7b1bba |
improvement(tests): db-mock migration tranche 4 — billing, webhooks/execution/logs, routes/misc (final) (#5866)
* improvement(tests): db-mock migration tranche 4 — billing, webhooks/execution/logs, routes/misc (final) * fix(tests): route agent-handler MCP server rows through queueTableRows |
||
|
Waleed
|
51307c40c0 |
improvement(tests): db-mock migration tranche 3 — lib/workflows, lib/copilot remainder, ee/core/misc (#5864)
* improvement(tests): db-mock migration tranche 3 — lib/workflows, lib/copilot remainder, ee/core/misc * improvement(tests): use the shared notLike operator in idempotency cleanup suite |
||
|
Waleed
|
52659d4a89 |
improvement(tests): db-mock migration tranche 2 — copilot, mothership, workspaces, connectors, mcp (#5863)
* improvement(tests): db-mock migration tranche 2 — copilot, mothership, workspaces, connectors, mcp * fix(testing): drain unconsumed ...Once overrides in resetDbChainMock vi.clearAllMocks clears call history only — a ...Once override queued by a previous test but never consumed survived into the next test. resetDbChainMock now mockReset()s every shared spy and stable wrapper, which restores the original implementation AND drains once-queues. |
||
|
Waleed
|
62a8ce4953 |
improvement(tests): db-mock migration tranche 1 — knowledge, billing/org, workflows/background (#5861)
* improvement(tests): migrate knowledge, billing/org, and workflows/background suites off private @sim/db factories
* improvement(tests): db-mock migration tranche 1 — knowledge, billing/org, workflows/background
- migrate 19 suites off private vi.mock('@sim/db') factories onto the shared
dbChainMock + queueTableRows API (net ~-1,260 lines of bespoke chain
plumbing); resolves the known shared-worker rival pairs (knowledge
processing-queue vs api utils; billing polluters; persistence/utils vs
schedules/deploy)
- add .for() to the mock's limit builder (drizzle .limit(1).for('update'))
with a contract test
- document the join-table queue fallback footgun on queueTableRows
|
||
|
Waleed
|
c083be9def |
improvement(auth): bump better-auth to 1.6.23 and add trusted-proxy client IP resolution (#5857)
* improvement(auth): bump better-auth to 1.6.23 and add trusted-proxy client IP resolution * chore(billing): record checkout-scope mirror re-verification against @better-auth/stripe 1.6.23 * chore(deploy): expose AUTH_TRUSTED_PROXIES in docker-compose.prod and Helm chart |
||
|
Waleed
|
49d3804bee |
fix(ci): save the Next.js build cache every run instead of freezing it at the lockfile key (#5859)
* fix(ci): save the Next.js build cache every run instead of freezing it at the lockfile key
The cache key was only runner.os + bun.lock hash, and GitHub caches are
immutable per key: the first run after a lockfile change saved the cache
once, then every later run hit the primary key and skipped the save
('Cache hit occurred on the primary key ... not saving cache'), so builds
compiled against a cache stale since the last lockfile bump. Suffix the
key with the commit SHA so each run saves its refreshed cache, and
restore via prefix match to the most recent entry.
* fix(ci): make the Next.js cache key unique per run attempt so reruns can save too
|
||
|
Waleed
|
3215a12da9 |
improvement(testing): consolidate @sim/db mocks into one table-aware chain mock (#5856)
* improvement(testing): consolidate @sim/db mocks into one table-aware chain mock - back databaseMock and dbChainMock with the SAME db instance so a module bound to either export hits identical chain fns — rival-mock divergence between the two @sim/testing db mocks is structurally impossible now - add queueTableRows(table, rows): FIFO per-table select routing keyed by schema-mock table identity, consumed at where() materialization and resolved by every downstream terminal (limit/orderBy/groupBy/for/joins) - delete createMockDb (duplicate chain implementation, no external users) - migrate the five suites that hand-rolled table routing + databaseMock delegation (billing plan/usage/usage-log, admin dashboard-organizations, workspaces/utils) onto queueTableRows; net -295 lines - add a contract test for the mock itself and a test script to @sim/testing so its tests actually run under turbo * fix(testing): harden table routing — join-table queues, direct-await from, mutation isolation - track the chain's tables as a list (from + joins) so rows queued for a join-only table route correctly; from-table queue checked first - make the from/join builder a lazy thenable so awaiting a select with no where clause resolves queued rows (dequeue at await, never double-consumed) - update/delete/set clear the routing context so a mutation's where() can never consume rows queued for a select - document the left-to-right chain-construction assumption; contract tests for all three behaviors * fix(testing): close routing over each chain's own tables for direct-await builders * refactor(testing): move all chain routing state into per-chain closures - shared dbChainMockFns entries become pure spy/override ports: their default implementation returns a sentinel that chain-local builders replace, while any mock* override on the spy wins verbatim - each select().from() captures its own immutable table list; where(), joins, terminals, and direct awaits all resolve through that closure, so partially-built chains for different tables interleave without cross-talk - no module-level routing state remains * fix(testing): lazy queue consumption at resolution and wrapper restore on reset - each chain holds one lazy rows supplier: the queued set is dequeued only when a default thenable actually resolves, so a chain answered by a per-test terminal override leaves its queued rows for the next chain - resetDbChainMock also mockReset()s the stable db entry-point wrappers so direct overrides on databaseMock.db.* cannot outlive a suite |
||
|
Theodore LiandClaude Opus 4.8
|
ae7b5eff6c |
feat(copilot): service account setup & reconnect in chat (#5786)
* feat(copilot): add service_account_get_setup_link handler
Resolves a loosely-specified integration name to the catalog slug whose
detail page mounts ConnectServiceAccountModal, and returns
`/integrations/{slug}?connect=service-account`. The agent surfaces it via
the existing <credential type="link"> tag, so the user gets a Connect
button and supplies the key material in Sim's own form — the agent never
handles the secret.
Exact matches beat fuzzy ones so a caller naming a specific service lands
on it (gmail stays Gmail rather than collapsing to Drive), and family
names resolve through an explicit canonical map rather than to whichever
member sorts first.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Phx1MLjf8Ui3M3VpwisZds
* fix(copilot): reject service account ids in oauth_get_auth_link
The fuzzy provider match falls back to substring containment, so
`slack-custom-bot` contains `slack` and resolved to the Slack OAuth
service. The tool then returned a personal-OAuth authorize URL and
reported success — a user who asked for a shared custom bot got a
Connect button that linked their own account instead. Every service
account id degraded this way (notion-, salesforce-, zoom-, linear-),
always silently.
Guard runs before the fuzzy pass and points at
service_account_get_setup_link. Keys off the id being a service-account
id, not off the integration offering one, so `slack` and `notion` still
resolve for OAuth.
Moves the narrowing predicate out of the integration catalog module so
callers that need only the predicate skip the integrations.json load and
the OAUTH_PROVIDERS walk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Phx1MLjf8Ui3M3VpwisZds
* feat(copilot): open the service account form in-chat instead of linking out
The tool handed back a /integrations/{slug}?connect=service-account URL,
so accepting the agent's offer navigated away from the conversation that
asked for the credential. Adds a `service_account` credential tag that
mounts ConnectServiceAccountModal over the chat; setup_url stays as the
headless/MCP fallback.
The tag carries a provider and no value — the secret is typed into Sim's
own form and never enters the transcript — so the validator gets a branch
alongside secret_input/sim_key rather than falling through to the
value-required check.
Extracts useServiceAccountConnectTarget so the chat and the integrations
page share one source of truth for the connect label and the preview
gate. Custom Slack bots ride the slack_v2 flag; without the shared gate
the chat would have surfaced a setup form the integrations page hides.
Modal is lazy-loaded off the deep path (not the barrel) to keep three
provider-specific setup forms out of the chat's initial chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Phx1MLjf8Ui3M3VpwisZds
* fix(copilot): gate service account tool on the same preview flag as the UI
The in-chat connect button hides itself when the provider's gating block
is preview-hidden (a custom Slack bot needs slack_v2). The tool didn't
check this, so it returned success for slack-custom-bot even when slack_v2
was preview-gated off — the agent said "here's the setup form" and the
button silently rendered nothing, leaving the user with no form at all.
Adds getServiceAccountGatingBlockType as the single source for the
provider→gating-block mapping, consumed by both the tool (server-side, via
getBlockVisibilityForCopilot) and the connect hook (client overlay). When
the gating block is hidden the tool now fails with a fall-back-to-OAuth
message instead of promising an invisible form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Phx1MLjf8Ui3M3VpwisZds
* feat(copilot): make the tool own service-account discovery
Removes the VFS auth-metadata exposure and returns connectNoun from the
service_account_get_setup_link result instead. The VFS aggregate was a
second, viewer-independent source of truth that couldn't agree with the
per-viewer preview gate (it always hid slack-custom-bot, even for viewers
with slack_v2 revealed, while the tool accepts it for them). The tool now
resolves the provider, applies the per-viewer gate, and returns either the
in-chat button + connectNoun or a fall-back-to-oauth error — one source of
truth. connectNoun stays DRY via getServiceAccountConnectNoun, shared with
the connect-button label.
* feat(copilot): make service-account setup a direct tag, no tool
The agent now emits the service_account credential tag directly from
intent — like secret_input — instead of round-tripping through a tool.
Removes service_account_get_setup_link (handler, registration, display
title, Go tool def) and restores auth.serviceAccount as the VFS discovery
field so the agent knows which providers support a service account.
The link-vs-tag distinction was the wrong axis: only oauth needs a tool,
because its button carries a minted URL that can't be reconstructed. The
service_account tag carries just a provider name the agent already knows,
so it needs no tool — discovery lives in the VFS (auth.serviceAccount,
GA-only, so slack's preview-gated custom bot is never proactively
offered), and the per-viewer gate lives in the renderer, which renders
nothing when a provider isn't available for the viewer (no OAuth
fallback — a shared credential and a personal one are different intents).
oauth_get_auth_link's service-account-id guard now points at the tag.
* feat(copilot): support service-account reconnect from chat
Reconnect had no service-account path — it required oauth_get_auth_link
and a link tag for every repair, so rotating a workspace service account
either errored or pushed the user through OAuth.
The service_account tag now takes an optional credentialId; when present
the renderer opens the modal in reconnect mode (rotates the secret on that
credential in place, id preserved) and labels the button "Reconnect X".
credentials.json now carries each credential's type (oauth vs
service_account) so the agent can branch: service accounts reconnect via
the tag + credentialId, oauth via oauth_get_auth_link as before.
* fix(copilot): coherent service-account rejection in oauth_get_auth_link
Review round on #5786:
- The service-account-id guard threw into the generic catch, which
overwrote its recovery hint with a "connect manually" message and a
workspace oauth_url — contradictory signals. It now returns a coherent
failure directly, before the try, with no oauth_url.
- Normalize spaces/underscores before the check so a readable form
("slack custom bot", "google service account") is caught too, not
passed to the fuzzy OAuth resolver.
- Remove listServiceAccountIntegrationNames — dead after the tool was
removed (its only caller was the deleted handler's error copy).
* fix(copilot): service-account discovery must un-gate after the block GAs
Review round on #5786: describeServiceAccountForOAuthProvider used
`getBlock(...)?.preview ?? true`, which treats a GA'd gating block — one
that dropped its `preview` flag, exactly slack_v2's documented migration —
as still gated, so the custom bot would stay omitted from VFS discovery
forever after GA even though the UI shows it. Reuse the canonical
isHiddenUnder(null, block) predicate instead, so a non-preview block is
visible. Adds service-account-gate.test.ts covering preview → omit, GA →
include, and missing → fail-closed with a mocked getBlock (the block
registry is globally stubbed, so the real slack_v2 preview flag isn't
observable through serializeIntegrationSchema).
* fix(copilot): align SA resolver normalization and reject blank credentialId
Review round on #5786:
- resolveServiceAccountIntegration only lowercased/trimmed, but the
oauth_get_auth_link guard normalizes spaces/underscores to hyphens
before rejecting a service-account id and steering the agent to a
service_account tag. The chat renderer then couldn't resolve those same
readable forms ("slack custom bot", "notion_service_account") and
rendered nothing. Apply the same normalization to the id lookups (raw
query still used for display-name matches).
- service_account tag validation rejected a blank provider but allowed a
whitespace-only credentialId, which is truthy — the renderer took the
reconnect path and tried to rotate a non-existent credential. Reject a
blank/whitespace credentialId.
* refactor(credentials): route the editor SA picker through the canonical connect hook
The workflow-editor credential selector (from #5800's merged picker) resolved
its service-account setup surface inline and mounted the modal with NO preview
gate — so a `credentialKind: 'service-account'` picker would offer a custom-bot
setup even when slack_v2 is preview-gated off, the leak the integrations page
and chat already guard against.
Route it through the shared useServiceAccountConnectTarget hook (the same
resolver chat and the integrations page use): suppress the setup action when
`hidden`, and use the hook's vendor-accurate label ("Add private app token",
"Set up a custom bot") as the default connect-row copy. Existing service
accounts stay selectable; the per-block `credentialLabels.serviceAccountConnect`
override still wins. One resolver now backs all three SA connect surfaces.
* docs(add-block): document credentialKind and the service-account picker
The add-block skill had no mention of credentialKind — the mechanism (#5800)
that controls whether an oauth-input offers OAuth, service-account, or a merged
picker — and its example was a plain oauth-input mislabeled "Service Account".
Documents the three credentialKind modes, that a default oauth-input already
lets users select an existing service account (they fold in), and the
credentialLabels / allowServiceAccounts companions. Regenerates the .claude and
.cursor projections.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
WaleedandMarcus Chandra
|
6f33a9485b |
feat(workflows): IDE-style reference viewer for workflows (#5854)
* feat(workflows): add IDE-style reference viewer for workflows
Adds a "Show references" viewer so you can see how workflows connect:
which workflows call a given workflow ("Used by") and which workflows it
calls ("Uses"), rendered as recursive, clickable trees.
- Opened via Cmd/Ctrl+click on a sidebar workflow row and a "Show
references" context-menu item.
- Resolves references through both the workflow / workflow_input blocks
(reusing isWorkflowBlockType) and published custom blocks
(custom_block_* -> source workflow), scoped to the workspace.
- Builds the whole workspace reference graph once from live workflow_blocks
state; cycle-safe DFS marks A->B->A loops as (cycle) leaves.
- Contract-bound GET /api/workflows/[id]/references with workspace-level
authz; React Query hook gated to fetch only when the modal opens.
- Unit tests for the pure graph/tree logic (cycles, self-refs, dangling
drop, custom-block + workflow_input resolution) and route tests
(401/400/403/200).
* fix(workflows): correct reference resolution for active mode, cycles, cache, and graph size
Addresses review findings on the reference viewer:
- Resolve the workflow-block child via resolveActiveCanonicalValue (the
shared SOT) instead of basic-first `||`, so an advanced-mode block whose
old basic workflowId value lingers resolves to the active manual value.
- Keep self-references (A -> A) and render them as a cycle leaf instead of
dropping the edge, matching the cycle-safe viewer's purpose.
- Set the references query staleTime to 0 so reopening the always-mounted
modal refetches live editor state instead of serving a stale cached graph.
- Bound converging paths: a node already expanded elsewhere in the tree is
emitted once more as a plain leaf (edge stays visible) rather than
re-expanded, so a densely reconverging graph can't grow exponentially.
* improvement(workflows): align reference viewer auth, coverage, and UI with platform conventions
- authorize via authorizeWorkflowByWorkspacePermission and derive the
workspace server-side (404/403 semantics; drops the client-supplied
workspaceId query param from the contract, hook, and modal)
- add workflow-tool call edges: workflow_input tools inside tool-input
sub-blocks now appear in both trees; non-call selector shapes stay
deliberately excluded (documented against remap-internal-ids)
- restore native cmd/ctrl+click open-in-new-tab on sidebar workflow rows;
references stay reachable from the context menu
- mount ReferencesModal on demand per row, deleting the prevIsOpen reset,
the enabled knob, and the staleTime-0 workaround (now 30s)
- align the tree with design tokens (--text-icon, --surface-hover, px-4
text gutter) and drop the hardcoded brand hex
- escape LIKE wildcards in the custom_block_ prefix match; import
MAX_CALL_CHAIN_DEPTH instead of mirroring it; remove dead fallbacks,
the duplicate not-found scan, and the redundant custom-block row map
* improvement(workflows): final polish on the reference viewer
- drop the vestigial isOpen prop (conditional mount owns visibility)
- unify on the emcn Workflow icon in tree rows
- inline the static className and derive nodes without an annotation
- remove one restating test comment
* fix(workflows): resolve tool references by active canonical mode and keep the reference cache live
- workflow_input tools inside tool-input now resolve basic/advanced via the
index-scoped canonicalModes override, mirroring execution (Cursor finding)
- staleTime back to 0: no mutation invalidates this key, so a reopen must
background-refetch; on-demand mounting keeps the cached tree painting
instantly (Greptile P1)
- modal header uses the em-dash label-entity convention; tree items carry
aria-level instead of a static aria-selected
* fix(workflows): cover legacy workflow-typed tools and retry depth-truncated expansions
- toolInputCallees matches both workflow tool type spellings via
isWorkflowBlockType and passes the tool's own type as the legacy
canonicalModes fallback, matching providers/utils resolution
- a depth-capped expansion no longer poisons the expanded set, so a
shallower path re-expands the node in full (Cursor finding)
- the allowed-but-workspaceless auth branch now returns 403, not the
authz result's 200
- tests: legacy tool type + per-tool index-scope isolation, diamond
re-expansion with a real subtree, depth ceiling, shallow-path retry
---------
Co-authored-by: Marcus Chandra <mzxchandra@gmail.com>
|
||
|
Waleed
|
5338485ac1 |
feat(sidebar): add Slack Community link to help dropdown (#5858)
* feat(sidebar): add Slack Community link to help dropdown * chore(sidebar): use existing Slack community invite link |
||
|
Theodore Li
|
d7d42faff8 |
fix(mothership): stop-button transitions freeze in place (#5838)
* fix(mothership): freeze the transcript on user stop instead of settle-scrolling * fix(mothership): swap stopped row into the shimmer slot and floor the sizer min-height * fix(mothership): detach auto-scroll on stop and dead-band the sizer floor * improvement(mothership): net-zero settle — equal tail regions, drained floor, one growth signal * fix(mothership): follow the post-stop drain to the end instead of freezing * fix(mothership): single stopped tail region and drain cleanup * fix(mothership): clamp-aware chase interrupt so the floor drain can't park the settle follow * fix(mothership): park the chase only on real upward top moves, not growth * improvement(mothership): stack the stopped status above the actions row * improvement(mothership): compact stopped-turn tail with the original 10px rhythm * fix(mothership): single drain cadence and a settle-window gesture kill switch * fix(mothership): debt-aware drain fast-path and settle-window handle retention |
||
|
Theodore LiandClaude Fable 5
|
b49fe16dae |
improvement(logs): show Redacting status while PII masking runs (#5855)
* improvement(logs): show Redacting status while log-persist PII masking runs Log-stage PII redaction happens at persist time and can take minutes on large payloads, during which the Logs page showed the run as Running long after execution finished. The persist path now flips the log row to 'redacting' (guarded on 'running' so a concurrent cancellation is never clobbered) right before the masking work starts — only when the logs redaction stage is actually enabled — and the terminal update overwrites it with the final status. The Logs UI renders an amber non-filterable Redacting badge (row + details sidebar via the shared STATUS_CONFIG), keeps polling the detail query during the phase, and keeps resolving live progress markers. No migration: status is a free-text column, and the contract already types it as string. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(logs): never let the cosmetic redacting write abort log finalization Review finding: the status flip was awaited without failure isolation, so a transient DB error there rejected applyPiiRedaction before masking and the terminal update never ran. The write is display-only; catch and warn instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Waleed
|
fad9728cbd |
improvement(tests): make vitest suites state-safe with auto-unstub of env/global stubs (#5853)
* improvement(tests): make vitest suites state-safe with auto-unstub of env/global stubs - add unstubEnvs/unstubGlobals to vitest config so vi.stubEnv/vi.stubGlobal are restored after every test - move module-scope fetch/crypto/window stubs into beforeEach across executor, data-drains, and knowledge suites so they hold under auto-unstub (several suites were silently hitting live Datadog/BigQuery/Snowflake/OpenAI/Anthropic endpoints when their module-scope stubs were cleared) - converge billing/workspaces/tools/hooks suites onto the shared @sim/testing mock instances and namespace spies instead of rival file-local vi.mock factories, with explicit beforeEach setup and afterAll restore - give lib/core suites private module instances via vite query-suffix imports where module-level state bakes env at import time - stub auth-client/NEXT_PUBLIC_APP_URL in suites that crashed at collection without a local .env Groundwork for eventually running the suite with isolate: false; no CI behavior change (isolation stays on). * fix(tests): complete databaseMock restore and route selectDistinctOn - restoreGlobalDb now mockReset()s entries whose original implementation was undefined instead of leaving the chain-mock delegation installed - add selectDistinctOn to the setup-level createMockDb and to the dashboard suite's delegated key set so the distinct-on path routes under either binding * fix(tests): run knowledge utils cases sequentially over shared per-test state |
||
|
Waleed
|
936536e553 |
improvement(url-state): platform-wide nuqs audit — migrate remaining view-state, codify conventions (#5851)
* improvement(url-state): migrate ee settings sections to nuqs deep-linkable view-state - audit-logs: types/time-range/start-date/end-date filters move to a co-located search-params.ts (reusing the logs kebab-token time-range parser); search binds to the shared settings ?search= via useSettingsSearch, replacing a hand-rolled debounce effect - access-control: group detail deep-links via ?group-id (push/replace-on-close); search via useSettingsSearch - custom-blocks: block detail deep-links via ?custom-block-id; create flow stays local; search via useSettingsSearch - data-drains + forks: search via useSettingsSearch; forks close now replaces history like the mcp reference pattern - polish: nullable-reason comment on logs startDate/endDate, stale debounce TSDoc now references useDebouncedSearchSetter * fix(url-state): review fixes — resolve custom-block deep links before opening detail, per-surface time-range fallback - custom-blocks: gate the detail view on the resolved block (matching mcp/ access-control), so a dead or still-loading ?custom-block-id no longer flashes a bogus create screen - parseAsTimeRange: unknown tokens now parse to null so each surface's .withDefault applies (logs keeps 'All time'; audit-logs keeps 'Past 30 days' instead of silently widening to all time on a malformed link) - audit-logs: date-picker cancel target can never be 'Custom range' itself on a dateless custom deep link - refresh shared ?search= consumer lists in TSDoc * improvement(url-state): platform-wide sweep — migrate the last three view-state stragglers, codify conventions Sweep across landing, workspace, and settings surfaces found only three remaining candidates (everything else verified clean or correctly non-URL): - knowledge document page: chunk enabled-status filter joins page/search/sort in the URL (?enabled=), resetting page in the same write - workflow-mcp-servers: detail Details/Workflows tab deep-links via ?server-tab, cleared alongside the server id on close - byok: provider search binds the shared settings ?search= via a controlled prop pair (modal/embedded consumers keep local state) Rule updates (.claude/rules/sim-url-state.md): shallow defaults documented, urlKeys kebab remapping, throttleMs deprecation, startTransition with shallow:false, shared-parser null-fallback rule, resolve-before-open gating, close-with-replace, and the reusable-component controlled-search pattern. * improvement(url-state): cleanup pass — replace-on-close for the fork activity view, TSDoc form for the logs nullable comment * fix(url-state): honor a custom time range only when both bounds are present A partial ?time-range=custom deep link (missing start/end) now falls back to the default preset window instead of displaying 'Custom range' while querying an unbounded result set. * fix(url-state): verification-round fixes — reject unparseable date params, tighten docs - new parseAsDateString parser (logs + audit-logs): an unparseable ?start-date=/?end-date= now parses to null (missing bound) instead of crashing the audit-logs render via Invalid Date .toISOString(), and hardens the same class in logs - audit-logs: remove the provably dead cancel-revert branch and its ref — the URL only holds 'Custom range' after Apply writes both bounds atomically - workflow-mcp-servers: reset a lingering ?server-tab= when opening a server so a dead deep link can't re-target the next open - knowledge document: drop the unreachable 'N selected' label branch - sim-url-state.md fact-check corrections: cover apps/sim/ee in paths, focusedBlockId -> currentBlockId (the real store field), note that history/clearOnDefault are nuqs v2 defaults, fix the Suspense cross-reference and parseAsIsoDate serialize detail, clarify the *UrlKeys naming convention; list byok in the shared-search consumer docs * fix(url-state): hold first paint while a custom-block deep link can still resolve A valid ?custom-block-id= no longer flashes the list while the blocks query is pending; a dead id still falls back to the list once loaded. * fix(url-state): include permissions loading in the custom-block deep-link paint hold canAdmin reads false while the permissions context loads, so the hold must gate on permissionsLoading too; drop the blocksPending conjunct — it shares one query with useCanPublishCustomBlock, so isLoading already covers it. |
||
|
Theodore LiandClaude Fable 5
|
70814bc4a2 |
feat(db): role-keyed dbFor clients for cleanup and exec workloads (#5583)
* feat(db): role-keyed dbFor clients for cleanup and exec workloads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy * fix(db): keep cleanup-invoked helpers and snapshot reads on their role pools Route markLargeValuesDeleted / pruneLargeValueMetadata (optional dbClient) and chat-cleanup's file collection through the cleanup pool, and getSnapshot through the exec pool, so the cleanup and inline-execution workloads stop borrowing the process-wide pool for these queries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy * fix(db): dbFor falls back to the process-role URL, not the base URL With DATABASE_URL_WEB/TRIGGER set (as in prod) and the sub-pool URLs unset, falling back to the base URL would silently shift execution-log and cleanup traffic to a different PgBouncer endpoint on deploy. Chain the fallback through the URL the process itself resolved so the rollout stays inert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy * feat(db): log which connection each dbFor sub-pool resolved to One line per role at first use: the dedicated DATABASE_URL_<ROLE> when set, otherwise an explicit fallback message naming the process connection it shares — so a missing/typo'd env var is visible at rollout instead of silent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy * feat(db): route pause/resume and large-value metadata persistence to the exec pool Coverage scan follow-up: the paused_executions / resume_queue / workflow_execution_logs transactions in human-in-the-loop-manager.ts and the large-value owner/reference registration writes on the execution path now use dbFor('exec'), matching the completion writes in the execution logger. All are self-contained; billing calls remain outside the moved transactions on the default client. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Waleed
|
e841e4e731 |
fix(mcp): cap transport response bodies to bound a hostile tools/call payload (#5850)
* fix(mcp): cap transport response bodies to bound a hostile tools/call payload A full-lifecycle comparison against LibreChat found the one place a reference client was stricter: the live transport had no response-body byte cap, so a hostile server could stream an unbounded tools/call result and OOM the process (discovery and OAuth bodies were already capped). Cap non-GET response bodies at 16 MiB — counted as they stream, not buffered — while leaving the standalone GET SSE notification stream uncapped (a cumulative cap would break its long-lived nature). Mirrors LibreChat's transport response-size cap. * fix(mcp): preserve url and redirected when capping a response body new Response() resets url/redirected; the SDK resolves relative auth-metadata URLs (resource_metadata) against response.url, so carry the originals over via defineProperty on the wrapped response. * fix(mcp): drop stale framing headers on a capped response body The wrapped body is the already-decoded stream, so content-encoding/content-length would misdescribe it — strip them, matching bufferUnderDeadline. |
||
|
Waleed
|
a0a437dc99 |
fix(deps): bump sharp to 0.35.3 and js-yaml to 4.3.0 for security advisories (#5848)
* fix(deps): bump sharp to 0.35.3 and js-yaml to 4.3.0 for security advisories * chore: fix import order in file-reader |
||
|
Waleed
|
fd0d08a3b8 |
improvement(mcp): push-driven settings freshness + lifecycle-audit fixes (#5842)
* improvement(mcp): subscribe settings page to the live push, lean on it over re-probing The list_changed → SSE push pipeline was already wired end-to-end but only mounted in the workflow-editor tool picker. Subscribe the settings MCP page to the same shared, reference-counted EventSource so tool changes reflect in real time there too — the reference-client model (discover once, refresh on push). With push now active on the settings page, raise MCP_SERVER_TOOLS_STALE_TIME from 30s to 5min (matching the server-side cache TTL) so revisiting the page leans on push + stored state instead of re-probing every connected server. There is no background poll (no refetchInterval) — this only affects refetch-on-visit-if-stale; real changes still arrive instantly via push. * fix(mcp): re-sync tools on SSE reconnect so a missed list_changed can't strand stale tools A dropped/reconnected EventSource may miss a tools_changed event during the gap, which — with a longer stale time — could leave the page showing old tools until a remount or manual refresh. Invalidate the workspace tools on reconnect (never on the first open), the standard resync-on-reconnect pattern for push clients. * fix(mcp): resync on re-subscribe so events missed while the tab was closed reconcile Leaving the settings tab tears down the shared EventSource; remounting created a new connection whose first open was skipped, so a tools_changed fired while unsubscribed wasn't reconciled (the 5min stale time also won't refetch on remount). Track a per-workspace 'ever subscribed' flag: skip resync only on the session's first subscription (queries fetch fresh then); resync on the first open of any re-subscription and on every reconnect. * fix(mcp): reconcile a recovered server's status on successful discovery + fix stale comment Lifecycle-audit findings: - The per-server tools queryFn invalidated the server list only on error. A server that recovered via the stale re-probe returned fresh tools while the cached status still showed failed → 'tools present but row red' until the list independently refetched (a window the 5min stale-time widened). Now a successful probe against a server the cache still shows non-connected refreshes the list so the row clears. - Correct the keep/drop comment: tools drop once the stored status leaves 'connected' (disconnected/error), not at a 3-failure threshold. * fix(mcp): correct stale SSRF pin docstring and bound the remaining OAuth callback steps Lifecycle-audit follow-ups (merged-code): - validateMcpServerSsrf's return docstring described 'pin subsequent connections', which no longer holds for the public path — the returned IP is a policy signal selecting the validate-at-connect guarded fetch; redirect/rebind safety comes from per-connect validation + followRedirectsGuarded, not pinning (only the self-hosted private carve-out still literally pins). Corrected so a future reader can't reintroduce a real pin from the misleading text. - The callback wrapped only the five post-auth steps in timedStep; the earlier loadOauthRowByState, getSession, and server SELECT (plus the provider_error clearState) were unbounded, contradicting the 'every awaited step bounded' invariant. Wrapped them so a wedged DB read surfaces as a labeled timeout. * fix(mcp): resync on a first SSE open that recovered from an earlier connection error If the initial EventSource connection errors before opening (and the initial tools query fails with retry:false), the first successful onopen was skipped, leaving the query stale. Track erroredBeforeOpen so a first open that followed a connection error also resyncs — only a clean first subscription still skips. * fix(mcp): make the SSE push subscription intrinsic to the tools query The 5min stale time assumed push, but useMcpToolsEvents was only mounted in the settings page and the tool picker — other consumers of the tools query (dynamic args, tool selector, canvas block via useMcpToolsQuery/useMcpTools) had no subscription, so list_changed updates could stay invisible for the full window. Mount useMcpToolsEvents inside useMcpToolsQuery so every tools consumer gets real-time push from the shared, reference-counted connection — no consumer is left re-probing. Removed the now-redundant explicit mounts from the settings page and tool-input. * test(mcp): clear the shared SSE collections instead of reassigning globalThis mcp.ts captures the connections Map and subscribed Set in module consts at import, so setting the globalThis property to undefined didn't reset what the module uses — subscription state leaked across tests. Clear the shared instances instead. * fix(mcp): drop the success-path status reconciliation heuristic The client-side serversList invalidation on a successful probe assumed the probe updated the stored status, but a server-side cache hit returns tools without touching status — so in the failed-cache-delete edge it fired a pointless refetch. The benefit (instant vs the 60s serversList stale-time for status recovery) doesn't justify the incorrect assumption; rely on the existing 60s stale-time + SSE push for status recovery instead. Keeps the corrected keep/drop comment. |
||
|
Waleed
|
e0e6f24f85 | improvement(deployments): shorten lock windows and add tx safety timeouts on the deploy path (#5841) | ||
|
Waleed
|
c708add95f |
fix(mcp): keep last-known-good tools instead of flashing red on a transient failure (#5839)
* fix(mcp): keep last-known-good tools instead of flashing red on a transient failure A connected server with cached tools was turning red 'Failed to discover' on a single transient discovery probe blip, even though the stored status stayed connected with tools — because useMcpToolsQuery dropped React Query's retained data on error, and the row's showDiscoveryError fired over a populated server. Now the aggregate keeps last-known-good tools across a failed refetch, and the row only hard-reds when there are genuinely no tools to show. A persistent failure still surfaces via the stored connectionStatus (gated by MAX_CONSECUTIVE_FAILURES), matching how Claude Code / LibreChat / OpenCode / VS Code handle a transient tools/list failure on a live connection. Validated against those clients' source + the MCP spec before changing; this supersedes the #5829 over-correction that showed the error even when tools existed. * fix(mcp): drop stale tools once a server is persistently failed, keep on transient Sharpens the last-known-good behavior with the transient-vs-persistent distinction the reference clients make: keep cached tools through a transient discovery failure (stored status still healthy) so a populated server doesn't blank, but drop them once the stored connectionStatus crosses its failure threshold to error/disconnected — so the shared aggregate the workflow editor consumes stops offering a dead server's stale tool schemas. |
||
|
Waleed
|
1410aae3e3 |
fix(auth): gate email-otp auto-signup behind DISABLE_EMAIL_SIGNUP (#5840)
* fix(auth): gate email-otp auto-signup behind DISABLE_EMAIL_SIGNUP * fix(auth): also close otp auto-signup under DISABLE_REGISTRATION |
||
|
Waleed
|
93fbf584a2 |
feat(chat): soft-delete sidebar chats with restore from Recently Deleted (#5830)
* feat(chat): soft-delete sidebar chats with restore from Recently Deleted * fix(chat): review round 1 — restore workspace authz, purge recheck, archived-list invalidation * test(chat): update SSE handler assertions for workspaceLists invalidation * fix(chat): bump updatedAt on restore, recheck retention cutoff in task cleanup * fix(chat): drop explicit feedback delete in task cleanup — chat FK cascade covers it * test(chat): cover restore route; guard legacy copilot delete from hard-deleting mothership chats * chore: revert unintended bun.lock drift from worktree install * fix(cleanup): recheck workflow archive cutoff on delete; export deletedAt in chat drain |
||
|
Waleed
|
943d184365 |
fix(mcp): follow tools/list pagination instead of silently truncating (#5833)
* fix(mcp): follow tools/list pagination instead of silently truncating The SDK's listTools() returns a single page; a server that paginates via nextCursor was silently truncated to page one. Follow the cursor bounded by four independent budgets (50 pages / 1000 tools / 5 MB / 60s aggregate wall-clock) plus a repeated-cursor guard — a page cap alone can't stop a server returning a fresh cursor with no new tools. Partial results from earlier pages are kept when a later page fails; only a page-one failure throws. Matches the LibreChat capped-cursor-loop pattern; caps live in MCP_CLIENT_CONSTANTS. * fix(mcp): count UTF-8 bytes, keep empty partials, and log page-cap accurately Review fixes on the tools/list pagination loop: - Buffer.byteLength(..., 'utf8') instead of .length so non-ASCII schemas can't overshoot the 5 MB budget by counting UTF-16 code units. - Track pagesFetched (not tools.length) for the partial-success decision, so a valid-but-empty first page followed by a failing page returns [] instead of throwing and marking the server unhealthy. - Explicit reachedEnd/page-cap distinction so a natural finish on exactly MAX_PAGES isn't mislogged as truncated. |
||
|
Vikhyath MondretiandSim Pi Agent
|
64353f210e |
feat(library): Best AI Agents for Data Extraction and RAG in 2026 (#5834)
* feat(library): Best AI Agents for Data Extraction and RAG in 2026 * fix(library): correct CSV import claims, fix comparison link, drop duplicate FAQ body * fix(library): remove unsupported GDPR compliance claim * improvement(library): update connector source count to match registry (50+) --------- Co-authored-by: Sim Pi Agent <pi@sim.ai> |
||
|
Theodore Li
|
2cbecb5baa |
improvement(mothership): stable thinking indicator and jump-free streaming scroll (#5828)
* improvement(mothership): stable thinking indicator and jump-free streaming scroll * fix(mothership): suppress shimmer over executing tool rows and seed chase interrupt baseline * fix(mothership): keep shimmer mounted through the slot collapse so it animates out * improvement(mothership): tighten transcript bottom padding, timed slot-exit latch, cleanup pass * fix(mothership): bridge hidden special-tag streaming with the shimmer * fix(mothership): hold sizer floor through reveal and reset chase deadline on park * improvement(mothership): swap actions into the thinking slot at settle, quicken the chase |
||
|
Vikhyath Mondreti
|
1b48eeb991 | improvement(subblocks): trust block registry over stored subblock type (#5832) | ||
|
Waleed
|
a360cd4d7c | fix(workflow): align condition/router handles with their rows (#5831) | ||
|
Waleed
|
1f9f65df70 |
improvement(mcp): make the OAuth experience visible and verbally consistent (#5829)
* improvement(mcp): make the OAuth experience visible and verbally consistent From a full UX-consistency audit of the MCP settings surfaces: - One name for one action: 'Authorize' / 'Reopen authorization' everywhere (list chip, row, detail) — was three different labels across surfaces. - The connecting state is now visible: the row subtitle shows a muted 'Waiting for authorization...' instead of continuing to shout the red 'OAuth authorization required' mid-flow. - The Authorize affordance is a visible chip on the list row (was buried in the overflow menu), so a blocked popup's 'retry' has an obvious target. - Removed the redundant aggregate discovery banner — every failing row already reports its specific error; two red messages for one failure. - The header Refresh chip no longer renders an error sentence as its label (short 'Failed'; the Status field carries the explanation). - Sentence-case sweep (Unnamed server, Not connected, Server name, Add MCP server / Edit MCP server, Add server, Test connection, Edit form, Delete MCP server), 'Search servers...' placeholder, row-subtitle/error text on the canonical tokens, and a Loading empty state instead of a blank flash. * fix(mcp): show stale-discovery failures, best-effort popup-close label clear, drop redundant branch - A failing latest discovery now shows its error even when cached tools exist (the stale tool count silently hid it). - Best-effort popup.closed poll clears only the 'Waiting for authorization...' label share; the flow entry stays registered so a completion that still arrives over the BroadcastChannel is honored (settleFlow skips the double-decrement). Under COOP misreport the worst case is an early label reset, never a dropped completion. - Remove the OAuth refresh-state branch made redundant by the 'Failed' change, plus its now-unused authType/error inputs. |
||
|
Theodore Li
|
e3f9deb65e | fix(slack): allow empty status to clear the assistant status indicator (#5827) | ||
|
Waleed
|
e51a867de3 |
fix(mcp): open the OAuth popup synchronously and bound the start request (#5824)
* fix(mcp): open the OAuth popup synchronously and bound the start request Two bugs behind 'pressed Connect/Reopen and nothing happened': - window.open ran AFTER awaiting /oauth/start, outside the browser's user activation, so the popup was silently blocked (worst on the add-server flow). Popup-first now: open about:blank synchronously in the click (named window, so a re-click focuses/reuses an existing authorization window), navigate it once the start returns; close it on failure/already_authorized; clear toast when genuinely blocked (and skip the request entirely). - /oauth/start had no client timeout, so a stalled start held the re-entrancy guard and the connecting label indefinitely. Bounded at 30s; on timeout the popup closes, the label resets, and retry is immediately available. * fix(mcp): harden popup-first reopen edges - Retire prior flows as soon as the named popup opens (the open already blanked any prior auth window; a failed start must not leave a windowless flow 'connecting' for the 10-minute safety timeout). - Check the COOP-fallback window.open result; when blocked, clear the state and toast instead of registering a windowless pending flow. - Feature-detect AbortSignal.timeout (Safari <16) with an AbortController fallback for the bounded /oauth/start. * fix(mcp): close the blank popup when post-start navigation fails * chore(mcp): correct connecting-count ordering comment |
||
|
Waleed
|
3458220ec9 |
fix(mcp): replace single-IP pinning with validate-at-connect SSRF guard (#5823)
* fix(mcp): replace single-IP pinning with validate-at-connect SSRF guard Swaps the MCP transport + OAuth guard from pin-one-resolved-IP to the standard pattern (LibreChat getSSRFConnect): DNS resolves normally and EVERY socket connect filters the resolved addresses against the private/reserved blocklist, closing the validate-then-trust window a rebind could race and removing the non-standard mechanism implicated in the headers-then-no-body stalls (no reference MCP client pins IPs; a pinned attempt welded to a bad flow cannot escape, while retries rotate via resolver round-robin and succeed). Adversarially reviewed before shipping; both findings closed here: - Redirects are now followed manually with per-hop validation: an IP-literal redirect target (which bypasses ANY connect-time lookup - Node skips the custom lookup for numeric hosts) is checked explicitly, closing a metadata- endpoint redirect hole the old pin also had. - Custom request headers are dropped on cross-origin hops, so a redirect to a second attacker host cannot harvest configured auth headers. Policy gating unchanged: the guard activates exactly where the pin did (validateMcpServerSsrf non-null; allowlist mode / localhost-on-self-hosted stay unguarded). Non-MCP consumers of the pinned fetch are untouched. 397 tests incl. new rebinding, mixed-answer, IP-literal-redirect, and hop-cap coverage. * fix(mcp): restore IPv4 preference and harden the guarded redirect follower - Restore validateUrlWithDNS's prefer-IPv4 resolution (a stale working-tree hunk accidentally reverted #5798 for the still-pinned non-MCP consumers). - Validate the INITIAL url's IP-literal in followRedirectsGuarded, not just redirect hops, so the exported guard is self-contained. - Drop entity headers (content-length/type/encoding) when a 301/302/303 switches a POST to a bodyless GET, which undici would otherwise reject. - Lift method/headers/body/signal from a Request input instead of silently downgrading a guarded POST Request to a bare GET. * fix(mcp): annotate headers double-cast and cancel redirect body before throw paths - Add the missing double-cast-allowed annotation on the sanitized-headers cast (strict boundary audit). - Cancel the redirect response body before the hop-cap / blocked-target throws so those paths can't leave a socket checked out on the long-lived Agent. * fix(mcp): keep self-hosted private-resolving hosts unguarded (old-pin parity) A DNS alias resolving to loopback/private is only reachable on self-hosted, where the policy explicitly permits it; the guarded lookup would filter the address and strand the connect where the old pin connected. Both MCP gates (transport + OAuth guard) now route private/loopback resolutions over the unguarded path, same as the localhost carve-out. Test IPs moved off RFC-5737 TEST-NET (which is correctly classified reserved). * fix(mcp): pin (not skip) self-hosted private resolutions; refuse cross-origin body forwards - The private/loopback carve-out now keeps the LEGACY PIN to the validated address instead of falling back to unguarded fetch — preserving both the old behavior and its anti-rebinding property for self-hosted DNS aliases. - Cross-origin redirect hops now also refuse to forward a request body (307/308 preserve method+body; post-pin those redirects really dial the new origin, so an open redirect could exfiltrate OAuth client secrets). Bodyless cross-origin redirects still follow. Tests for both. * chore(mcp): true up stale pinned-era doc comments |
||
|
Theodore Li
|
e9898ea24d | fix(ci): run Build App on the larger runner in GitHub mode (#5826) | ||
|
Theodore Li
|
4b979780d2 |
fix(ci): make GitHub-hosted fallback actually build and test (#5814)
* fix(ci): make GitHub-hosted fallback actually build and test * fix(ci): drop unused build-args passthrough so the heap ceiling can't be overridden * chore(ci): trim comments to the non-obvious constraints * fix(ci): build the app image on a larger runner in GitHub mode |
||
|
Vikhyath Mondreti
|
995167dd18 | improvement(admin): cleanup included usage settings for enterprise (#5825) | ||
|
Waleed
|
daa2416cba | improvement(search-modal): remove matched-character bolding and move SearchHighlight into knowledge (#5822) |