feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991)

* feat(realtime): add shared room identity + authorization spine (#5929)

Introduces the foundation for a unified realtime "room" model spanning the
Socket.IO presence server (apps/realtime), the durable SSE event log, and the
ephemeral pub/sub fanout — all of which today reinvent their own room identity,
naming, and authorization.

- @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a
  roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the
  ~40 existing io.to(workflowId) callsites and presence state keys are unchanged;
  every other room type is namespaced so id spaces cannot collide.
- @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the
  exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace
  resolver per room type, then the shared resolveEffectiveWorkspacePermission +
  permissionSatisfies gate.

Pure foundation, no behavior change: nothing consumes these yet. Prune graph
stays at 14/25 (platform-authz already depended transitively on realtime-protocol
via apps/realtime).

* refactor(realtime): generalize presence server to multi-room [2/N] (#5930)

* refactor(realtime): generalize presence server to multi-room (RoomRef)

Generalizes the Socket.IO presence layer from single-workflow-room-per-socket
to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second
domain (workspace files, next PR) can reuse the same membership + presence
engine. Behavior-preserving for workflow collaboration.

IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/
getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The
workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the
manager into WorkflowRoomService, composed over the generic manager.

Backward-compat by design (no workflow migration, no regression):
- Workflow Socket.IO room name stays the bare workflowId (roomName() maps
  workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched.
- Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix
  IS "workflow").

Multi-room correctness (from adversarial audit):
- socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id).
- The SHARED socket:{id}:session key is deleted only when the socket leaves its
  LAST room (refcount via HLEN) — a leave from one room no longer breaks the
  other room's handlers.
- disconnect enumerates the socket's stored rooms and rebroadcasts presence per
  room, instead of picking an arbitrary socket.rooms entry.
- presence broadcasts use a per-room-type event name (workflow keeps the bare
  presence-update; others are namespaced).

Workflow handlers wrap manager calls with a shared workflowRoom(id) helper;
UserPresence.workflowId -> room (the client never reads that field).

Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new
multi-room tests (refcounted session, presence isolation, multi-room disconnect,
per-type event names). tsc clean, boundaries + prune (14/25) green.

* fix(realtime): harden multi-room disconnect + id-guard room removal

Two fixes from an adversarial regression audit of the multi-room refactor:

- Disconnect now handles `disconnecting` (where `socket.rooms` is still populated
  and authoritative) and falls back to the live Socket.IO room set for any room
  the manager's stored state no longer tracked. This restores reliable presence
  cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was
  evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect.
- REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the
  last-room session cleanup) when the stored id matches the room being removed,
  matching the memory manager's existing id guard. Prevents a mismatched-room
  call from wiping a different room's mapping or the shared session.

+1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean.

* fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded

Greptile 4/5 follow-up: the disconnecting-time fallback ignored
removeUserFromRoom's boolean and rebroadcast presence even when the removal
reported false. Now it only treats a room as removed (and rebroadcasts) when the
manager confirms it — symmetric with removeSocketFromAllRooms, which already only
returns rooms it actually removed.

* fix(realtime): exclude the disconnecting socket from its farewell broadcast

Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on
disconnect, the socket's presence entry can outlive it (room hashes have no TTL)
and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the
socket was in (union of the manager's removed rooms and the live Socket.IO
membership) and passes the disconnecting socket id as excludeSocketId, so it is
never shown as a collaborator regardless of whether the Redis delete succeeded.
Any orphaned entry is still reclaimed by the next join's stale-presence sweep.

broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are
unchanged. +1 test.

* fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix)

Presence broadcasts now reconcile the stored list against the live Socket.IO
membership (io.in(room).fetchSockets()) before emitting, via a shared
filterVisiblePresence helper. This closes the residual behind the earlier
disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL)
could reappear in a LATER join's presence snapshot until the 75-min stale sweep.
Now such an entry is never emitted, because a non-live socket is filtered out of
every broadcast. Combined with excludeSocketId (which handles the disconnecting
socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty
result while entries remain, emit the unfiltered list rather than hide live
collaborators.

Also drops a dead guard in the disconnect union loop (rooms already removed are
skipped by the wasInRooms check) and the now-unused isSameRoom import.

+1 ghost-guard test. 122 realtime tests pass.

* feat(files): live presence avatars + live file tree via realtime rooms (#5932)

* refactor(tables): adopt shared durable event-log core (#5934)

* fix(realtime): address post-merge review-comment findings (#5937)

* fix(realtime): address post-merge review-comment findings

A re-audit of every inline review comment on the merged stack surfaced real
issues that the thread-resolutions and prior audits missed. Fixes:

Presence server (#5930 comments):
- connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await.
  Socket.IO clears the room set once the synchronous part of a `disconnecting`
  handler returns, so reading it after `await removeSocketFromAllRooms` saw an
  empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback
  misses live rooms".)
- workflow-room-service: restore the original managers' final unconditional
  room-state wipe via a new `deleteRoom(room)` manager method, so a deleted
  workflow leaves no lingering presence/meta even if a per-socket removal failed
  or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".)

Files (#5932 comments):
- workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal
  (all direct-upload paths: multipart fallback, copilot create, /api/files/upload,
  v1 files — the presigned path already notified). (Cursor: "Creates miss live
  tree fan-out".)
- use-workspace-files-room: clear the pending retry timer on join success; and a
  module-scoped intended-room guard defers the unmount `leave` so a rapid remount
  re-claims the room and skips a stale leave — fixing presence flap + a
  leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount
  churns files presence".)
- workspace-files handler: roll back a partial join (leave room + remove presence)
  in the catch, mirroring the workflow join. (Cursor: "Join failure skips
  membership rollback".)

+2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean,
api-validation + boundaries green.

* fix(files): scope workspace-files leave to a workspace (deferred-leave safety)

Self-review of the deferred-leave guard found a real bug: leave-workspace-files
was not workspace-scoped, so after a workspace switch (A->B) the deferred leave
from A would evict the socket from its new room B. The leave now carries the
workspaceId and the server no-ops if the socket's current files room differs.
Also excludes the leaving socket from the leave broadcast (consistent with
disconnect).

* fix(realtime): close files-room presence leak + validate join payload

Architecture-audit findings:

- S1 (real Redis leak): the files room inherited the shared manager but not the
  workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no
  `disconnecting` event) left its presence entry in the no-TTL room hash forever.
  Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness +
  remove not-live-AND-stale entries, matching the workflow 75min threshold) and
  run it on files join; also filter the join ack through `filterVisiblePresence`
  so a joiner never briefly sees an un-swept ghost.
- S2: validate the client-supplied `workspaceId` on files join before it reaches
  the DB query (matches the /api/workspace-files-changed guard; fails closed).
- N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node
  route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block".

+1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128
realtime tests pass, both apps tsc clean, biome clean.

* fix(realtime): workflow-deletion always notifies + cleans by socket.io membership

Review-round findings on #5937:
- Always emit `workflow-deleted` (was guarded by users.length>0), so a socket
  still in the Socket.IO room after a Redis presence eviction is told the
  workflow is gone before socketsLeave kicks it — the editor no longer keeps
  showing a deleted workflow. (Cursor: "Silent kick skips deletion event".)
- Clean per-socket state for the UNION of live Socket.IO members and
  presence-tracked sockets, so an evicted/late-joined socket's room mapping +
  session are dropped too — not just presence-snapshot sockets. (Greptile: "Room
  deletion leaves reverse state".)
- deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a
  failed wipe isn't reported as a clean deletion; the request surfaces it.
  (Greptile: "Room deletion failures are suppressed".)

The two "deferred leave drops new membership" P1s were already fixed by the
workspace-scoped leave in a prior commit (leave carries { workspaceId }; server
no-ops on mismatch). 128 tests pass, tsc + biome clean.

* refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave

Removes the one non-idiomatic construct (a module-level mutable
`intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace
CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no
remount; list<->detail is sequential) — a dev-StrictMode-only case. The real
cross-workspace race is already handled by the workspace-scoped leave: if B's
join runs first (auto-leaving A), A's leave no-ops because the socket's current
files room is B. Simpler, idiomatic, prod-correct.

* feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941)

Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes.

* feat(files): collaborative document editing — client provider + editor (#5946)

Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch.

* feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957)

* feat(tables): live cell-selection presence — protocol + server + client hook

The realtime spine for Google-Sheets-style table presence (mode A, socket):

- @sim/realtime-protocol/table-presence: centralized wire protocol (events +
  TableCellSelection {anchor, focus, editing} + payloads) so server emits and
  client subscriptions can't drift.
- ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS
  (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName /
  presenceEventName / disconnect cleanup / authorizeRoom all derive automatically.
- apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a
  table-cell-selection relay (mirrors the workflow selection channel), broadcasting
  via roomName(room) since table rooms are namespaced. UserPresence gains a cell
  field threaded through the memory + Redis managers (Lua ARGV[7], null clears).
- Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts.
- use-table-room.ts client hook: joins over the shared socket, tracks the roster
  (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection.

Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits
(last-write-wins via the durable log) are the follow-up PR.

* feat(tables): render live cell-selection presence in the grid

Wires the table presence room into the grid UI:
- Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) —
  renders <PresenceAvatars> in the header and passes remoteSelections +
  emitCellSelection down to the grid.
- Grid emits its local selection: an effect resolves the index-based
  anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an
  editing flag for the active cell) through the throttled emitter.
- RemoteSelectionOverlay: draws each remote viewer's selection in their color
  (getUserColor), a darker fill while editing, and name-on-hover — measured from
  live cell rects in the content wrapper's space (scrolls with the grid),
  hidden when rows are virtualized off-window, pointer-events-none so it never
  blocks cell clicks (hover via pointer hit-test).

* test(tables): cover the table presence handler

Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the
cell-selection relay (asserts it persists via updateUserActivity and broadcasts
on the namespaced roomName, not the bare id) and leave.

* feat(tables): propagate manual cell edits live (last-write-wins)

A manual row edit now appends a lightweight 'edit' event to the durable table
stream; collaborators refetch the row (via the existing debounced rows-invalidate
the job events use) so the winning value shows live. The event carries no value —
peers refetch in their own wire format, so there's no auth-specific value
translation on the wire, and last-write-wins falls out of the DB's committed order
(the Google-Sheets model). Edits that also trigger a dispatch already emit
dispatch/cell events; the debounce coalesces the two.

* refactor(tables): apply /simplify findings

- Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO
  ordering guarantees a peer is in the roster before their selection delta).
- TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste).
- Make TableGrid's presence props required + drop the unused empty-default/guard
  (only table.tsx mounts it, always passing both).
- Drop the unused rowId from the 'edit' event (the handler invalidates all rows).
- Overlay: subscribe scroll/resize/pointer listeners once per scroll element and
  cache the wrapper origin, so incoming deltas re-measure without re-subscribing
  and the pointer hit-test never forces a per-move layout read.
- Server: cache the immutable socket session so a selection delta no longer reads
  it from Redis every time.

* refactor(tables): apply /cleanup findings

- Fix the remote-selection name label contrast: text-white is unreadable on the
  light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a.
- Re-measure via useLayoutEffect so a moving peer selection updates before paint
  (no one-frame position lag).
- Drop 'mothership' from a comment (constitution copy rule).

Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment);
the rest confirmed clean — all state/memos/callbacks/effects are load-bearing,
presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate
granularity is right.

* feat(tables): propagate every table mutation live (edit + schema signals)

Comprehensive live collaboration for all user table mutations, via two value-less
durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged):

- edit (rows refetch): single + batch row create, cell/row update, batch update,
  delete by id/filter, and upsert.
- schema (definition + rows refetch): column add/update/delete, workflow-group
  add/update/delete, table rename, and CSV import (which can add columns).
- Client handles 'schema' by invalidating the table detail (exact) + rows.

Execution paths (column run, cancel-runs) and async jobs (delete/import-async,
job-cancel) already propagate via cell/dispatch/job events — verified applyJob
refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a
deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal
(which would 404).

* refactor(tables): apply comprehensive /cleanup audit findings

Holistic + react-query + comment audits over the whole PR:

- Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's
  querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect,
  crashing every other viewer's page. CSS.escape it, and validate + whitelist the
  untrusted cell payload server-side (shape + 200-char id bound) before it is
  stored/rebroadcast.
- Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl
  that the client discarded (identity comes from the roster). Drop them + the
  getUserSession lookup/cache entirely — the delta is now { socketId, cell }.
- React Query: schema handler also invalidates lists() (parity with the local
  column-mutation set); document that the mutating client self-refetches by design.
- Comment tightenings; biome fixed a stale import order in workspace-files.ts.

* fix(tables): broadcast single-cell selections (focus falls back to anchor)

Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as
a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor
and focus to resolve — so the most common selection never broadcast and clicking even
cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics.

* fix(tables): reviewer + regression + per-LOC audit findings

Cursor review round (5 findings) + regression audit + per-LOC audit:
- Presence roster snapshot now KEEPS the cell we already hold for a known socket, so
  a join/leave broadcast can't revert a fresher CELL_SELECTION delta.
- Reset the selection throttle on table switch (was unmount-only), so a pending
  selection for table A can't flush into table B's room after a switch.
- Metadata writes (column widths, display) use a new lightweight 'metadata' signal
  that refetches only the definition — a resize no longer forces peers to refetch rows.
- Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver
  (a live refetch moves cells without a scroll/resize).
- Document the actor self-refetch create caveat (scrolled multi-page insert) accurately.
- isCellRef narrows to a partial instead of casting to the full type then re-checking;
  drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs.

* fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize

Cursor round on b8f28b04b:
- Remove the 'metadata' signal entirely. The grid seeds columnWidths/pinnedColumns
  from metadata ONCE (metadataSeededRef) and deliberately never re-applies them (to
  avoid clobbering a local in-progress resize), so refetching the definition on a peer
  never surfaced their width/pin change — an ineffective path. Width/pin live-sync needs
  reconciliation that doesn't clobber a local resize; that's a deliberate follow-up, not
  a no-op refetch. Structural changes still propagate via 'schema'.
- Overlay now also observes the content layer with the ResizeObserver, so a column
  resize (which grows the content, not the scroll container) re-measures remote outlines.
- Presence-merge comment now states both sides of the trade-off.

* fix(tables): re-broadcast local selection on (re)join

Cursor Medium: a selection made before the room join completes (or held across a
reconnect) was dropped server-side and never re-sent, so peers didn't see it until
the local user moved it again. Track the current selection in a ref (set on every
emit, cleared on table switch) and re-emit it from handleJoinSuccess once the room is
joined.

* fix(tables): re-broadcast selection when a peer's row change shifts it

End-to-end lifecycle audit (Low-Med): the selection emit resolved the stable
(rowId, columnId) only on selection/editing change, not when a live edit/schema
refetch inserted/deleted/reordered rows. The index-based local selection then sat
on a different logical row than the rowId peers held, so your outline showed on the
old row until you moved. Re-run the emit on rows/displayColumns change and dedup an
unchanged result (also drops the redundant null-on-open emit) so the broadcast stays
consistent with the local highlight.

* fix(tables): schema invalidates run-state/enrichment + guard stale join

Cursor round on cdc8796b8 (2 Medium):
- schema handler used detail exact:true, so it skipped the activeDispatches +
  enrichmentDetails sibling queries the local invalidateTableSchema refreshes via a
  prefix match. After a peer deletes/restructures a workflow group, peers could keep a
  stale running badge or enrichment panel. Now invalidates both siblings too (rows stay
  on the debounce).
- Guard against a stale join stealing the room: a fast table A->B switch could let A's
  async authorize finish after B, leave B, and strand the socket in A. Added a
  per-socket monotonic join generation checked after authorize (mirrors the file-doc
  relay's guard) + a test.

* feat(tables): live column width/pin/order sync

Collaborators now see each other's column resizes, pins, and reorders live —
the last piece of Google-Sheets-style layout parity.

- New lightweight `metadata` durable event kind (distinct from `schema`): only the
  table definition carries UI metadata, so peers refetch the definition alone — no
  rows/run-state refetch. The metadata PUT route now signals it.
- The grid reconciles server metadata against its in-progress gesture: the column
  being actively resized keeps its live local width, and an in-flight column drag
  blocks a reorder apply — so a peer's change never reverts the local action. Each
  field is reference-guarded (React Query structural sharing keeps unchanged
  sub-objects stable), so an unrelated peer change doesn't re-apply the others.

* fix(tables): escalate to schema signal when a reorder scrubs group deps

Independent audit of the metadata-sync commit found a stale-run-state hole: a
columnOrder PUT that moves a column left of a workflow group's leftmost column
makes updateTableMetadata scrub that group's dependencies and write a new schema —
a real structural change. But the route only fired the lightweight 'metadata'
signal (detail-only refetch), so peers' and the actor's activeDispatches /
enrichmentDetails queries stayed stale (a lingering running badge / enrichment
panel) — exactly what the 'schema' handler exists to prevent.

updateTableMetadata now reports whether it scrubbed the schema; the route emits
signalTableSchemaChanged in that case and the light signalTableMetadataChanged
otherwise. Width/pin/plain-reorder stay on the cheap detail-only path.

* feat(realtime): accurate in-file presence + collaborative-caret polish (#5965)

Per-session file-doc presence (avatars count other sessions like the canvas), StrictMode-safe stable Y.Doc (fixes blank-doc on join), flush caret cap + restored hover hit-slop, and three join-lifecycle race fixes unifying file-doc + workspace-files on one intent-tracked monotonic generation model. All findings root-caused with regression tests.

* feat(files): smarter bullet delete/indent and untitled-file title sync (#5971)

* improvement(files): smarter bullet delete/indent, fix empty-nested-bullet heading corruption

Backspace at the start of a list item now outdents a nested item or clears a
top-level item to a paragraph in place instead of deleting the row and jumping
the caret to the previous block; Enter on an empty nested item outdents. Empty
non-trailing top-level items still collapse cleanly since they cannot round-trip
as a lifted paragraph.

Also strips nested empty list-item marker lines on serialize: a nested empty
bullet re-parsed as a Setext heading underline, silently turning its parent line
into an H2 and dropping the bullet. Top-level empty items are preserved.

* feat(files): sync an untitled file's name with its leading heading

While a file is still named untitled(.md), typing a leading heading auto-renames
the file after it (debounced), and renaming the file first seeds a leading H1
from the new name. One-shot: coupling stops once the file has a real name, and
the heading seed always prepends so existing content is never clobbered.

* fix(files): count inline atoms in list-item emptiness, keep multi-block items on Backspace

Addresses review findings on the list Backspace logic:
- Emptiness now uses the caret block's content.size (counts inline images/mentions),
  not textContent, so a bullet holding only a non-text atom is no longer treated as
  empty and deleted.
- An empty first block whose item has sibling blocks removes only that block instead
  of lifting the whole item out of the list.

* fix(files): preserve the untitled to named heading seed across a rename during editor load

The parent captures the file name at mount (before content/session finish loading) and
passes it as the transition baseline, so a rename that lands in the loading window is still
seen as an untitled to named transition and the leading heading seed is not skipped.

* fix(files): drop the name-to-heading seed, keep title sync one-way

Removes the effect that inserted a leading H1 when an untitled file was renamed. On the
collaborative Files page every open client observed the untitled-to-named transition and
inserted into the shared doc, producing duplicate headings; it could also re-insert a heading
a user had just deleted while a rename was in flight. Seeding document content from an async
rename transition is the wrong model on a shared editor. The primary direction — typing a
leading heading renames a still-untitled file — is unaffected (it never mutates the doc).

* fix(files): keep empty lines between paragraphs on reload

The chunked markdown parser (parseMarkdownToDoc) parses each block stripped of the
blank lines between them, so it dropped the empty paragraphs @tiptap/markdown builds
from runs of blank lines — a saved visual blank line silently vanished on the next
load (the settle/reopen re-seed goes through the chunker). The whole-document parser
preserves them, but whether a gap yields an empty paragraph is a global, block-type-
dependent decision (kept between two paragraphs, dropped after a heading), so it can't
be reconstructed block-locally. Route documents with empty-paragraph blank-line spacing
to the whole-document parser for exact fidelity — the same tradeoff NON_CHUNKABLE makes;
ordinary single-blank-line separation still takes the fast chunked path. Adds a suite
asserting chunked output matches the whole-document parser for leading/trailing/between
gaps and around lists/headings.

* fix(files): only auto-name an untitled file when the user can edit

The debounced untitled→filename hook ran on every onUpdate — including the mount-time
seed and for view-only viewers — without checking edit permission, so a read-only user
could schedule a rename they have no permission to make (a spurious, server-rejected
write). Gate the derive-title on editor.isEditable (canEdit + settled + collab-ready,
the same signal the autosave path uses), at both schedule and fire time.

* fix(files): normalize line endings before the empty-paragraph guard; Enter/Backspace symmetry

- markdown-parse: EMPTY_PARAGRAPH_SPACING/NON_CHUNKABLE tested the raw body, but a classic
  \r-only file (blank lines are \r) would miss the \n-anchored guard and still be chunked,
  dropping empties. Normalize line endings once up front so the routing guards, the chunker,
  and the parser all see the same \n. +CRLF/CR test cases.
- keymap: Enter on an empty first block of a multi-block item now removes only that block
  (removeEmptyWrappedBlock) instead of exiting the list, mirroring the Backspace hasSiblingBlocks
  case — the trailing check no longer swallows multi-block items. +test.

* fix(files): editor audit follow-ups (trailing-blank read-only, collab rename, over-strip)

A 4-agent independent audit (UX vs inkeep + SOTA, cleanliness, adversarial correctness)
surfaced these:

- HIGH regression: files ending in a blank line opened READ-ONLY. The empty-paragraph
  routing preserved a TRAILING empty paragraph, but postProcess collapses trailing newlines
  → serialize/parse non-idempotent → isRoundTripSafe flipped the file read-only. A trailing
  empty paragraph can't be serialized stably, so parseMarkdownToDoc now strips trailing empty
  paragraphs and the guard no longer routes on trailing blanks. Interior/leading empties are
  unaffected. +regression tests.
- Medium: the debounced untitled→filename rename fired on remote Yjs edits too, so every peer
  renamed and could rename from a not-yet-synced heading. Gate on isChangeOrigin (local edits
  only; false for non-collab surfaces).
- Medium: stripEmptyListItemLines over-stripped a nested empty item that follows a same-indent
  sibling (a real placeholder the parser keeps). Narrowed to the actual Setext hazard — an empty
  item DIRECTLY under a shallower parent line — matching the function's own docstring intent.
  Probe-verified. +test.
- Low: corrected untitled-title.ts docstring that described a reverse name→heading coupling
  removed during review.

* fix(files): a remote edit must not cancel the local rename debounce

The isChangeOrigin gate cleared the debounce timer BEFORE bailing on a remote update, so
a peer's edit arriving within the 600ms window cancelled the local user's pending rename.
Bail on isChangeOrigin first, before touching the timer; only local edits clear/reschedule it.

* docs(files): correct EMPTY_PARAGRAPH_SPACING rationale after trailing-strip

The stacked trailing-empty-paragraph strip made the older comment overstate a
correctness necessity it no longer owns, mislabel trailing runs of 2+ blanks,
and advertise dead CRLF handling. Reword to match what the code actually does.

---------

Co-authored-by: Waleed Latif <walif6@gmail.com>

* fix(realtime): access-revalidation multi-room safety + cleanup pass

Fix a blocker surfaced by a full cleanup/simplify audit of the branch: the
access-revalidation sweep (staging's workflow-only #5917) treated every entry
in socket.rooms as a workflow id, but the generalized multi-room model puts
namespaced files/tables/file-doc rooms on the same io. It would resolve those
as bogus workflows, get null, and evict files/tables collaborators every ~30s.
collectScanTargets now decodes each room name with parseRoomName and sweeps
only workflow rooms; added a regression test and fixed the now-false TSDoc.

Other audit fixes (all behavior-preserving):
- workflow.ts reuses resolveAvatarUrl (drops db/user/eq imports duplicated
  from avatar.ts)
- PresenceAvatars: mr-1 was baked into the shared component, silently adding a
  margin to the workflow sidebar stack; moved to an optional layout className,
  re-applied on the tables/file-doc header surfaces only
- table DELETE routes only signal collaborators when rows were actually removed
  (matches PUT)
- events.ts definition kind: drop the never-emitted reason:'schema', fix its doc
- event-log: rename buildMemory -> buildEntry (it builds the entry on the Redis
  success path too, not just the memory fallback)
- remove dead resolveWorkspaceIdForRoom export; parallelize per-socket removals
  in handleWorkflowDeletion; gate the table columnIndexById map on remote
  selections; move file-doc module TSDoc off the FileDocOwner interface; fix a
  stale @returns

* fix(realtime,tables): close table-presence race + v1/copilot live-collab gaps

Validated each issue with subagents before implementing the cleanest fix.

- tables LEAVE in-flight-join race (B8): the table handler tracked no current-table
  intent, so an unscoped/same-table leave during an in-flight authorize left the
  socket stranded in the room (present in the roster, broadcasting a ghost until
  disconnect). Mirror workspace-files: a closure-local currentTableId + a leave that
  advances joinGeneration to cancel the racing join. + 3 regression tests.
- v1 + copilot live-collab signal gap (D1): tables edited via the v1 public API or
  Sim/copilot emitted no edit/schema signal, so open collaborators didn't live-update.
  Add the signals at those call sites (add-only, matching the existing route seam) —
  never in the service, so execution writes can't double-emit. Sync-only for copilot
  bulk ops, guarded on affected/deleted count; async job branches stay covered by
  their kind:'job' events; create/delete/get untouched.
- table join read consolidation (B5): sweepStalePresence returns its roster so the
  same-tab dedup reuses it instead of a second getRoomUsers.
- shared authorize slice (B6): extract only the guard-safe authorize->allowed branch
  into resolveRoomJoinAuth, shared by the three room handlers (the full preamble stays
  inline — file-doc's generation capture sits mid-ladder and must not move).
- resize-revert flicker (E3): a peer's value-less metadata event forces a refetch that
  could momentarily revert a just-finished local resize; a pendingWidthWriteRef keeps
  local widths leading until the width PUT settles.
- embedded-mode stray emit (E5): gate emitCellSelection on a bound table id so the
  embedded surface stops broadcasting cell selections the server drops.

* fix(tables): close two copilot live-collab signal gaps + harden presence sweep

Follow-ups from a comprehensive review of the branch:
- copilot batch_update_rows and import_file's inline append branch wrote rows
  but emitted no live-collab signal, so collaborators didn't see those edits
  live (the append's sibling replace branch already signalled). Add the guarded
  signal to both, matching the internal route.
- sweepStalePresence now reads the roster before the fetchSockets liveness probe
  and returns it on a probe failure, so same-tab dedup still runs during a
  transient fetchSockets outage instead of being skipped.
- reword an internal comment off the retired "mothership" term.

* fix(realtime): guard table join commit + rollback against supersession; drop no-op eviction cleanup

Review round on #5991:
- Table join re-checked the generation only once after authorize, then awaited
  leave/sweep/avatar before joining + registering presence. A table switch or
  leave in that window stranded the socket in the wrong room, and the failure
  catch could tear down a newer successful join. Resolve the avatar up-front,
  re-check generation immediately before the membership commit (matching the
  file-doc join), and skip the rollback/error for a superseded join. + a
  post-authorize-window regression test.
- access-revalidation cleanup treated removeUserFromRoom's no-op false as a
  transport failure and re-enqueued a still-connected socket forever. Only retry
  when the socket is still mapped to the room (a healthy null mapping means the
  entry is already gone). Repurposed the expired-mapping test to lock it.

* fix(realtime): guard table join leave-prior against superseding join

Round 2 on #5991: a superseded join's leave-prior could still run — during its
getRoomForSocket await a newer join commits to its room, so currentRoom is that
newer room and the superseded join would leave/remove/broadcast it before the
final guard aborts. Re-check the generation immediately after the lookup await,
before the leave mutation. Extended the post-authorize-window test to assert the
superseded join never tears down the newer join's room.

* fix(realtime): roll back a table join superseded during addUserToRoom

Round 3 on #5991: after the final generation guard, A could join + register
presence while a newer join B commits to its room during addUserToRoom's await
— B's leave-prior can't observe A's half-written entry, so A's late write wins
and strands the socket. Re-check after addUserToRoom and roll back A's own
Socket.IO join + presence (scoped to A's room, never touching B). + a regression
test hanging addUserToRoom mid-commit.

* refactor(realtime): DRY table-join supersession guards; fix stale comment

Cleanliness pass after the review rounds (no behavior change):
- Extract the four identical `joinGeneration !== joinAttempt || socket.disconnected`
  checks into a named `superseded()` helper (the catch keeps its intentionally
  narrower check).
- Remove a stale guard comment that was left stranded above the avatar resolve.
- Document the best-effort rollback catch.

* fix(realtime): file-doc rebind must not drop the current doc or leave a writable ghost

Two Cursor findings on the file-doc client-id ownership rebind:
- On a document switch, the prior room was left BEFORE the ownership check, so a
  CLIENT_ID_IN_USE rejection dropped the socket from the old doc without joining
  the new one (contradicting its own comment). Run the ownership check first, and
  leave the previous doc only once the rebind is guaranteed to succeed.
- Reclaiming a client id removed the stale prior socket from owners + awareness
  only; its socketToRoomName + Socket.IO membership remained, and handleMessage's
  SYNC path gates on socketToRoomName (not owners), so it stayed able to write
  document frames until disconnect. Fully evict the reclaimed socket. + 2 tests.

* refactor(realtime): serialize table join/leave to fix map-corruption at the root

Round 4 on #5991 surfaced a race the generation guards structurally cannot fix:
two concurrent joins for one socket race on the single-valued socket→room map —
a stalled addUserToRoom for table A lands late, clobbers a newer join's map entry
to A, and the rollback then wipes it, stranding the socket (map empty while it
holds table B). Guards protect JS suspension points; they can't stop an in-flight
Redis write from landing late.

Fix per architecture review: serialize this socket's JOIN + LEAVE on a per-socket
promise chain so their multi-step async Redis commits can never interleave —
restoring the atomic-commit property the synchronous sibling handlers get for free.
This DELETES the leave-prior guard and the post-commit rollback (the code that
caused the bug); four generation guards collapse to two identical superseded()
checks (skip a superseded queued op + one pre-commit check). Reworked the
interleaving-specific tests into a fast-switch-skips-superseded test; the leave-
cancels-join tests are unchanged. Local to the tables handler — no shared-infra change.

* fix(realtime): always roll back a failed table join; re-elect file-doc seeder on reclaim

Two review findings:
- Table join: the catch skipped rollback when superseded, but a socket.join that
  landed before addUserToRoom threw leaves the socket in the Socket.IO room with no
  matching socket->room map entry — unreclaimable by any later op (cleanup keys off
  the map). Under serialization the skip is unnecessary (the newer op hasn't
  committed), so always roll back. Simpler + fixes the strand.
- File-doc reclaim: fully evicting the prior socket didn't release the seeder role
  if it held it, so electSeederIfNeeded (which no-ops while seederSocketId is set)
  never re-elected and an unseeded doc stayed empty until the deadline. Clear the
  role on eviction so the join's election picks a new seeder. + 2 regression tests.

* fix(realtime): close revoke-race ghost presence in workflow join

An access-revalidation revoke landing between socket.join and addUserToRoom
socketsLeaves the socket while its presence mapping does not yet exist, so
cleanupEvictedSocket finds nothing to remove and the join then writes presence
for a socket already out of the room — a ghost collaborator until the stale
sweep. Hoist resolveAvatarUrl (the only await in that gap) above the re-auth
check so the whole re-auth -> socket.join -> addUserToRoom section is await-free,
matching the invariant the handler already relies on for the pre-join re-auth.
+ ordering regression test.

* refactor(realtime): serialize workflow join/leave; drop dead room-authz limb

Comprehensive independent audit follow-ups:

- workflow.ts join/leave now use the same opChain + joinGeneration serialization
  as the sibling handlers (tables, file-doc, workspace-files). It was the only
  async presence path left unserialized, so a rapid workflow switch A->B (or a
  leave racing an in-flight join) could strand presence in room A — a ghost
  collaborator still receiving A's operation broadcasts until disconnect. The
  join now aborts a superseded op at start and again right before the membership
  commit, and the catch always rolls back a partial join.
- leave-workflow drops the '&& session' gate: an idle user whose 1h session key
  expired (while the 24h room mapping is still live) can now leave cleanly
  instead of being stranded until disconnect. The room ref alone suffices.
- authorizeRoom: remove the dead ROOM_TYPES.WORKFLOW resolver + its
  getActiveWorkflowContext import. Workflow authorizes through its own path and
  never flows through authorizeRoom; the map now honestly covers only the
  workspace-scoped types (files, file-doc, table).
- Remove unused isSameRoom (zero callers) and a needless useMemo in
  PresenceAvatars (plain derivation, single copy).
- Tests: 4 workflow serialization/leave regressions.

All gates green: tsc (sim/realtime/packages) 0, 204 realtime + 11 protocol
tests, biome, api-validation, boundaries, prune 14/25.

* fix(realtime): align session TTL, harden committed joins from post-success rollback

Per-module comprehensive audit follow-ups:

- redis-manager: SESSION_TTL now tracks SOCKET_ROOMS_TTL (was 1h vs 24h). The room
  set outlived the session, and since getRoomForSocket reads the room set while the
  workflow handlers gate edits/presence on `room && session`, an active-but-idle
  collaborator got wedged into 'session expired' after 1h — sticky until reload
  (activity only EXPIREs the already-gone session; only addUserToRoom re-HSETs it).
  Both keys refresh together, so they now expire together (restores the pre-refactor
  consistency, where both shared one TTL).
- workflow.ts + tables.ts: a 'committed' flag stops the join catch from rolling back
  a genuinely-joined user when a trailing ack/broadcast/metric step fails on a Redis
  blip (a pure getUniqueUserCount log-metric failure could otherwise kick a live
  collaborator after success was already acked).
- workflow.ts: leave-prior now guards `currentRoom.id !== workflowId` (a same-workflow
  re-join no longer leave→re-adds and flickers peers' presence), and the join ack is
  liveness-filtered via filterVisiblePresence — both for parity with the tables handler.
- platform-authz: honest docstring + 400 message for the workspace-scoped-only
  authorizeRoom map (workflow authorizes via its own path).
- caret-presence: corrected an over-stated batching comment.
- +1 workflow regression test (post-success failure keeps the user joined).

Gates: tsc (sim/realtime/packages) 0, 205 realtime + 11 protocol tests, biome,
api-validation, boundaries, prune.

* fix(realtime): narrow join commit-guard to post-success; skip empty presence broadcast

Two Cursor findings on the prior audit-fix commit:

- The 'committed' guard in join-workflow/join-table was too broad: a failure
  BETWEEN the membership commit and the success ack (e.g. getWorkflowState) hit
  'if (committed) return' and emitted neither success nor error, hanging the
  client while it sat in the room. Replaced with the narrower shape: only the
  purely-decorative post-success steps (peer broadcast + log-metric) are wrapped
  best-effort; anything before the success ack still rolls back and surfaces a
  retryable error, so the client retries instead of hanging — while the original
  goal (a benign broadcast/metric blip never kicking a live, acked user) holds.
- broadcastPresenceUpdate read the roster via getRoomUsers, which swallows a Redis
  transport error to []. On a disconnect broadcast that emitted an empty roster and
  cleared every remaining collaborator's presence until the next healthy update.
  Split out a throwing readRoomUsers; broadcastPresenceUpdate now skips the
  broadcast on a read failure (getRoomUsers keeps its swallow contract).
- Tests: pre-success failure rolls back + retryable error (no hang); post-success
  failure keeps the user joined.

Gates: realtime tsc 0, 206 realtime tests, biome, boundaries, prune.

* fix(realtime): harden seeder recovery, join-generation, and misc robustness

Final line-by-line audit follow-ups (all LOW/MED, no P0/P1):

- file-doc: a sole client whose seed FETCH fails was added to triedSeeders,
  re-election found nobody, and the document stayed permanently empty until
  reload. Re-offer seeding a bounded number of rounds (MAX_SEED_ROUNDS) before
  giving up. Also bound clientId to a non-negative integer (it is an ownership key).
- tables + workspace-files: validate the room id BEFORE advancing joinGeneration,
  so a malformed/rejected join can't cancel a legitimate in-flight join.
- workflow + tables + workspace-files: suppress the client-facing join error when
  the op was already superseded (a retryable error naming the abandoned room could
  make a client re-join and cancel its newer join). The rollback still runs.
- redis-manager: set isConnected=true only after scriptLoad succeeds (and reset it
  on failure) so isReady() can't report ready while the Lua SHAs are null.
- connection: apply the presence-bearing filter to the manager-removed set too
  (symmetry with the fallback path).
- http.ts: validate workflowId on the four workflow endpoints (matching the files one).
- client: clear a pending join-retry timer before rescheduling (reconnect churn no
  longer orphans a stray extra join); clear caret fade timers on plugin destroy;
  seed-effect cleanup reports NOT-ready (safe direction).
- Tests: bounded seeder recovery, cell-selection strip-junk, TABLE round-trip,
  presenceEventName.

Gates: tsc (sim/realtime/packages) 0, 208 realtime + 12 protocol tests, biome,
api-validation, boundaries, prune.

* fix(realtime): gate isReady() on loaded script SHAs, not just connection

Follow-up to the prior isConnected change, which was incomplete: the redis client's
'ready' event flips isConnected=true on connect — before initialize() loads the Lua
scripts — so a bare isConnected check reports ready while removeUserFromRoom /
updateUserActivity would silently no-op on a null SHA. Gate isReady() on the SHAs
too, so the POST endpoints return a retryable 503 during that startup window instead
of proceeding against unloaded scripts. Standard readiness-probe discipline.

* fix(files): offline read-only fallback when the realtime doc never syncs

When the realtime server is unreachable (offline, server down, socket never connects),
the collaborative editor would sit blank and read-only forever — content only arrives
via provider sync events that never fire. Add a bounded connect-deadline to the Yjs
provider: if no first sync lands within CONNECT_DEADLINE_MS, latch fatal and emit a
synthetic non-retryable join-error — the exact path a real fatal rejection already uses,
which seeds the file's stored content read-only. Latching fatal also stops a late
reconnect from syncing server state in and merge-duplicating the locally-seeded content
(the documented Yjs 'non-empty doc ignores initial value' gotcha).

Deliberately NOT adding durable Yjs snapshot persistence / server-side seeding: TipTap
can't run the markdown->Yjs conversion server-side (Collaboration extension errors under
jsdom), and a durable binary snapshot would create a dual source of truth with the
markdown file (edited by copilot / PUT / download). The client-seeder + bounded
re-election is the correct architecture for a markdown-is-truth model; this closes its
one real user-facing gap without persistence, a migration, or dual-truth.

Timer cleared on first sync, on a real fatal rejection, and on destroy. +2 tests.

Gates: sim tsc 0, 496 editor tests, biome. Needs live offline->reconnect verification.

* fix(realtime): validate workflow join id before generation bump; scope file-doc join rollback to its target

Two Cursor findings:
- join-workflow bumped joinGeneration before validating workflowId (unlike tables /
  workspace-files, which I'd already fixed). A malformed/empty join could advance the
  counter and cancel a legitimate in-flight workflow switch. Validate the id first. +test.
- The file-doc join catch called cleanupFileDocForSocket unconditionally, which keys off
  socketToRoomName. During a document SWITCH that fails before rebinding (e.g. a throw in
  client-id reclaim), that binding still points at the socket's PRIOR, valid document —
  so the rollback tore down a document the socket was validly in. Only run that cleanup
  when the binding already points at THIS join's target; otherwise the socket never
  registered as an owner here and the only leftover is a freshly-created empty room,
  dropped by destroyRoomIfIdle.

Gates: realtime tsc 0, 209 tests, biome, boundaries, prune.

* fix(files): drop late sync frames once fatal; file-doc join error suppression + retry-budget reset

Final safety-audit findings:
- CRITICAL: FileDocProvider.handleMessage had no fatal guard. After the connect
  deadline latched fatal and the editor fell back to a read-only local seed, a
  late SyncStep2 (slow server / flaky network / deploy) was still applied — merging
  server state into the seeded doc (content duplication) and flipping synced=true,
  which un-gated autosave and would persist the duplicate to the real file. fatal
  guarded (re)join but not inbound sync. Now handleMessage returns early when fatal.
  +test (late SyncStep2 after the deadline is ignored, doc stays empty + gated).
- file-doc join catch now suppresses the client-facing error when superseded
  (matches workflow/tables/workspace-files) so a retryable error for an abandoned
  file can't make a client re-join and cancel the newer one.
- table/workspace-files room hooks reset the retry budget on (re)connect so a prior
  full exhaustion doesn't block retries after a reconnect.
- presence-visibility: corrected a stale TTL comment.

Gates: tsc (sim/realtime) 0, 209 realtime + collab/hooks suites, biome, boundaries, prune.

* feat(collab-doc): server-authoritative Yjs seeding (#6008)

Server-authoritative Yjs seeding for collaborative file documents: the realtime relay
fetches a Yjs seed built from the file's markdown (via the shared TipTap engine) and
applies it once per room, replacing the client-seeder election/handshake entirely.

- DOM-free markdown<->Yjs conversion core (markdownToYDoc / yDocToMarkdown /
  applyMarkdownToYDoc) reusing the client markdown engine for parity by construction
- Internal x-api-key seed endpoint + realtime fetch; single attempt bounded under the
  client readiness deadline, guard-release for join-driven retry, read-only fallback on
  persistent failure
- Client readiness gate = synced && server seed flag; jsdom wired for the Next standalone
  build (serverExternalPackages + outputFileTracingIncludes)

Foundation only — copilot-into-doc + markdown projection + durable persistence are Stage C.

* feat(collab-doc): Sim merge endpoint for copilot-into-doc (Stage C foundation)

buildFileDocMergeUpdate(docState, markdown) computes the minimal Yjs diff that turns a live
document into target markdown, via applyMarkdownToYDoc (a real updateYFragment diff, not a
replace) — so a copilot rewrite merges with concurrent user edits instead of clobbering them.
Exposed over the internal x-api-key /api/internal/file-doc/merge endpoint the realtime relay
will call: the relay owns the doc, the app owns the conversion engine, so the relay ships the
current state and applies the returned diff. Tested incl. concurrent-edit no-clobber.

* feat(collab-doc): realtime apply-edit — merge copilot markdown into a live doc

The relay can now stream a copilot edit into open editors: applyMarkdownToLiveFileDoc finds
the seeded live room, ships its state to the app's /merge endpoint for a minimal CRDT diff,
applies it (relaying to every editor, reconciled with concurrent user edits), and reports
'no-live-room' so the caller falls back to a direct file write when nothing is open.

- Generalize the realtime->app request module (file-doc-seed.ts -> file-doc-app.ts) with a
  shared POST helper + fetchFileDocSeed/fetchFileDocMerge
- POST /api/file-doc/apply-edit on the internal x-api-key HTTP surface, returning { applied }
- Tests for the seeded-room merge relay and the no-live-room fallback

* feat(collab-doc): stream copilot edits into open editors (Stage C)

edit_content now, after its durable file write, best-effort merges the same markdown into the
file's live collaborative document (markdown files only). If a collaborator has it open, the
edit streams into their editor as a CRDT merge — reconciled with their concurrent typing —
instead of the file silently changing under them; the editor's existing autosave mirrors the
merged doc back to the file. No-op when nothing is open. Never blocks or fails the edit.

* fix(collab-doc): strip frontmatter on merge; gate live-merge to markdown

- buildFileDocMergeUpdate now strips YAML frontmatter (splitFrontmatter().body) exactly as
  the seed does. Copilot passes full-file content, so without this the frontmatter merged
  into the doc as editor content and autosave wrote it back over the file (corruption).
- Gate the live-doc merge on isMarkdownFileName (new server-safe helper) instead of the
  over-broad !isDoc, so code/text edits don't pay the realtime round-trip for a format the
  collaborative editor never renders.

* fix(collab-doc): make frontmatter collaborative so a merge can't revert it

The editor re-attaches its open-time frontmatter on every autosave, so the Stage C merge
(which triggers an autosave with no user action) could write stale YAML back over a copilot
frontmatter change — silently dropping it.

Carry the file's frontmatter in the doc's config map instead of locking it at open: the seed
stores it, the merge updates it (only when it actually changed, preserving the no-op diff),
and the editor re-attaches THAT value on save — falling back to the locked copy before the
seed lands and for non-collaborative docs. A server-side frontmatter change is now reflected
rather than reverted. New FILE_DOC_SEED.frontmatterKey; seed/merge tests cover it.

* fix(collab-doc): re-sync draft on a frontmatter-only merge

A server edit that changes only the frontmatter updates the config map but not the body
fragment, so TipTap's onUpdate never fires — the autosave draft kept the stale open-time
frontmatter, and an explicit save could revert the live change. Observe the config map and,
on a frontmatter-only change, re-attach the new frontmatter to the current body and push a
fresh draft (guarded on a synced body so it never races the seed's own onUpdate).

* fix(collab-doc): order the apply-edit/merge timeouts (outer > inner)

The Sim->realtime apply-edit timeout (4s) was shorter than the nested realtime->Sim merge
timeout (8s), so the outer call could abort while the relay was still merging — the relay
then applied the merge after edit_content had returned, racing a follow-on edit.

Split the shared realtime->app timeout: the seed keeps 8s (it reads a cold blob), the merge
gets a tight 3s (it is a pure conversion, no I/O), and the outer apply-edit is raised to 6s
so it always outlives the inner merge. Cross-referenced in comments to prevent drift.

* fix(collab-doc): address deep-audit findings (jsdom trace, timeouts, gate, races)

A 4-agent LOC audit + precedent research (TipTap/Hocuspocus/Yjs docs confirm the core
patterns are idiomatic) surfaced these real issues:

- The /merge route also lazy-requires jsdom but was missing its outputFileTracingIncludes
  entry — a Docker/standalone build would 500 with MODULE_NOT_FOUND. Add it.
- The four conversion timeouts encoded an ordering invariant (merge<applyEdit, seed<readiness)
  living only in prose across two apps. Hoist to a shared FILE_DOC_TIMEOUTS in
  @sim/realtime-protocol with a test asserting the ordering; both apps import it.
- The copilot live-merge gate used an extension-only check, but the editor treats any
  text/markdown-MIME file as markdown. Replace isMarkdownFileName with a MIME-aware
  isMarkdownFile mirroring the client, so those files stream too.
- applyMarkdownToLiveFileDoc had no per-file serialization; overlapping merges could each
  diff the same stale snapshot and apply out of order. Serialize per file via a promise chain.
- Editor hardcoded 'config'/'initialContentLoaded' instead of FILE_DOC_SEED constants (drift).
- Add .max() bounds to the merge contract body; document the best-effort-merge failure window
  honestly (open editor + merge failure can drop a copilot edit until reload — closed by the
  deferred durable-doc work).

* feat(collab-doc): multi-replica shared Yjs backend + server-side markdown persistence

Make collaborative file-doc editing correct across multiple ECS tasks (the
per-process Y.Doc previously assumed one replica per file).

- Shared Yjs backend over Redis Streams (apps/realtime file-doc-store): each
  file's stream is the ordered, replayable log of updates; a multiplexed XREAD
  tailer converges every task's in-memory doc. Coordinated single-seeder
  election (SET NX + empty-stream recheck) fixes split-brain seeding.
- Doc-sync fans out to local clients + the stream; awareness stays on the
  Socket.IO adapter. Snapshot+XTRIM compaction trims only integrated entries.
- Server-side persistence: project the live doc back to markdown via a new
  /api/internal/file-doc/persist endpoint, debounced during editing and flushed
  on last-disconnect, from the authoritative stream state. Collaborative editors
  no longer client-autosave, closing the copilot clobber-window.
- Copilot merges apply through the stream (reach the live doc on any task) and
  serialize cross-task via a Redis merge lock.
- Degrades to the original single-replica behavior when REDIS_URL is unset.

* fix(collab-doc): harden distributed locks and durability from review

Address Greptile + Cursor review of the multi-replica backend:

- Merge lock: retry LONGER than the lock TTL (guaranteed acquisition, never
  merges against a shared base while a peer holds the lock) and AWAIT the stream
  write before releasing, so the next task never diffs a stale base.
- Distributed locks (seed/merge/compact) now use ownership tokens + a
  compare-and-delete release (Lua), so a lock that expired and was re-acquired by
  another task is never stolen; acquisition fails CLOSED on Redis error.
- Seed: publish the seed to the stream AWAITED under the lock before releasing,
  so a later seeder's empty-stream fence always sees it (closes the fence's
  publish-after-release gap); TTL kept at the readiness deadline.
- Persist: persist the AUTHORITATIVE stream state even when this task's local doc
  was never seeded, and capture the local fallback synchronously so a
  last-disconnect flush never encodes an already-destroyed doc.
- Publish: retry a transient xAdd failure so a Redis blip can't silently drop an
  edit from the shared log.

* fix(collab-doc): only persist a doc a user actually edited

Cursor review: a copilot durable write landing while a doc is being seeded could
have the stale seed projected back over it on last-disconnect, clobbering the
copilot edit even with no user changes.

Gate server-side persistence on a genuine user edit (socket-origin update): a
seed-only or merge-only doc is never projected back to the file (copilot writes
the file durably itself), so it can't clobber a concurrent external write.

* fix(collab-doc): close review-round race/durability gaps

Address Greptile P1 + Cursor findings:

- Seed publishes to the shared stream AWAITED *before* seeding the local doc, so
  a publish failure leaves the doc unseeded and the stream empty for a clean
  retry rather than serving an unpublished local seed a peer would re-seed over
  (split-brain).
- flushPersist falls back to the synchronously-captured local snapshot when
  getStreamState throws (not only when it returns null), so a transient Redis
  read no longer drops the final durable write as the room is torn down.
- streamHasContent fails CLOSED (returns true on xLen error): a Redis blip can no
  longer let the seed fence pass and double-seed.
- Client collabReady initializes from the collaborative prop, so a collaborative
  editor never has a mount-window where client autosave could clobber the server
  write.

* fix(collab-doc): unstick seed retry and persist tailed edits

Cursor review:

- ensureServerSeed clears serverSeedStarted when it aborts at the streamHasContent
  fence, so a fail-closed Redis xLen (or a genuine peer-seed) no longer strands the
  room unseeded with no retry.
- Persistence dirty-tracking now marks a doc edited on any post-seed update,
  including a peer's edit relayed via the tailer (REDIS_ORIGIN), tracked via a
  seededObserved flag. The last task to leave persists real edits even if it only
  tailed them; the seed transition itself is still never counted, so a
  seeded-but-unedited doc is never projected back over the file.

* fix(collab-doc): match client markdown post-processing on server persist

Cursor review: yDocToFileMarkdown serialized the body with yDocToMarkdown only,
but the editor save path runs postProcessSerializedMarkdown before applyFrontmatter.
Server persist could therefore write markdown differing from a client save (empty
list markers, callout un-escaping) — spurious blob churn / round-trip drift despite
the byte-identical claim.

Apply postProcessSerializedMarkdown in yDocToFileMarkdown so a server persist is
byte-identical to a client save and the client's dirty-check baseline. Add a
regression test guarding the composition.

* fix(collab-doc): close durability gaps at deploy boundaries + audit polish

From a comprehensive from-scratch audit (correctness, SOTA, cleanliness,
feature-completeness):

- Persist max-wait: a continuous edit burst kept resetting the 5s debounce and
  never persisted; cap it so a burst flushes at least every 20s, bounding
  unpersisted edits.
- Graceful-shutdown flush: flushAllFileDocRooms awaited in shutdown so a rolling
  deploy / scale-in secures open edited rooms to durable markdown before exit,
  instead of relying on the stream + a surviving task.
- Compacted-snapshot catch-up now marks the doc edited (REDIS_SNAPSHOT_ORIGIN): a
  snapshot folds seed+edits into one frame, so a task catching up purely from it
  no longer treats real edits as an unedited seed and skips persisting.
- Polish: delete dead __setFileDocStoreForTest; bounded retry loop; rename
  acquirePersistSlot -> tryClaimPersistWindow with accurate docs; tailer
  object-identity guard; fix stale comments (seed route, edit-content autosave).

* improvement(realtime): post-review fixes for collab dirty-state + relay lifecycle

- collab editor no longer latches a spurious 'Unsaved changes' prompt: report
  dirty only when the client owns durability (canAutosave), since in a
  collaborative session the relay persists the doc server-side
- guard shutdown against a double SIGINT/SIGTERM running teardown twice
- disconnect local sockets before httpServer.close so shutdown exits gracefully
  instead of hitting the forced-exit timer (local-only, deploy-safe)
- return 400 (not silent 200) on an invalid workspaceId in the files-changed fanout
- clear a pending join-retry timer on reconnect so it can't fire a duplicate join

* fix(realtime): make collab-doc seeding atomic to close split-brain window

An adversarial concurrency audit found a split-brain double-seed vector: the
seed used an advisory SET NX PX lock + a SEPARATE xLen fence + an unconditional
xAdd (a non-atomic check-then-append). If the seed lock's TTL expired mid-seed
(a >4s stall after the 8s seed fetch), a second task could acquire the freed
lock over a still-empty stream, both fences read empty, and both append seeds
with distinct Yjs client ids -> duplicated document content.

- add an atomic SEED_IF_EMPTY_SCRIPT (append-iff-empty in one Redis step) +
  store.seedIfEmpty(); the emptiness check and append are now inseparable, so
  two tasks racing (even both past an expired lock) can never both seed
- ensureServerSeed uses seedIfEmpty instead of streamHasContent-fence +
  publishAndWait; the seed lock is now purely an efficiency optimization
  (avoid a duplicate fetch), not a correctness dependency
- fix the misleading comment that claimed a copilot merge is not counted as an
  edit in the multi-replica path (it round-trips as REDIS_ORIGIN and does count;
  a safe idempotent over-persist, never a lost edit)
- add interleaving tests: seedIfEmpty atomicity/fence, the split-brain
  regression under an expired lock, a peer edit during attach catch-up, and
  concurrent two-task compaction

* docs(realtime): align seed comments with the atomic-append correctness model

Greptile flagged lingering doc drift: the module + shouldSeed comments still
credited the seed lock + empty-stream check as the split-brain fix. Reframe them
so the atomic seedIfEmpty is the exactly-once guarantee and shouldSeed is an
efficiency gate only.

* feat(realtime): live workspace tables list, sharing one invalidation-room impl (#6053)

* feat(realtime): live workspace tables list, sharing one invalidation-room impl

Bring the tables list to parity with the files list: a create/rename/move/delete/
restore now propagates to every viewer live instead of waiting out the 30s
staleTime. Following the files pattern, but factoring the two into one shared
implementation rather than copy-pasting.

- add ROOM_TYPES.WORKSPACE_TABLES + its authz resolver (workspace-id-addressed,
  reuses the workspace resolver like workspace-files)
- extract setupWorkspaceInvalidationRoom (server) and useWorkspaceInvalidationRoom
  (client) — the presence-free, workspace-scoped live-list room; files and tables
  now both bind to it, so they can never drift. Event/room names derive from the
  room type. Replaces the standalone workspace-files handler + hook
- notifyWorkspaceTablesChanged fanout fired from the table service (createTable,
  renameTable, moveTableToFolder, deleteTable, restoreTable) so it covers both the
  HTTP routes AND copilot, which call the service directly
- relay /api/workspace-tables-changed endpoint; wire the hook into the tables page
- consolidate the handler test into one suite run against both room types

* feat(realtime): live tables list also covers table-folder mutations

Fold in the follow-up: a table folder create/rename/move/delete/restore now
propagates to the tables list live too, so the browser is fully consistent.

- generic notifyFolderResourceChanged(resourceType, workspaceId) dispatches the
  workspace live-list signal by resource type (a map, not a special-case if), so
  file/knowledge_base/workflow are no-ops today and gain liveness by adding a map
  entry when they adopt an invalidation room
- fired from the shared folder lifecycle (createFolder/updateFolder/deleteFolder/
  restoreFolder), covering routes AND copilot
- the tables room hook now invalidates the table folders query too, not just the
  tables list, since the page renders both

* fix(realtime): skip per-table live-list notify during a folder cascade

A folder delete/restore already fires one folder-level notifyFolderResourceChanged
for the whole subtree, but the cascade also calls deleteTable/restoreTable per
table — each awaiting its own notifyWorkspaceTablesChanged. A folder with many
tables would run N+1 sequential relay calls (each bounded by NOTIFY_TIMEOUT_MS),
blocking the mutation. Add a skipNotify option the cascade passes so only the one
folder-level notify fires.

* feat(collab-doc): Hocuspocus binary persistence + Next 16 seed/persist fixes (#6059)

* fix(collab-doc): make server-side seed conversion work under Next 16 / Turbopack

Opening a file left both collaborators read-only and stalled ~12s: the server-side
seed (markdown -> Yjs, run through the headless editor engine) was failing, so the
doc never seeded and the editor never left its readiness gate. Two root causes,
both latent until a real build/runtime (typecheck + unit tests don't exercise
either), surfaced by the Next 16 upgrade:

1. Build boundary: the server seed route imported the shared editor schema
   (`createMarkdownContentExtensions`), which pulled in the React node-view
   components (`useEffect`) -> 'client component in a Server Component'. Split each
   node's React-free schema into its own `*-schema.ts` (code-block, image,
   raw-markdown-snippet); the client editor still injects the React node views via
   the existing `nodeViews` param, unchanged.

2. Runtime DOM: the converter installs a jsdom `window` on `globalThis`, but
   Turbopack's server bundle gives bundled `@tiptap/core` a `window` that does NOT
   read `globalThis`, so `elementFromString` threw 'no window object available'.
   Externalize the `@tiptap/*` packages the converter uses (native Node require, so
   their `window` reads the real global) and fix the converter's DOM guard to gate
   on `window` (what TipTap checks) with no sticky flag.

Verified: seed route returns 200 with the Yjs update; 514 collab-doc + editor tests
pass; schema byte-identical after the split.

* feat(collab-doc): persist the Yjs binary and load it on cold-start (Hocuspocus pattern)

Adopt the industry-standard Hocuspocus store/load-document pattern so a cold room
open loads the file's last-persisted Yjs binary directly instead of re-converting
markdown -> Yjs on every open. Rebuilding the CRDT from markdown on each connect is
the exact anti-pattern Tiptap/Yjs warn against (fresh client ids -> duplicated
content); it also forced the fragile server-side headless-editor conversion on every
open. Now conversion runs only on a genuine first open or an external markdown edit.

- new table workspace_file_collab_state(file_id PK->workspace_files cascade,
  doc_state bytea, source_hash, updated_at): the Yjs binary + a hash of the markdown
  it was derived from (bounded <=~1MB by the 256KB round-trip gate). Mirrors
  Hocuspocus's extension-database (binary in a DB column). Migration 0275.
- persist upserts the binary (tagged with the exact markdown just written)
- cold-start seed returns the cached binary when its source_hash matches the file's
  current markdown; otherwise converts (and the next persist refreshes the cache)
- also externalize yjs / y-protocols / lib0 alongside @tiptap: bundling loaded a
  second yjs copy, so @tiptap/y-tiptap's 'instanceof Y.XmlElement' failed on
  app-created nodes ('Unexpected case') during Yjs -> markdown

Verified end-to-end: seed -> persist -> seed returns the exact persisted binary (a
cache hit, no re-conversion). 18 collab-doc + 51 realtime file-doc tests pass.

* fix(collab-doc): best-effort cache read + drop dead barrel

- seed: a cache-read failure (transient DB error, not-yet-migrated cache table)
  no longer aborts a cold room open — the durable markdown is already in hand, so
  fall through to conversion. Symmetric with persist's best-effort cache write.
  Addresses the Cursor Bugbot finding on the read/write asymmetry.
- remove the collab-doc index.ts barrel: nothing imported it (every consumer uses
  direct ./seed / ./merge / ./converter imports), so it was dead re-export surface.
  De-export COLLAB_DOC_FIELD accordingly — it is used only inside converter.ts.

* fix(collab-doc): stream every external file write into open editors, not just edit_content (#6070)

* fix(collab-doc): stream every external file write into open editors, not just edit_content

A copilot/mothership edit to an open markdown file did not appear live in another
user's editor: the live-doc merge bridge (mergeEditIntoLiveFileDoc) was wired into the
edit_content tool ONLY. Every other server-side write — the file tool
(/api/tools/file/manage), function_execute (/api/function/execute via
writeWorkspaceFileByPath), create_file overwrite, and the PUT /content route — went
straight to updateWorkspaceFileContent and skipped the merge, so the durable file
changed but the open editor never updated. Confirmed from live logs (the mothership
'Prepend sentence' ran read + file + function_execute — zero apply-edit calls) and
Redis (the prepended text was absent from the doc stream).

Centralize the merge at the one chokepoint every external writer shares:
- updateWorkspaceFileContent gains an opt-out "syncLiveDoc" (default on) and, after
  the durable write, merges markdown writes into any open collaborative doc (best-effort;
  no-op when nobody has it open). Any current OR future writer is covered automatically.
- persist.ts opts out (syncLiveDoc:false) — it IS the doc→markdown projection, so merging
  it back would be a persist→merge→persist self-loop.
- create_file opts its empty shell out (real content arrives via a later write) so an open
  editor never flickers to empty on overwrite; threaded through writeWorkspaceFileByPath.
- edit_content drops its now-redundant explicit merge call (the chokepoint handles it).
- binary writers (image/video/audio/ffmpeg/download) are naturally excluded — the merge is
  gated to markdown, the only format the collaborative editor renders.

Also bump the api-validation route baseline 994→996 to match the true route count already
on this branch (pre-existing ratchet drift from an earlier merge; NOT added by this PR).

* fix(collab-doc): defer setEditable out of the render phase (flushSync warning)

The collab editability-reapply effect called editor.setEditable synchronously. In collab
mode isEditable flips from readiness (synced + seeded), which is driven by a Yjs
config.observe firing synchronously inside Y.applyUpdate — so the effect can run while React
is mid-render. TipTap's React binding commits setEditable's transaction with flushSync, which
throws "flushSync was called from inside a lifecycle method. React cannot flush when React is
already rendering." Defer the setEditable to a microtask (runs right after the current commit,
before paint), guarding against a destroyed editor or a stale value before it fires.

Only the collab path (this effect) hit the warning; the streaming/settle effect's setEditable
calls run on the non-collab path where isEditable isn't driven by a mid-render Yjs observer.

* fix(rich-markdown-editor): defer non-collab settle/stream mutations off the render phase (flushSync) (#6073)

* fix(rich-markdown-editor): defer non-collab settle/stream mutations off the render phase (flushSync)

The non-collaborative streaming/settle effect called editor.setContent / setEditable /
setTextSelection / focus directly in the effect body. setContent mounts the custom node views
synchronously through the @tiptap/react flushSync path (tiptap#3764), so when this effect runs
while React is mid-render it throws "flushSync was called from inside a lifecycle method." This
is the second flushSync source (the collab editability effect was the first, fixed separately);
it fires on the agent-streaming-into-a-non-collab-editor surface.

Defer the effect-body view mutations to a microtask via a small runOffRender helper (runs right
after the current commit, before paint; no-ops if the editor was torn down). The settle block is
deferred as ONE microtask so setContent -> collapse selection -> setEditable -> focus keep their
order. The streaming rAF tick is left untouched — it already runs off-render, so it keeps writing
content directly. queueMicrotask is TipTap's own documented remedy for this warning.

497 rich-markdown-editor tests (incl. stream-settle-selection) pass; tsc + lint + api-validation
+ boundary + prune green. Needs a live check: stream an agent into a non-collab markdown file and
confirm it still renders smoothly.

* chore(rich-markdown-editor): trim verbose flushSync-defer comments

* fix(rich-markdown-editor): drop superseded settle/stream microtasks via a run token

runOffRender previously only guarded editor.isDestroyed, so if React ran the next reconcile
pass (a newer stream or settle) before a queued microtask flushed, the stale microtask could
still apply setContent/setEditable/setTextSelection over the newer state. Tag each effect run
with an incrementing token; a deferred mutation applies only when its run is still the latest
(and the editor is alive). A run token fits this effect's several early-return exits better
than a per-exit cleanup flag. Addresses Greptile/Cursor review.

* fix(rich-markdown-editor): never drop the settle selection-collapse under a superseded run

The run token drops a superseded settle's microtask, but the settle had already flipped its state
flags synchronously — so a pre-empting steady-sync run took the non-settle path and never collapsed
the selection, leaving a post-stream select-all painting the leaf-in-selection decoration. Track the
collapse as a debt (pendingCollapseRef): whichever deferred run ultimately applies — settle or the
steady-sync path — clears it, so the collapse runs exactly once on the latest content. Addresses the
Cursor review finding.

* feat(tables): show live cell-selection carets in the embedded chat panel (#6081)

The table cell-selection presence room was joined only on the dedicated /tables/[id] page
(useTableRoom was passed an empty id in embedded mode). Join it in embedded too, so the
mothership chat resource panel shows collaborators' live cell selections and broadcasts the
local one. tableId is already resolved from props in embedded (the data event stream already
uses it un-gated), and authz runs on join, so this is safe. Avatars are unaffected — they
render only in the !embedded Resource.Header, so the panel gets carets without avatars.

* feat(collab-doc): If-Match optimistic concurrency so persist never clobbers an out-of-band edit (#6085)

* feat(collab-doc): optimistic-concurrency guard so persist never clobbers an out-of-band edit

The relay projected the live Yjs doc back to durable markdown unconditionally (last-write-wins), so
a persist already in flight when an external write landed could overwrite it. Add RFC 7232 If-Match
optimistic concurrency end to end, reconciling through the CRDT (never rejecting user work):

- updateWorkspaceFileContent gains an expectedUpdatedAt guard: the write commits only if the file is
  still at that version (checked against the SELECT ... FOR UPDATE-locked row, so it is atomic with
  the write), else it throws the new ContentVersionConflictError without clobbering.
- persistFileDoc takes expectedVersion and returns a discriminated result (persisted | missing |
  conflict). On conflict it returns the current durable content + version instead of writing.
- The relay tracks the durable version its live doc is synced to — set on seed, advanced when a
  durable write is merged in (apply-edit carries the version), and on each successful persist. It is
  held cluster-wide in Redis (filedoc:syncver:{name}) so whichever task persists reads the same
  version, with the per-room value as the single-pod fallback.
- flushPersist sends that version as If-Match. On a conflict it merges the current durable content
  into the live doc (so the out-of-band edit AND the live edits converge) and retries (bounded), so
  even a last-leave flush racing an external write persists the reconciled result rather than losing
  the session's edits.

Threads the version through the seed + persist contracts and the apply-edit payload. No schema change
(reuses workspace_files.updatedAt as the version token). Tests: app-side CAS (match writes, mismatch
throws + cleans up the orphan upload), relay conflict handled gracefully without clobber/loop; 236
realtime + 76 sim collab/uploads tests, tsc x2, lint, api-validation, boundaries, prune all green.

* chore(collab-doc): heartbeat-refresh the synced-version key TTL alongside its stream

Keep filedoc:syncver:{name} alive as long as the room's stream (it was only re-set on
seed/merge/persist), so an open-but-idle doc's persist If-Match token can't expire and force a
needless reconcile.

* fix(collab-doc): stop persist-conflict retries when there is no live doc to reconcile

On an If-Match conflict with no live doc to reconcile into (last collaborator gone, no shared
stream), applyMarkdownToLiveFileDoc returns no-live-room; re-projecting the same pre-teardown
snapshot would only re-conflict, so break the retry loop immediately and leave the out-of-band
(durable) content authoritative — the intended conflict policy. Addresses Greptile review.

* fix(collab-doc): close three optimistic-concurrency edge cases from review

- Single-pod persist retry projected the pre-reconcile snapshot (captureState always returned the
  initial localState), while the synced version had been advanced by the reconcile — so the If-Match
  could pass and clobber the reconciled edit. captureState now re-reads the live doc on each attempt
  (falling back to the pre-teardown snapshot only once the room is gone).
- The synced version was recorded from this task's own seed FETCH before knowing whether this task's
  seed actually won; a peer winning with a different version could leave a newer token than the stream
  content. Record it only inside the didSeed branch (the task whose seed won); peer-seeded tasks read
  the winner's cluster value.
- Persist wrote UNCONDITIONALLY when no version was available (relay version momentarily missing), which
  could clobber non-empty durable content. It now returns conflict for a non-empty file with no version
  (reconcile/retry once the version is re-established); an empty file's first write stays unconditional.

* fix(collab-doc): defer (not reconcile) on missing version, and use the freshest version token

- Missing-version persist now returns 'deferred' instead of 'conflict'. A missing version token (a
  Redis blip on a peer-seeded task) is NOT a genuine out-of-band change, so triggering a reconcile
  would wipe live edits (incoming-wins) even though nothing changed durably. Deferred means: don't
  write, don't reconcile — leave the edits in the stream and let a later persist write them once the
  version is re-established.
- currentVersion now takes the MAX of the cluster (Redis) and local room versions rather than always
  preferring Redis, so a lagged/failed fire-and-forget Redis set can't shadow a newer local value and
  cause spurious If-Match conflicts. Versions are monotonic epoch-ms, so the larger is the later sync.

* fix(collab-doc): make persist If-Match teardown-race-immune and recover missing version on final flush

Close two last-leave concurrency holes Cursor flagged:

- Thread the reconciled version LOCALLY through the persist retry loop. After a
  conflict+reconcile the correct next If-Match is exactly result.version, so carry
  it in a local var instead of re-deriving from room.syncedVersion/Redis. On a
  last-leave flush destroyRoomIfIdle removes the room from the map before the async
  flush finishes, so mergeMarkdownIntoRoom's recordVersion can no longer update
  room.syncedVersion — threading makes each retry's precondition correct by
  construction, immune to that dropped mutation and to a best-effort Redis re-read.

- Cache the resolved version back into room.syncedVersion in currentVersion() so a
  peer-seeded/tail-only task (which never sets it locally) or a later transient
  Redis read failure still resolves it from the last value seen (monotonic max,
  never regresses).

- On a FINAL flush, briefly retry resolving the If-Match when the version read
  momentarily fails, rather than deferring and stranding the session's edits in the
  TTL'd stream — the version is cluster-wide and heartbeat-refreshed.

* fix(collab-doc): stamp cluster sync version the moment the seed wins, before the liveness guard

The winning seeder set the If-Match token (room + Redis filedoc:syncver) only after the
liveness/seeded guard that follows seedIfEmpty. But the tailer can integrate the just-appended
seed DURING the seedIfEmpty await, so isDocSeeded(room.doc) is already true when the guard runs
and it returns early — leaving the stream holding seed content with no cluster version. Later
persists then send no If-Match, the app returns `deferred`, and session edits stay only in the
TTL'd stream (the exact stranding this PR prevents elsewhere).

Move the version stamp to immediately after seedIfEmpty wins, before the guard. Recording it only
once our seed won (not from the fetch) is preserved, so it still can't shadow a peer's winning
seed.

* fix(collab-doc): make the synced-version token monotonic at every write site

The If-Match token is written fire-and-forget from the seed stamp, merges, and persists, both
locally and to Redis. An out-of-order write (e.g. a seed's lagged setSyncedVersion landing after a
later merge's) could regress it below the version the live doc already incorporates, causing
spurious If-Match conflicts — and on a last-leave flush with no live room to reconcile into, a
spurious conflict leaves durable authoritative and drops the session's edits.

- setSyncedVersion now writes via SET_VERSION_IF_NEWER_SCRIPT (Redis-side compare-and-set): it
  overwrites only when the new value is greater, refreshing the TTL either way.
- recordVersion / the persisted branch / the seed stamp all take Math.max instead of assigning
  room.syncedVersion directly.

Versions are monotonic epoch-ms, so "newer" is a plain numeric compare, exact within a Lua double.

* fix(collab-doc): close three last-leave persist edge cases from review

- Stale snapshot after reconcile (High): the multi-task captureState fell back to the pre-await
  localState snapshot even after a reconcile advanced ifMatch, so a failed stream re-read could
  persist the pre-reconcile state against the new version and clobber the out-of-band edit the
  reconcile just incorporated. NULL localState after a reconcile so a failed read aborts instead.

- Lock miss aborts reconcile (Medium): a merge-lock acquisition failure returned 'no-live-room',
  indistinguishable from an absent stream, so flushPersist treated transient contention as
  terminal. Return a distinct 'merge-unavailable' and handle it as retry-later (edits stay in the
  stream), never as "nothing to reconcile into".

- Peer syncver never recovers (Medium): the winner's setSyncedVersion was fire-and-forget with
  swallowed errors — the only way a peer-seeded task learns the durable version — so a dropped
  write left that peer deferring forever. Make it retry (bounded) like appendUpdate/seedIfEmpty;
  the monotonic script keeps a racing retry a no-op.

* fix(collab-doc): scope the persist If-Match to a content version so metadata bumps can't clobber edits

The optimistic-concurrency validator was `updatedAt`, which rename/move/delete/restore also bump
with no content change. A racing live-doc persist then saw a stale token, got `conflict`,
reconciled the pre-edit durable body via updateYFragment (incoming-wins on overlap), and wiped the
user's in-flight edits.

Scope the validator to content (RFC 7232 semantics — validate the representation, not the row):
- New `workspace_files.content_updated_at` (NOT NULL, `now()` fast-default — no table rewrite).
  Advances ONLY on content writes (upload / overwrite / create); metadata writes never touch it.
- The FOR UPDATE CAS, the merge-notify version, and the seed version all use `content_updated_at`.
  A rename now leaves it unchanged, so the persist If-Match still matches -> no spurious conflict,
  no reconcile, no lost edits. Genuine out-of-band content writes still conflict and reconcile.
- Consolidated the collab schema into one migration (the collab-state table + the new column) per
  request, rather than a separate follow-up migration.

Relay/store/contracts unchanged (still a numeric monotonic version).

* chore(collab-doc): condense the densest persist comments (no behavior change)

Cleanup pass: tighten the three longest comment blocks added while hardening the persist path
(currentVersion cache, ifMatch threading, final-flush version retry) without dropping any invariant.
No dead code found (biome lint clean; all new symbols referenced).

* fix(collab-doc): persist must return the content version, not updatedAt

Follow-up to the content-scoped If-Match: persistFileDoc still returned `updatedAt` as the version
in both the persisted and conflict results, while the CAS/seed/merge all guard on
`content_updated_at`. A content write sets both to the same instant, so it was coincidentally
correct — until they diverge: if a metadata write bumps `updatedAt` past `content_updated_at`, the
conflict path returned the larger `updatedAt`, so the relay's re-persist sent an If-Match the CAS
(which checks `content_updated_at`) could never match → perpetual conflict → dropped reconciled
edits. Return `contentUpdatedAt` in both paths so the relay's token always matches what it's checked
against.

* fix(collab-doc): defer persist whenever the version is missing; guard the content-version test

- Empty-file CAS race (Medium): the unconditional-write carve-out for size===0 read `record.size`
  outside the write transaction, so a concurrent first content write could land after the check and
  be clobbered. With content_updated_at NOT NULL every existing file always has a real version, so a
  missing expectedVersion is always transient — always defer, never write unconditionally. Removes
  the TOCTOU hole.
- Content-version test (Low): the merge-chokepoint test kept updatedAt == contentUpdatedAt, so it
  passed even if wired to the wrong field. Mock distinct values and assert contentUpdatedAt, so a
  regression to updatedAt now fails the test.

* fix(collab-doc): don't reconcile a conflict the live doc already reflects (would wipe newer edits)

flushPersist reconciled the durable body into the live doc on every conflict. But when the conflict
comes from a racing self-persist (or an apply-edit the chokepoint already merged), the durable body
is a STALE SUBSET of the live stream, and the incoming-wins updateYFragment merge moves the doc
backward — wiping newer in-flight edits, which the retry then persists.

Before reconciling, re-check the freshest synced version. If it already covers the conflict version,
the live doc has already incorporated that content (or is ahead), so skip the reconcile and just retry
with the freshest version as If-Match — the re-projection captures the current live stream, preserving
every edit. Only a genuine out-of-band change the live doc hasn't incorporated (freshest < conflict
version) is reconciled in. freshest never exceeds the durable version, so this can't loop.

* fix(collab-doc): make content_updated_at monotonic per file; skip-reconcile can't loop

The If-Match token was stamped with app-local new Date() on each content write, so cross-instance
clock skew could stamp a later write with an EARLIER content_updated_at — breaking the version
ordering the whole optimistic-concurrency scheme (and the skip-reconcile branch's freshest>=version
assumption) depends on. Under skew the relay's monotonic syncedVersion could exceed the durable
version, sticking the If-Match: persist conflicts forever, exhausts retries, drops the session's edits.

- Stamp content_updated_at strictly after the current committed value (we hold the row's FOR UPDATE
  lock): new Date(max(now, currentFile.contentUpdatedAt + 1ms)). Monotonic per file regardless of
  clocks; also removes same-millisecond collisions. updatedAt stays plain wall-clock (display/sort).
- Skip-reconcile branch retries with result.version (the durable value the CAS will match), never
  freshest (which could exceed it and loop). Belt-and-suspenders now that the version is monotonic.

* refactor(collab-doc): drop the destructive in-persist reconcile; adopt-version-and-retry on conflict

The in-persist reconcile projected the durable body back over the live doc via updateYFragment
("make the doc match"). That is destructive: when the live stream is already ahead — the common case,
because the write chokepoint (mergeEditIntoLiveFileDoc) already merged the out-of-band change into the
stream — it moved the doc backward and wiped newer in-flight edits. This produced a run of races
(stale snapshot, wipe-newer-edits, version-lag skip miss) that a full-document reconcile fundamentally
can't avoid, since deciding when it's safe relies on a laggy cross-task version token.

Remove it. On conflict, adopt the durable version as the new If-Match and retry: captureState re-reads
the current stream (which holds the out-of-band change AND the live edits), so the re-projection
persists the converged result. The durable change reaches the live doc via the chokepoint, never here.
Trade-off: the only unmerged out-of-band write is one whose chokepoint merge itself failed (rare,
logged), which we accept over the frequent reconcile-wipes-edits race.

- flushPersist: conflict -> ifMatch = result.version, retry (bounded). No applyMarkdownToLiveFileDoc.
- conflict response drops `markdown` (contract + relay type + persist) — no body needed, saves a blob
  fetch. applyMarkdownToLiveFileDoc stays (still used by the apply-edit route / the chokepoint).

* fix(collab-doc): don't let a last-leave conflict retry clobber via the stale local snapshot

Regression from dropping the reconcile: on conflict the retry adopts result.version and re-reads
captureState. But after single-pod last-leave teardown the room is already destroyed, so captureState
falls back to the pre-teardown localState (which lacks the out-of-band change); the retry then CAS-passes
and overwrites the committed external write — undoing the external-wins last-leave policy.

Null localState on the first conflict, so the retry can only use freshly-read authoritative state
(stream / live doc). When none is available (single-pod room gone, or a transient stream-read failure)
captureState returns null and the retry stops, leaving durable content authoritative. Covers both the
single-pod and multi-task-stream-unavailable variants of the stale-snapshot clobber.

* fix(collab-doc): stop (don't re-persist) on a persist conflict — closes the commit-window clobber

The conflict retry adopted the durable version and immediately re-persisted the current stream,
assuming the stream already held the out-of-band change. But an external write commits durable BEFORE
its chokepoint merge (mergeEditIntoLiveFileDoc) reaches the stream, so a persist landing in that window
CAS-passed with a stream that still lacked the external content and clobbered the committed write — not
just the rare merge-failed path, but a race on every external write, worst at last-leave flushes.

Make persist a single attempt: on conflict, STOP and leave durable authoritative. The chokepoint merges
the change into the stream and — only once it is actually there — advances the synced version via its own
recordVersion; a later flush (debounced or final) then projects the converged stream with a matching
token. The session's edits stay in the stream meanwhile. The conflict handler deliberately does NOT
advance the synced version, or the next flush would clobber with a still-behind stream. Removes the retry
loop and PERSIST_CONFLICT_RETRIES.

* improvement(tables): fire the live-rows signal on async delete, run cancel, and column run (#6094)

* improvement(tables): fire the live-rows signal on async delete, run cancel, and column run

These three table operations mutate row data but emitted no `rows` change signal, so open editors'
grids stayed stale until a manual refresh (enrichment *results* already stream live via `cell` events;
these are the bulk paths that don't emit per-cell events):

- Async row delete (`runTableDelete`): signal as rows drop out (throttled with the existing progress
  event) and once more on completion — the `job` progress event only drives the delete meter, not the
  rows query. Covers the delete-async route and the copilot bulk-delete, since both share the runner.
- Cancel runs (`cancel-runs` route): cancelling clears each affected row's exec state; the
  `dispatch: cancelled` events drop the run overlay but the client then renders authoritative DB state,
  so refetch. Only when something was actually cancelled.
- Run column (`columns/run` route): starting a run bulk-clears the target group's cells to pending;
  refetch so the cleared cells show. Only when a dispatch was actually created.

Guarded so no signal fires on a no-op/failure. Adds a delete-runner test asserting the completion signal.

* fix(tables): guarantee the live-rows signal on every mutating path (review)

- Delete runner (Greptile P1): a batch could commit and the job then cancel/supersede before the next
  throttled progress signal or `markJobReady`, bypassing both signals and leaving deleted rows on
  screen. Track `deletedAny` and fire the grid refetch in a `finally`, so it runs on EVERY exit —
  completion, cancel/supersede, mid-batch lock, or a rethrown error after a partial delete.
- cancel-runs / columns/run routes (Cursor): the `cancelled > 0` / `if (dispatchId)` guards don't
  always reflect DB row changes — cancel tombstones exec state even when 0 dispatches were active, and
  a run bulk-clears cells then can return a null dispatchId. Signal unconditionally; a stale-but-harmless
  refetch beats a missed one.
- Tests: assert the delete signal fires on the mid-run-cancel-after-delete path and NOT when nothing
  was deleted.

* fix(tables): mark deletedAny before the page delete so a mid-page lock still refreshes the grid

`deletePageByIds` commits in internal batches, so a delete lock landing mid-page can persist earlier
batches and THEN throw TableLockedError — the catch returns without a count, so setting `deletedAny`
from the return value missed it and the finally skipped the grid refetch. Set `deletedAny = true` before
the call (any attempt may commit rows); an attempt that commits nothing only over-refetches (harmless).
Adds a test asserting the signal fires when a page throws a mid-page lock.

* fix(files): make embedded resource file view collaborative (#6095)

* fix(files): make embedded resource file view collaborative

The /chat resource panel rendered saved files through FileViewer without
the collaborative opt-in, so a file open on the Files page and the same
file open in the embedded panel never joined the same file-doc room —
no live carets and no live content sync between the two surfaces.

Pass collaborative on the EmbeddedFile FileViewer. Collaboration still
self-gates on canEdit + non-streaming + workspace doc, so the agent
token-stream preview (the dedicated streaming-file path, canEdit=false)
is untouched.

* fix(files): refcount file-doc room membership per shared socket

Two collaborative surfaces in one tab (the Files editor and the embedded
chat resource panel) share one Socket.IO connection, so both providers for
the same file JOIN the same room over that socket. The server's LEAVE does
socket.leave(name) with no membership refcount, so the first provider's
destroy() would strand the second still-mounted one — no more live content
or presence.

Count live providers per file per socket (keyed by the stable Socket object,
so it survives reconnects) and emit LEAVE only when the last provider for a
file tears down. The single-provider path is unchanged (0->1->0).

* feat(tables): propagate shared saved-view changes to collaborators live (#6100)

Table views (named filter/sort/layout presets) are table-wide shared state —
every reader sees every view — but view create/update/delete had no realtime
signal, so a collaborator only saw another user's view changes on their own
staleTime/focus refetch.

Add a 'views' table event kind + signalTableViewsChanged, emitted from the
views service (createTableView/updateTableView/deleteTableView, on real
success only), and a client handler that invalidates the views query alone
(no rows/definition refetch — a view is presentation state on the loaded
table). Mirrors how row/schema/metadata changes already propagate.

* test(tables): cover the views realtime signal (emit + on-success-only) (#6101)

- events.test.ts: signalTableViewsChanged appends a single 'views' event
  carrying the tableId (through the real memory buffer).
- views/service.test.ts: create/update/delete emit signalTableViewsChanged
  on real success, and DON'T on a no-op (a PATCH/DELETE targeting a missing
  view changes nothing, so it must not signal). Mirrors delete-runner's
  signal-path coverage; drives the DB via the shared dbChainMock.
- Add tableViews to the comprehensive @sim/db/schema test mock so the
  service tests can queue the in-transaction existence row.

* chore(ci): reconcile api-validation baselines after the staging merge

The staging merge unioned realtime-rooms's own `as unknown as` cast
(lib/collab-doc/converter.ts) with staging's zod-recursive-type cast
(lib/api/contracts/tables.ts), so the non-test double-cast count is 9 —
both casts pre-existed and were individually accepted on their branches.
Also tighten rawJsonReads 6->5 to the true current count. Fixes the strict
API contract boundary audit on realtime-rooms.

* feat(copilot): stream file edits into the live collaborative Y.Doc (keep embedded view collaborative) (#6108)

* feat(copilot): stream file edits into the live collaborative Y.Doc

Copilot's file edits previously only reached the live doc once, at the final
edit_content write, so a collaborative editor watching the file saw nothing
until completion (streaming looked broken) and the client-side preview path
was suppressed in collab mode.

Make copilot a CRDT peer: as it streams append/update/patch content, merge the
growing markdown into the file's live Y.Doc via the existing apply-edit path
(a minimal updateYFragment diff, concurrent-edit-safe), throttled to ~250ms.
version is omitted for these intermediate merges — they advance the live doc
for viewers but are not durable checkpoints; the final edit_content write
carries the real contentUpdatedAt and reconciles the durable file. Per the
relay's persist gating, server-internal merges never schedule a persist, so a
copilot-only stream produces zero intermediate file writes.

- notify.ts: mergeEditIntoLiveFileDoc version is now optional (streaming omits it).
- file-preview-adapter.ts: throttled live-doc merge at the edit_content stream hook.

* fix(copilot): order + gate streaming live-doc merges; fast collab first render

Harden the streaming merge (adversarial review):
- Order + bound: dispatch through a per-file in-flight guard (drop-while-in-flight)
  so a stale out-of-order snapshot can never land after a newer one and regress
  the doc, and relay load is capped at one request per file regardless of rate.
- No wipe: gate append/patch on the base file content having loaded — a base-less
  snapshot would diff to a delete-everything wipe of the seeded doc; update streams
  a full rewrite from scratch and needs no base.
- Markdown-only gate: non-markdown files have no collaborative room, so skip the
  wasted relay round-trip.

Fast collab first render (Issue 2): render the already-fetched markdown read-only
via generateHTML while the collaborative doc seeds, with the editor mounted-but-
hidden in the same layout box for a seamless swap on collabReady. Pure HTML — it
never touches the Y.Doc (client seeding duplicates the doc), and generateHTML
escapes text (raw-HTML snippets render escaped), so no XSS.

* test(copilot): cover streaming file edits into the live collaborative Y.Doc

Drives edit_content args_delta stream events through processFilePreviewStreamEvent
and asserts the live-doc merge: fires with the growing FULL previewText and no
version arg; is throttled (~250ms per file); is skipped for non-markdown files
and for a base-less append (the delete-everything wipe guard); and runs at most
one-in-flight per file. Verified to fail if any gate/guard is removed.

* fix(collab-doc): coordinate live-doc merge ordering in one place; close durable-clobber race

The second review found a residual: the durable edit_content write went through a
different path than the adapter's in-flight guard, so a late straggler streaming
merge could land after it and, via a persist, clobber the durable file's tail.

Move the per-file coordination into mergeEditIntoLiveFileDoc (the one place both the
streaming and durable paths call): a streaming (versionless) merge is dropped while
one is in flight for the file; a durable (versioned) write instead WAITS for the
in-flight streaming merge, so the final content is always the last merge applied and
can't be regressed by a straggler. Simplifies the adapter (drops its Set + helper).

Relocate the one-in-flight test to notify.test.ts (streaming-drops-while-busy +
durable-waits-then-applies-last); the adapter test keeps throttle/gates/previewText.

* fix(copilot): address review — order merges, exclude update, gate throttle, unhide stream

Review round on #6108:
- Greptile P1 (durable merges lose ordering): serialize ALL merges per file on one chain in
  mergeEditIntoLiveFileDoc (each chains after the current tail), so concurrent durable writes
  can't resume-and-fire out of order. notify now exposes isLiveDocMergeInFlight.
- Cursor High (update stream blanks the doc): only append/patch stream — they build on the loaded
  base; update is a from-scratch rewrite whose partial snapshot would diff the full doc toward a
  fragment, so it applies atomically at the durable write.
- Cursor Medium (throttle advances on a dropped merge): the adapter gates on !isLiveDocMergeInFlight,
  so the send throttle advances only on an actual dispatch — no lag, no backlog behind a slow relay.
- Cursor Medium (placeholder hides a live stream): show the fast-render placeholder only when not
  streaming, so a stream that starts before the doc seeds shows through the editor.
- Soften merge.ts/notify.ts comments per the lifecycle audit: only UNTOUCHED regions are preserved;
  a region the merge rewrites reconciles toward copilot's content.

Tests updated: notify covers chain ordering + isLiveDocMergeInFlight; adapter covers append streaming,
throttle, non-markdown/base-less/update skips, and the in-flight skip.

* fix(collab-doc): reject stale durable merges at the relay (cross-process ordering)

The in-process merge chain only orders merges within one apps/sim process. Two durable
writes for the same file on DIFFERENT processes could reach the relay out of dispatch
order; the relay recorded the version monotonically but still APPLIED the older markdown,
regressing the live doc while the token stayed high (a later persist could then write the
stale content back over the durable file).

Enforce ordering at the relay — the single cross-process coordination point — using the
existing Redis primitives: under the per-file Redis merge lock, read the cluster-wide
synced version and SKIP a versioned merge that is not newer (a newer durable write already
landed). Make recordVersion await setSyncedVersion so it is durable before the lock
releases, so the next holder's staleness check reads a consistent value. Streaming
(versionless) merges are unaffected — they carry no durable version and are ordered
per-process by the caller.

Adds a relay test asserting a stale/idempotent versioned merge returns 'stale' and never
computes or publishes a diff.

* fix(copilot): match durable path — detect markdown by MIME type + name at the stream gate

The streaming gate checked isMarkdownFile with only the filename, while the durable merge
uses type + name — so a text/markdown file without a .md extension was skipped mid-stream
(it self-corrected at the durable write). Pass editIntent.contentType so streaming detects
the same set of markdown files as the durable path.

* test(copilot): assert throttle follow-through after an in-flight merge clears

* fix(collab-doc): order streaming merges by streamedAt so a late snapshot can't regress a newer durable write

* refactor(collab-doc): tidy merge-order docs + relay order object; cover multi-replica streaming stale-check

* fix(collab-doc): order streaming merges by causal base version, not wall-clock

A streaming snapshot now carries baseVersion (the durable contentUpdatedAt it was
built from) instead of a wall-clock streamedAt. The relay drops the snapshot when a
newer durable write landed since that base, so a concurrent human save can no longer
be clobbered in the live doc and then persisted over the durable file. Skew-immune:
both keys are DB-monotonic contentUpdatedAt values.

* fix(collab-doc): derive streaming baseVersion as contentUpdatedAt ?? updatedAt

Match the version line the seed/persist use so a legacy file with no content
version still ships an ordered streaming snapshot instead of an unordered one.

* fix(collab-doc): fail-closed on a streaming snapshot with no baseVersion

The live-merge gate now requires a numeric baseVersion, not just loaded base
content. A rare base with no file record (hence no version) would otherwise ship
an unordered snapshot the relay can't stale-check, risking a clobber of a
concurrent durable write. Skip the live merge instead; the durable write reconciles.

* docs(collab-doc): document the accepted concurrent-independent-streams limitation

* chore(ci): reconcile api-validation baseline after the staging merge

The merge commit auto-merged the baseline at 1000; bump totalRoutes/zodRoutes to
1003 for staging's three new contract-bound routes (nonZodRoutes still 0).

* test(files): update storage-accounting assertion to the mergeEditIntoLiveFileDoc options object

* fix(collab-doc): trace the full yjs/tiptap external stack into the file-doc route bundles

The seed/merge/persist internal routes run the collab-doc converter (markdown <-> Yjs
via headless TipTap) server-side. Those deps are serverExternalPackages, and the
standalone tracer only force-included jsdom — it does NOT follow yjs's ESM subpath
imports of lib0 (lib0/logging, ...), so Docker/standalone builds shipped node_modules
without them and the seed route 500'd (Cannot find module 'lib0/logging'). That left
every collaborative document unseeded and permanently read-only on deployed envs.
Force yjs, lib0, y-protocols, and @tiptap into the trace for all three routes.

* fix(collab-doc): copy the full yjs/lib0 stack into the app image

The seed/merge/persist routes run the converter (markdown <-> Yjs) server-side. yjs is a
serverExternalPackage and the Next standalone tracer copies lib0 only partially — it drops the
ESM subpath file lib0/logging.js that yjs.mjs imports via lib0's exports map, so the seed 500s
('Cannot find module lib0/logging') and every collaborative doc is stuck read-only. Verified in
the running dev container: /app/node_modules/lib0 had 37/38 files, logging.js missing.

outputFileTracingIncludes can't fix it — its globs resolve against apps/sim, but these deps hoist
to the monorepo-root node_modules, so the glob matches nothing (my prior next.config attempt was a
no-op; reverted). Instead COPY the complete lib0/yjs/y-protocols from the deps stage in the runner,
overwriting the partial trace — the same pattern already used for isolated-vm.

* feat(files): stream copilot edits into the collaborative doc smoothly (#6122)

* feat(files): stream copilot edits into the collaborative doc smoothly

- apply the agent stream client-side into the live Yjs binding as minimal
  updateYFragment diffs (like main's setContent, but incremental) so it renders
  smoothly AND broadcasts to every peer via CRDT — a collaborator on /files sees
  the stream for free
- gate the apply on collabReady so diffs never land on an unseeded doc; keep the
  read-only placeholder visible until the seed swaps in
- run streamed ops under a dedicated tx origin so they stay out of the user's
  undo stack
- delete the throttled server-side streaming merge and the baseVersion ordering
  machinery it needed (relay + notify + session contract); the durable final
  write still reconciles open editors and seeds late joiners

* fix(files): apply agent stream as a true CRDT peer + guard base-less snapshots

Review round 1 (Greptile P1s):
- apply the stream against a private shadow replica (seeded from the live doc at
  stream start) and relay only the agent's own delta into the shared doc, so a
  concurrent peer edit to a region the agent snapshot didn't include is no longer
  reverted (previously the whole-body reconcile deleted it)
- gate append snapshots on "must extend the base": a base-less append fragment
  (emitted before the base loads) can no longer reconcile the seeded doc to a wipe;
  patch still legitimately replaces a mid-region
- gate the apply on collabReady so diffs never land on an unseeded doc; keep the
  placeholder visible until the seed swaps in
- plumb streamOperation through the preview surfaces to drive the append gate
- add a peer-edit-preservation test (fails under whole-body reconcile) and refresh
  the undo-isolation + broadcast tests for the session API

* fix(files): destroy the agent shadow deterministically on settle

Cursor round 1 (Low): endAgentStream ran inside runOffRender, whose microtask is
dropped when a rapid follow-up stream bumps the run token — leaking the shadow
Y.Doc. Split it out into an unguarded microtask queued after the (droppable) final
apply, so the shadow is always destroyed.

* fix(files): agent stream frames skip the relay's durable persist

Cursor round 1 (High): client-applied stream frames broadcast over the sync
channel, so the relay stamped a socket origin and ran schedulePersist — durably
writing partial agent content mid-stream, attributed to the watching user (the old
server-merge applied with no origin and never did). Restore that behavior:

- new FILE_DOC_MESSAGE_TYPE.SYNC_NO_PERSIST wire tag; the provider tags
  AGENT_STREAM_ORIGIN updates with it (normal user edits stay SYNC)
- the relay applies it under an AgentSyncOrigin (carries the socket id for
  broadcast exclusion, but is not a plain string) so originSocketId() is null →
  no edited/schedulePersist/lastEditorUserId; excludeSocketId() still excludes the
  sender, and the update still publishes to the stream so peers converge
- the copilot's final edit_content write remains the authoritative durable persist
- tests: relay applies+fans-out but never persists a SYNC_NO_PERSIST frame
  (verified it fails if applied as a socket edit); provider tags agent edits

* fix(files): open the stream shadow at start + private extend baseline

Cursor round 2:
- High (settle skips apply without session): the stream shadow is now opened on
  the first ready frame, BEFORE the extend gate — so an `update` rewrite (whose
  every frame is gated out until settle) and a stream that finishes before seed
  still get a session, and settle applies the final body via the reused-or-on-demand
  shadow instead of leaving the doc stale until the durable reconcile.
- Medium (peer edits stall the stream): the extend gate now reads a private
  `lastStreamedBodyRef` (the agent's own last frame), snapshotted at stream start,
  not `lastSyncedBodyRef` which `onUpdate` clobbers on peer edits — so a collaborator
  typing can't make the growing snapshot stop prefixing the shown body and freeze it.
- Medium (multi-replica over-persist): pre-existing, documented "safe over-persist"
  (a peer task tails the frame as REDIS_ORIGIN and marks edited) — refreshed the
  stale comment to describe the SYNC_NO_PERSIST source; copilot's edit_content write
  remains the authoritative durable persist.

* fix(files): fail-close base-less previews + operation-based stream hold

Cursor/Greptile round 3 (High + Medium) — remove the fragile string-prefix
"extend gate", which was the root of both findings:

- Server: `buildFilePreviewText` now fails closed for an `append` whose base
  content hasn't loaded (returns undefined, like patch/update), so a base-less
  fragment never reaches the client. This eliminates the base-less wipe at
  settle (Greptile P1) at the source; an empty file (existingContent === '')
  still previews normally.
- Client: the collab streaming tick no longer string-prefixes the raw preview
  against the editor's canonical markdown (the '*' vs '-' / emphasis mismatch
  that froze every append frame — Cursor). The mid-stream hold is now purely
  operation-based: `update` waits for settle; append/patch/create apply each
  frame via the (peer-safe) shadow reconcile. lastStreamedBodyRef is now a plain
  dedup guard, not a prefix baseline.

Keeps the shadow, durable write, and SYNC_NO_PERSIST unchanged.

* fix(files): elect a single agent-stream writer across tabs

Cursor round 4 (High): with the stream applied client-side, two tabs/windows on
the same chat could each derive streamingContent (the reconnect/resume path
re-consumes preview events) and each independently insert the stream under a
different Yjs clientID, duplicating content until the durable reconcile.

Fix — single-writer election via the file-doc awareness (new agent-stream-leader):
- a client applying an agent stream announces `agentApplying` on its own awareness
- only the leader (min clientID among announcers) applies mid-stream AND at settle;
  a non-leader renders the leader's ops via Yjs and does not apply (a non-leader
  applying the final body would re-insert the whole doc as a duplicate)
- re-checked each frame, so it converges to one writer the moment awareness
  propagates; the sub-frame startup race is reconciled by the durable write
- single-client (the common case) is unaffected: it is the only announcer, so it
  always leads

* fix(files): gate the settle apply locally, not on a settle-time re-election

Cursor round 5 (High): the settle recomputed leadership from live awareness and
the leader cleared its announcement immediately, so a straggler peer that settled
afterward became the sole announcer, self-elected, and applied finalBody through
its base-seeded shadow — re-inserting the whole doc as a duplicate.

Fix: gate the settle apply on a LOCAL didApplyStreamRef (set only when this client
actually applied a mid-stream frame — i.e. it was the mid-stream leader whose
shadow is up to date), not on a settle-time re-election. A client that never
applied (non-leader, a held `update`, or a pre-seed stream) skips the final apply
and converges via Yjs + the durable write. The mid-stream leader election
(isAgentStreamLeader) is unchanged, so exactly one client's didApplyStreamRef is
ever true.

* fix(files): open the agent-stream shadow lazily on lead (no stale handoff)

Greptile round 6 (P1): the leader race — (a) a mid-stream leadership handoff
could apply from a stale pre-stream shadow, and (b) two tabs starting the same
stream before awareness converges could both lead briefly.

- (a) fixed: the shadow is now opened LAZILY in the tick, only when this client
  actually leads, seeded from the CURRENT doc — so a handoff successor diffs
  against the prior leader's ops (never a stale base) and a non-leader builds no
  shadow at all. Announce candidacy via a dedicated ref (decoupled from the
  shadow); settle still gates the final apply on didApplyStreamRef (leader-only).
- (b) the pure startup race is inherent to eventually-consistent election. It is
  now the only residual: bounded to two tabs starting the SAME stream within the
  awareness-propagation window, transient (converges in a frame or two), and
  never persisted (SYNC_NO_PERSIST + the durable edit_content reconcile). Resumes
  are sequential, so the common multi-tab case elects cleanly. Documented inline;
  a server-granted lease would close it fully but at a round-trip cost on the
  common single-tab path, which isn't worth it.

* fix(files): idempotent settle apply (update lands client-side; no straggler dup)

Cursor round 6 (Medium): a lone client's `update` never applied client-side —
held mid-stream, then skipped by the didApplyStreamRef settle gate — so the
rewrite depended entirely on the durable merge (stale if delayed/failed).

Root cause was over-correcting round 5. Now that the shadow is opened lazily in
the tick (current-seeded), the round-5 base-shadow duplication is already gone,
so didApplyStreamRef is unnecessary. Replaced it: settle applies the final body
via `agentStreamSessionRef.current ?? beginAgentStream(editor)` — the leader
reuses its up-to-date shadow (last throttled frame), while a client that never
applied (non-leader, held `update`, pre-seed) opens a FRESH current-seeded shadow.
Reconciling current->final is idempotent: a straggler that settles after another
wrote the final reconciles to a noop. So a lone `update` applies at settle (no
wait on the merge), and there's still no settle-time election or base-shadow dup.

* fix(files): broadcast agent frames to the whole room (same-socket siblings)

Cursor round 7 (Medium): SYNC_NO_PERSIST frames applied under an origin carrying
the sender socket id, and excludeSocketId dropped that whole socket from the
relay fan-out. A second FileDocProvider on the same socket (chat preview + Files
editor) then missed all mid-stream ops and stayed stale until the durable
reconcile — a regression from the old no-origin server merge, which reached both.

Fix: the agent origin is now a plain AGENT_SYNC_ORIGIN symbol, and agent frames
broadcast to the WHOLE room (no socket excluded), matching the old behavior — so a
same-socket sibling provider stays live; the emitting provider no-ops on its own
echo (the ops are already applied locally). originSocketId still returns null for
the symbol, so it keeps skipping edited/schedulePersist. Removed excludeSocketId
and the socket-carrying origin object. Updated the relay test to assert the
whole-room broadcast (verified it fails if the sender is excluded).

* fix(files): tag agent stream frames no-persist across replicas

A peer task tailing an agent-streamed preview frame previously applied it
as REDIS_ORIGIN, marking the seeded room edited and making a transient
startup-race duplicate eligible for that task's last-disconnect flush. Mark
agent frames with a stream field so peers apply them as REDIS_AGENT_ORIGIN,
excluded from the edited/persist gate. The copilot's durable edit_content
write stays the sole authority over file bytes.

* fix(files): reseed agent shadow on lead regain + agent-only compaction

Two multi-writer edge cases surfaced in review:

- rich-markdown-editor: a client that led, lost leadership, then regained it
  reused its stale shadow (which never saw the interim leader's ops), re-emitting
  ops for content already present. Tear the shadow down when a client observes it
  is not the leader, so a regain rebuilds fresh from the current doc.

- file-doc-store: compaction always stamped its snapshot REDIS_SNAPSHOT_ORIGIN
  (marks peers edited). A long agent-only stream crossing the threshold could
  fold preview content into a persist-eligible snapshot. Track whether a room
  integrated any real edit and stamp an agent-only snapshot REDIS_AGENT_ORIGIN
  so it stays no-persist.

Both covered by falsification-verified tests.

* fix(files): close realEdited data-loss race + elect a settle writer

Independent audit surfaced two real gaps:

- file-doc-store: realEdited was latched AFTER appendUpdate's awaits, but the
  edit already sits in room.doc synchronously. A concurrent agent-frame
  compaction could read realEdited=false, snapshot that real content, and stamp
  it a no-persist agent frame — a lost edit. Latch it synchronously (same tick
  as the doc mutation) before any await. Deterministic falsifiable test added.

- rich-markdown-editor: at settle every tab applied the final body, and a
  non-leader's local microtask runs before the leader's final propagates, so
  both insert the tail (Yjs keeps both) -> duplicated tail. Elect a single
  settle writer (reliable — awareness is long converged by settle), reading
  leadership before clearing the announcement. Corrects the overclaiming
  idempotency comment and the handoff pick-up comment.

Adds a y-tiptap internals upgrade-guardrail test.

* fix(files): own presence per client id, not one-per-socket

The shared workspace socket hosts one collaborative provider per mounted view,
so the chat file preview and the standalone Files editor for the same file each
bind their own Yjs client id over ONE socket. The relay owned a single client id
per socket, so the later JOIN overwrote the earlier and dropped its awareness —
which silently broke the single-writer agent-stream election (a peer stopped
seeing the streaming provider's announcement and could self-elect, duplicating
streamed text for the whole stream).

Track ownership per (socket, client id): a socket owns a set of client ids; the
awareness gate accepts a frame only if every id it carries is owned; cleanup
drops all of a socket's ids; the roster stays one-entry-per-session. Reclaim and
the same-user reconnect path evict just the reclaimed id, dropping the old socket
only if it empties. Falsification-verified test added.

* fix(files): make streamed file-preview accumulation replay-safe

Guard deriveFilePreviewSession against re-delivered/replayed content events:
apply a delta/snapshot only when previewVersion strictly advances, so a client
re-render or stream replay can't double-append the tail (the duplicated-content
bug) or regress on an older snapshot.

* fix(files): fix new-file collab streaming latch and agent-edit duplication

- Latch collab readiness so a new file's post-seed `synced` flap can no longer
  re-gate agent streaming (the stream previously showed only the seed and the
  rest appeared only on reload)
- Relay defers the durable edit_content merge to an actively-streaming client:
  the client shadow stream and the server merge were both writing the same
  content into the live doc, duplicating it when the server ran ahead
- Render the collaborator caret bar out of flow so a peer caret never nudges
  the surrounding text by ~1px
- Remove dead code: unused FileDocMessageType alias, unnecessary
  LiveFileDocMergeOrder export

Covered by tests: readiness latch (flap/offline/latch cases), relay merge
deferral (single- and multi-replica), plus verified-failing guards.

* test(copilot): fix loadWorkspaceFileTextForPreview mock to return { text } not a bare string

The adapter reads previewBase.text to seed an append/patch base; the mock returned
a bare '' so previewBase.text was undefined, making a base-less append fail closed
(no file_preview_content). My PR's fail-close change exposed the wrong-shaped mock.

---------

Co-authored-by: mzxchandra <129460234+mzxchandra@users.noreply.github.com>
This commit is contained in:
Waleed
2026-07-31 18:48:12 -07:00
committed by GitHub
parent ecf1d7dbbb
commit 10bfb5d139
164 changed files with 33791 additions and 2151 deletions
+3
View File
@@ -33,9 +33,12 @@
"@sim/workflow-types": "workspace:*",
"@socket.io/redis-adapter": "8.3.0",
"drizzle-orm": "^0.45.2",
"lib0": "0.2.117",
"postgres": "^3.4.5",
"redis": "5.10.0",
"socket.io": "^4.8.1",
"y-protocols": "1.0.7",
"yjs": "13.6.31",
"zod": "4.3.6"
},
"devDependencies": {
+50 -26
View File
@@ -50,16 +50,14 @@ function makeManager(sockets: FakeSocket[], presence: Partial<UserPresence>[] =
const manager = {
io: { sockets: { sockets: socketMap } },
isReady: () => true,
getWorkflowUsers: vi.fn().mockResolvedValue(presence),
getWorkflowIdForSocket: vi.fn().mockResolvedValue(null),
removeUserFromRoom: vi
.fn()
.mockImplementation(async (_socketId: string, workflowId?: string) => workflowId ?? null),
getRoomUsers: vi.fn().mockResolvedValue(presence),
getRoomForSocket: vi.fn().mockResolvedValue(null),
removeUserFromRoom: vi.fn().mockResolvedValue(true),
broadcastPresenceUpdate: vi.fn().mockResolvedValue(undefined),
}
return manager as unknown as IRoomManager & {
getWorkflowUsers: ReturnType<typeof vi.fn>
getWorkflowIdForSocket: ReturnType<typeof vi.fn>
getRoomUsers: ReturnType<typeof vi.fn>
getRoomForSocket: ReturnType<typeof vi.fn>
removeUserFromRoom: ReturnType<typeof vi.fn>
broadcastPresenceUpdate: ReturnType<typeof vi.fn>
}
@@ -84,8 +82,11 @@ describe('access-revalidation sweep', () => {
expect.objectContaining({ workflowId: 'wf-1' })
)
expect(socket.leave).toHaveBeenCalledWith('wf-1')
expect(manager.removeUserFromRoom).toHaveBeenCalledWith('sock-1', 'wf-1')
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith('wf-1')
expect(manager.removeUserFromRoom).toHaveBeenCalledWith(
{ type: 'workflow', id: 'wf-1' },
'sock-1'
)
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith({ type: 'workflow', id: 'wf-1' })
})
it('keeps a socket whose access is still valid', async () => {
@@ -141,7 +142,30 @@ describe('access-revalidation sweep', () => {
expect(mockResolveRole).toHaveBeenCalledWith('user-1', 'wf-1', 'read')
// The security scan must stay Redis-free — presence is never consulted.
expect(manager.getWorkflowUsers).not.toHaveBeenCalled()
expect(manager.getRoomUsers).not.toHaveBeenCalled()
})
it('never evicts a socket joined only to a non-workflow room (files/tables/file-doc)', async () => {
// The sweep shares one io with the files/tables/file-doc handlers. Those rooms are
// namespaced (`workspace-files:ws-1`, `table:t-1`), so treating every socket.rooms
// entry as a workflow id would resolve a bogus permission → null → evict the socket
// from its files/table room every pass. Non-workflow rooms must be filtered out.
const filesSocket = makeSocket('sock-1', 'user-1', 'workspace-files:ws-1')
const tableSocket = makeSocket('sock-2', 'user-2', 'table:t-1')
const manager = makeManager([filesSocket, tableSocket])
// Even if the role resolver would say "no access", these must never be swept.
mockResolveRole.mockResolvedValue(null)
const sweep = startAccessRevalidationSweep(manager)
await sweep.runOnce()
sweep.stop()
expect(mockResolveRole).not.toHaveBeenCalled()
expect(filesSocket.leave).not.toHaveBeenCalled()
expect(filesSocket.emit).not.toHaveBeenCalled()
expect(tableSocket.leave).not.toHaveBeenCalled()
expect(tableSocket.emit).not.toHaveBeenCalled()
expect(manager.removeUserFromRoom).not.toHaveBeenCalled()
})
it('evicts only the revoked socket, not co-members of the room', async () => {
@@ -199,28 +223,28 @@ describe('access-revalidation sweep', () => {
sweep.stop()
expect(manager.removeUserFromRoom).toHaveBeenCalledTimes(2)
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith('wf-1')
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith({ type: 'workflow', id: 'wf-1' })
})
it('defers cleanup when removal fails with expired socket mappings', async () => {
it('drops eviction cleanup when the socket is no longer mapped to the room (no infinite retry)', async () => {
const socket = makeSocket('sock-1', 'user-1', 'wf-1')
const manager = makeManager([socket], [{ socketId: 'sock-1', role: 'read' }])
// Mapping keys already expired (lookup resolves null) AND the removal fails
// (the Redis manager swallows the transport error into null) — the failed
// removal must still defer instead of reading as success.
manager.removeUserFromRoom.mockResolvedValueOnce(null)
// A healthy lookup shows the socket is no longer mapped to any workflow room (its presence
// is already gone), and removeUserFromRoom reports a no-op `false`. This is "already clean",
// not a deferrable failure — the cleanup must drop it, never re-enqueue a still-connected
// socket forever. (A genuine failure — still mapped + false — is covered by the next test.)
manager.getRoomForSocket.mockResolvedValue(null)
manager.removeUserFromRoom.mockResolvedValue(false)
mockResolveRole.mockResolvedValue(null)
const sweep = startAccessRevalidationSweep(manager)
await sweep.runOnce()
expect(manager.broadcastPresenceUpdate).not.toHaveBeenCalled()
await sweep.runOnce()
sweep.stop()
expect(manager.removeUserFromRoom).toHaveBeenCalledTimes(2)
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith('wf-1')
// Attempted once, then dropped — not re-enqueued across passes, and no broadcast.
expect(manager.removeUserFromRoom).toHaveBeenCalledTimes(1)
expect(manager.broadcastPresenceUpdate).not.toHaveBeenCalled()
})
it('defers cleanup when the manager swallows a removal failure into null', async () => {
@@ -228,8 +252,8 @@ describe('access-revalidation sweep', () => {
const manager = makeManager([socket], [{ socketId: 'sock-1', role: 'read' }])
// Live mapping but the removal reports nothing removed — the Redis manager
// swallows transport errors into null, so this is the only failure signal.
manager.getWorkflowIdForSocket.mockResolvedValue('wf-1')
manager.removeUserFromRoom.mockResolvedValueOnce(null)
manager.getRoomForSocket.mockResolvedValue({ type: 'workflow', id: 'wf-1' })
manager.removeUserFromRoom.mockResolvedValueOnce(false)
mockResolveRole.mockResolvedValue(null)
const sweep = startAccessRevalidationSweep(manager)
@@ -243,7 +267,7 @@ describe('access-revalidation sweep', () => {
sweep.stop()
expect(manager.removeUserFromRoom).toHaveBeenCalledTimes(2)
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith('wf-1')
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith({ type: 'workflow', id: 'wf-1' })
})
it('skips removal when the socket has since moved to a different workflow', async () => {
@@ -251,7 +275,7 @@ describe('access-revalidation sweep', () => {
const manager = makeManager([socket], [{ socketId: 'sock-1', role: 'read' }])
// Between the membership snapshot and cleanup, the socket switched to a
// workflow it can still access — removal must not touch its new presence.
manager.getWorkflowIdForSocket.mockResolvedValue('wf-2')
manager.getRoomForSocket.mockResolvedValue({ type: 'workflow', id: 'wf-2' })
mockResolveRole.mockResolvedValue(null)
const sweep = startAccessRevalidationSweep(manager)
@@ -357,7 +381,7 @@ describe('access-revalidation sweep', () => {
const manager = makeManager([socket], [{ socketId: 'sock-1', role: 'read' }])
// A Redis outage where commands hang in the offline queue instead of
// failing: the cleanup lane stalls, but scans must keep running.
manager.getWorkflowIdForSocket.mockReturnValue(new Promise(() => {}))
manager.getRoomForSocket.mockReturnValue(new Promise(() => {}))
mockResolveRole.mockResolvedValue(null)
const sweep = startAccessRevalidationSweep(manager)
+43 -14
View File
@@ -1,9 +1,10 @@
import { createLogger } from '@sim/logger'
import type { AccessRevokedBroadcast } from '@sim/realtime-protocol/events'
import { parseRoomName, ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { sleep } from '@sim/utils/helpers'
import type { AuthenticatedSocket } from '@/middleware/auth'
import { ROLE_REVALIDATION_TTL_MS, resolveCurrentWorkflowRole } from '@/middleware/permissions'
import type { IRoomManager } from '@/rooms'
import { type IRoomManager, workflowRoom as wf } from '@/rooms'
const logger = createLogger('AccessRevalidation')
@@ -65,9 +66,14 @@ interface ScanTarget {
* Collects this pod's authenticated sockets with the workflow room each has
* joined, in stable socket order.
*
* The workflow room is derived from the socket's own `rooms` set (pod-local, no
* Redis round-trips): a socket joins exactly one workflow room, so its rooms are
* `{ ownSocketId, workflowId }`. Only local sockets are evaluated — sockets are
* Rooms are derived from the socket's own `rooms` set (pod-local, no Redis
* round-trips). A socket may occupy several rooms of different types at once
* (workflow canvas, workspace-files browser, table, file-doc), all on the same
* io — so each name is decoded with {@link parseRoomName} and only **workflow**
* rooms are swept here. Non-workflow room names are namespaced (`type:id`) and
* resolve to a non-workflow type; sweeping them as workflow ids would resolve a
* bogus permission, come back `null`, and spuriously evict the socket from its
* files/table room every pass. Only local sockets are evaluated — sockets are
* sticky to a pod, so every socket is swept by exactly one pod using that pod's
* warm role cache (mirroring the per-pod reasoning of the write-path cache).
*/
@@ -78,7 +84,9 @@ function collectScanTargets(io: IRoomManager['io']): ScanTarget[] {
if (!authed.userId) continue
for (const room of socket.rooms) {
if (room === socket.id) continue
targets.push({ workflowId: room, socket: authed, userId: authed.userId })
const ref = parseRoomName(room)
if (ref?.type !== ROOM_TYPES.WORKFLOW) continue
targets.push({ workflowId: ref.id, socket: authed, userId: authed.userId })
}
}
return targets
@@ -129,9 +137,20 @@ export function startAccessRevalidationSweep(roomManager: IRoomManager): AccessR
async function cleanupEvictedSocket(socketId: string, workflowId: string): Promise<void> {
const key = `${socketId}:${workflowId}`
try {
// A fully-disconnected socket already had its presence removed by the
// disconnect handler (removeSocketFromAllRooms), so there is nothing left to
// clean. Dropping here also keeps the boolean removeUserFromRoom below from
// reporting a false "not a member" for an already-gone entry and retrying it
// forever (the pre-generalization manager returned the target on a no-op).
if (!io.sockets.sockets.get(socketId)) {
pendingCleanups.delete(key)
return
}
// Unlike removeUserFromRoom, this read does not swallow transport errors,
// so a Redis outage lands in the catch below and defers the cleanup.
const currentWorkflowId = await roomManager.getWorkflowIdForSocket(socketId)
const currentRoom = await roomManager.getRoomForSocket(socketId, ROOM_TYPES.WORKFLOW)
const currentWorkflowId = currentRoom?.id ?? null
if (currentWorkflowId !== null && currentWorkflowId !== workflowId) {
// The socket has since moved to a different workflow it can still
// access; that join's room switch already removed this room's presence
@@ -148,16 +167,26 @@ export function startAccessRevalidationSweep(roomManager: IRoomManager): AccessR
return
}
const removed = await roomManager.removeUserFromRoom(socketId, workflowId)
if (removed === null) {
// The sweep always passes the target room, and both managers report a
// performed removal by returning it — the Redis manager swallows
// transport errors into null, so null means the removal did not happen
// (even when the socket's mapping keys have already expired).
throw new Error('room-state removal not confirmed')
// A null mapping here is the normal case (the socket's mapping key may have
// expired) and does NOT mean "skip" — the eviction still removes the presence
// entry from the known target room via the explicit ref below.
const removed = await roomManager.removeUserFromRoom(wf(workflowId), socketId)
if (!removed) {
// `false` conflates two outcomes: the entry was already gone (a no-op), or a
// transport error the manager swallowed. Only retry when the socket is still mapped
// to THIS room — then a false result is a genuine, deferrable failure. When a healthy
// getRoomForSocket above returned no workflow mapping (`currentWorkflowId === null`),
// the presence entry is already gone, so the cleanup is complete: dropping it avoids
// re-enqueuing a still-connected socket forever. (A real Redis outage throws at
// getRoomForSocket and is deferred by the outer catch, never reaching here.)
if (currentWorkflowId === workflowId) {
throw new Error('room-state removal not confirmed')
}
pendingCleanups.delete(key)
return
}
await roomManager.broadcastPresenceUpdate(workflowId)
await roomManager.broadcastPresenceUpdate(wf(workflowId))
pendingCleanups.delete(key)
} catch (error) {
pendingCleanups.set(key, { socketId, workflowId })
+29
View File
@@ -0,0 +1,29 @@
import { db, user } from '@sim/db'
import { createLogger } from '@sim/logger'
import { eq } from 'drizzle-orm'
import type { AuthenticatedSocket } from '@/middleware/auth'
const logger = createLogger('PresenceAvatar')
/**
* The avatar URL for a presence entry: the socket's authenticated image when
* present, otherwise a single lookup of the user's stored image. Never throws —
* presence must not fail on an avatar lookup, so a DB error resolves to `null`.
*/
export async function resolveAvatarUrl(
socket: AuthenticatedSocket,
userId: string
): Promise<string | null> {
if (socket.userImage) return socket.userImage
try {
const [record] = await db
.select({ image: user.image })
.from(user)
.where(eq(user.id, userId))
.limit(1)
return record?.image ?? null
} catch (error) {
logger.warn('Failed to load user avatar for presence', { userId, error })
return null
}
}
+66 -8
View File
@@ -1,4 +1,6 @@
import { createLogger } from '@sim/logger'
import { parseRoomName, ROOM_TYPES, type RoomRef, roomName } from '@sim/realtime-protocol/rooms'
import { cleanupFileDocForSocket } from '@/handlers/file-doc'
import { cleanupPendingSubblocksForSocket } from '@/handlers/subblocks'
import { cleanupPendingVariablesForSocket } from '@/handlers/variables'
import type { AuthenticatedSocket } from '@/middleware/auth'
@@ -6,6 +8,14 @@ import type { IRoomManager } from '@/rooms'
const logger = createLogger('ConnectionHandlers')
/**
* Room types whose presence lives in the room manager (Redis-backed), so a disconnect must
* remove the socket + broadcast a correction. The workspace-files and file-doc rooms are
* NOT here: workspace-files carries no presence (native Socket.IO membership only), and
* file-doc broadcasts its own server-authenticated roster via `cleanupFileDocForSocket`.
*/
const PRESENCE_BEARING_TYPES = new Set<RoomRef['type']>([ROOM_TYPES.WORKFLOW, ROOM_TYPES.TABLE])
export function setupConnectionHandlers(socket: AuthenticatedSocket, roomManager: IRoomManager) {
socket.on('error', (error) => {
logger.error(`Socket ${socket.id} error:`, error)
@@ -15,20 +25,68 @@ export function setupConnectionHandlers(socket: AuthenticatedSocket, roomManager
logger.error(`Socket ${socket.id} connection error:`, error)
})
socket.on('disconnect', async (reason) => {
// `disconnecting` (not `disconnect`): here `socket.rooms` is still populated and
// authoritative, so presence is cleaned up even if the Redis room-set key was
// evicted or TTL-expired (which would leave the manager's stored rooms empty).
socket.on('disconnecting', async (reason) => {
try {
// Snapshot the live Socket.IO room membership SYNCHRONOUSLY, before any
// await: Socket.IO clears `socket.rooms` via leaveAll() as soon as the
// synchronous portion of this `disconnecting` handler returns (i.e. at the
// first await below), so reading it afterwards would see an empty set and
// the eviction fallback would be dead.
const liveRoomNames = [...socket.rooms]
// Clean up pending debounce entries for this socket to prevent memory leaks
cleanupPendingSubblocksForSocket(socket.id)
cleanupPendingVariablesForSocket(socket.id)
// Clear the socket's collaborative-document awareness (removes its caret for
// everyone else) and drop the room if it was the last editor. `endOfLife` drops the
// socket's join-generation entry — safe only here, on true disconnect (see cleanup).
cleanupFileDocForSocket(socket.id, roomManager.io, true)
const workflowIdHint = [...socket.rooms].find((roomId) => roomId !== socket.id)
const workflowId = await roomManager.removeUserFromRoom(socket.id, workflowIdHint)
// A socket may occupy multiple rooms (one per type). Remove it from every
// room the manager knows about.
const removedRooms = await roomManager.removeSocketFromAllRooms(socket.id)
if (workflowId) {
await roomManager.broadcastPresenceUpdate(workflowId)
logger.info(
`Socket ${socket.id} disconnected from workflow ${workflowId} (reason: ${reason})`
)
// Union with the snapshotted Socket.IO membership (authoritative, and it
// survives a Redis eviction/TTL lapse that would leave the manager's tracked
// rooms empty). Attempt removal for any room the manager didn't already
// remove — best-effort, since a transient Redis error can't be recovered here.
const wasInRooms = new Map<string, RoomRef>()
// Only presence-bearing rooms get a corrective broadcast. Manager-removed rooms are
// presence-bearing by construction today (only workflow/table write the socket→room hash),
// but filter symmetrically with the fallback path below so a future room type that ever
// tracks presence here can't emit a bogus presence-update no client listens to.
for (const room of removedRooms) {
if (PRESENCE_BEARING_TYPES.has(room.type)) wasInRooms.set(roomName(room), room)
}
for (const name of liveRoomNames) {
// `wasInRooms.has(name)` already excludes every room the manager removed (same
// room-name key via the roomName/parseRoomName bijection). Skip room types with no
// manager-tracked presence (workspace-files, file-doc): removing there is a no-op and
// broadcasting a correction would emit a dead presence-update no client listens to.
if (name === socket.id || wasInRooms.has(name)) continue
const ref = parseRoomName(name)
if (!ref || !PRESENCE_BEARING_TYPES.has(ref.type)) continue
wasInRooms.set(name, ref)
await roomManager.removeUserFromRoom(ref, socket.id)
}
// Broadcast a correction to every room this socket was in, EXCLUDING this
// socket — so it is never shown as a ghost collaborator even if its presence
// entry outlived a failed removal (transient Redis error; the hashes have no
// TTL). Any orphaned entry is additionally reclaimed by the next join's
// stale-presence sweep.
for (const room of wasInRooms.values()) {
await roomManager.broadcastPresenceUpdate(room, socket.id)
}
if (wasInRooms.size > 0) {
const rooms = Array.from(wasInRooms.values())
.map((room) => `${room.type}:${room.id}`)
.join(', ')
logger.info(`Socket ${socket.id} disconnected from [${rooms}] (reason: ${reason})`)
}
} catch (error) {
logger.error(`Error handling disconnect for socket ${socket.id}:`, error)
+127
View File
@@ -0,0 +1,127 @@
import { FILE_DOC_TIMEOUTS } from '@sim/realtime-protocol/file-doc'
import { env, getBaseUrl } from '@/env'
/**
* The relay's client for the app's internal file-doc endpoints. The app owns the markdown↔Yjs
* conversion engine (TipTap + jsdom) and blob/DB access; the relay owns the live document. So for any
* operation that needs conversion, the relay delegates here over the shared `x-api-key` channel. The
* timeouts (and their ordering vs. the app-side bounds) live in the shared `FILE_DOC_TIMEOUTS`.
*/
function postToApp(path: string, payload: unknown, timeoutMs: number): Promise<Response> {
return fetch(`${getBaseUrl()}${path}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'x-api-key': env.INTERNAL_API_SECRET },
body: JSON.stringify(payload),
signal: AbortSignal.timeout(timeoutMs),
})
}
/**
* Ask the app to build a server-authoritative seed (markdown → Yjs) for a file's collaborative
* document. Returns the Yjs update to apply, or `null` for a genuinely empty/missing file (an empty
* document is correct). THROWS on a transport failure (non-2xx / network / timeout / malformed body)
* so the caller can tell a real empty from a failure it should be allowed to retry.
*/
export async function fetchFileDocSeed(
workspaceId: string,
fileId: string
): Promise<{ update: Uint8Array; version: number } | null> {
const response = await postToApp(
'/api/internal/file-doc/seed',
{ workspaceId, fileId },
FILE_DOC_TIMEOUTS.seedRequestMs
)
if (!response.ok) {
throw new Error(`Seed fetch failed for file ${fileId}: ${response.status}`)
}
const body = (await response.json()) as { update?: unknown; version?: unknown }
const update = body?.update
// A well-formed response is `{ update: base64-string | null, version: number | null }`. Anything
// else is a contract violation, not a "genuinely empty file" — throw so the caller retries rather
// than silently treating a malformed body as empty and stranding the room unseeded.
if (update === null) return null
if (typeof update !== 'string' || typeof body?.version !== 'number') {
throw new Error(`Seed fetch for file ${fileId} returned a malformed body`)
}
return { update: new Uint8Array(Buffer.from(update, 'base64')), version: body.version }
}
/**
* Ask the app to merge new markdown into the live document as a minimal Yjs diff — Stage C, so a
* copilot edit streams into open editors instead of the file changing underneath them. The relay
* ships the document's current state and applies the returned diff (which Yjs reconciles with any
* concurrent user edits). THROWS on a transport failure or malformed body.
*/
export async function fetchFileDocMerge(
fileId: string,
docState: Uint8Array,
markdown: string
): Promise<Uint8Array> {
const response = await postToApp(
'/api/internal/file-doc/merge',
{ fileId, docState: Buffer.from(docState).toString('base64'), markdown },
FILE_DOC_TIMEOUTS.mergeRequestMs
)
if (!response.ok) {
throw new Error(`Merge fetch failed for file ${fileId}: ${response.status}`)
}
const body = (await response.json()) as { update?: unknown }
if (typeof body?.update !== 'string') {
throw new Error(`Merge fetch for file ${fileId} returned a malformed body`)
}
return new Uint8Array(Buffer.from(body.update, 'base64'))
}
/**
* Result of a persist attempt (mirrors the app's `persistFileDoc` contract):
* - `persisted` — written; `version` is the new durable version the relay records as synced.
* - `missing` — the file is gone.
* - `conflict` — the file changed out-of-band since `expectedVersion`; NOT written. `version` is the
* current durable version the relay adopts as its new If-Match to re-persist the current live stream.
*/
export type PersistResult =
| { status: 'persisted'; version: number }
| { status: 'missing' }
| { status: 'conflict'; version: number }
| { status: 'deferred' }
/**
* Ask the app to project a live collaborative document back to durable markdown and write it to the
* file (Yjs → markdown, through the exact editor engine) — the server-authoritative durable path that
* replaces the editor's client autosave. `expectedVersion` (the durable version the live doc synced
* from) is the optimistic-concurrency guard: on a mismatch the app returns `conflict` (rather than
* clobbering) so the caller reconciles and retries. THROWS only on a transport/contract failure.
*/
export async function fetchFileDocPersist(
workspaceId: string,
fileId: string,
userId: string,
docState: Uint8Array,
expectedVersion?: number
): Promise<PersistResult> {
const response = await postToApp(
'/api/internal/file-doc/persist',
{
workspaceId,
fileId,
userId,
docState: Buffer.from(docState).toString('base64'),
...(expectedVersion !== undefined ? { expectedVersion } : {}),
},
FILE_DOC_TIMEOUTS.persistRequestMs
)
if (!response.ok) {
throw new Error(`Persist failed for file ${fileId}: ${response.status}`)
}
const body = (await response.json()) as PersistResult
if (
body?.status !== 'persisted' &&
body?.status !== 'missing' &&
body?.status !== 'conflict' &&
body?.status !== 'deferred'
) {
throw new Error(`Persist for file ${fileId} returned a malformed body`)
}
return body
}
@@ -0,0 +1,490 @@
/**
* @vitest-environment node
*/
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import * as Y from 'yjs'
/**
* One shared in-memory Redis backing per test, so several {@link FileDocStore} instances (modelling
* several ECS tasks) all talk to the "same Redis". A minimal fake of just the stream/lock ops the
* store uses.
*/
interface Backing {
streams: Map<string, { id: string; message: Record<string, string> }[]>
kv: Map<string, string>
seq: number
/** Number of upcoming xAdd calls to fail with a transient error (to exercise publish retry). */
failXAdd: number
}
const state = vi.hoisted(() => ({ backing: null as Backing | null }))
const seqOf = (id: string) => Number(id.split('-')[0])
function makeClient(): any {
const b = () => {
if (!state.backing) throw new Error('backing not initialized')
return state.backing
}
const client: any = {
connect: async () => {},
quit: async () => {},
on: () => client,
duplicate: () => makeClient(),
xAdd: async (key: string, _star: string, fields: Record<string, string>) => {
if (b().failXAdd > 0) {
b().failXAdd--
throw new Error('transient xAdd failure')
}
const id = `${++b().seq}-0`
const arr = b().streams.get(key) ?? []
arr.push({ id, message: { ...fields } })
b().streams.set(key, arr)
return id
},
xRange: async (key: string) => (b().streams.get(key) ?? []).map((e) => ({ ...e })),
xLen: async (key: string) => (b().streams.get(key) ?? []).length,
xTrim: async (key: string, _strategy: string, minid: string) => {
const arr = b().streams.get(key) ?? []
b().streams.set(
key,
arr.filter((e) => seqOf(e.id) >= seqOf(minid))
)
},
xRead: async (streams: { key: string; id: string }[]) => {
const res: { name: string; messages: { id: string; message: Record<string, string> }[] }[] =
[]
for (const { key, id } of streams) {
const after = (b().streams.get(key) ?? []).filter((e) => seqOf(e.id) > seqOf(id))
if (after.length) res.push({ name: key, messages: after.map((e) => ({ ...e })) })
}
if (res.length) return res
await new Promise((r) => setTimeout(r, 5))
return null
},
set: async (key: string, val: string, opts?: { NX?: boolean }) => {
if (opts?.NX && b().kv.has(key)) return null
b().kv.set(key, val)
return 'OK'
},
del: async (key: string) => {
b().kv.delete(key)
return 1
},
eval: async (script: string, opts: { keys: string[]; arguments: string[] }) => {
const [key] = opts.keys
// Atomic seed-if-empty (SEED_IF_EMPTY_SCRIPT): append the entry iff the stream is empty, in one
// synchronous step — mirroring Redis's atomic Lua execution, so two concurrent evals can never both
// append (the second sees a non-empty stream).
if (script.includes('xlen')) {
const [field, value] = opts.arguments
const arr = b().streams.get(key) ?? []
if (arr.length > 0) return 0
const id = `${++b().seq}-0`
arr.push({ id, message: { [field]: value } })
b().streams.set(key, arr)
return 1
}
// Compare-and-delete Lua (RELEASE_LOCK_SCRIPT): del only if the stored value matches the token.
const [token] = opts.arguments
if (b().kv.get(key) === token) {
b().kv.delete(key)
return 1
}
return 0
},
expire: async () => 1,
}
return client
}
vi.mock('redis', () => ({ createClient: () => makeClient() }))
import { FileDocStore, REDIS_AGENT_ORIGIN, REDIS_ORIGIN } from '@/handlers/file-doc-store'
const REDIS_URL = 'redis://fake'
const NAME = 'workspace-file-doc:file-1'
function docWithText(text: string): Y.Doc {
const doc = new Y.Doc()
doc.getText('body').insert(0, text)
return doc
}
/** The delta a doc emits when `text` is inserted — what the relay would `publish`. */
function updateFor(text: string): Uint8Array {
const doc = docWithText(text)
const update = Y.encodeStateAsUpdate(doc)
doc.destroy()
return update
}
let stores: FileDocStore[] = []
async function newStore(): Promise<FileDocStore> {
const store = new FileDocStore(REDIS_URL)
await store.init()
stores.push(store)
return store
}
describe('FileDocStore', () => {
beforeEach(() => {
state.backing = { streams: new Map(), kv: new Map(), seq: 0, failXAdd: 0 }
stores = []
})
afterEach(async () => {
await Promise.all(stores.map((s) => s.shutdown()))
})
it('elects exactly one seeder across tasks (no split-brain seed)', async () => {
const a = await newStore()
const b = await newStore()
// shouldSeed returns a lock token (truthy) for the winner, null for the loser.
const [aTok, bTok] = await Promise.all([a.shouldSeed(NAME), b.shouldSeed(NAME)])
expect([aTok, bTok].filter(Boolean)).toHaveLength(1)
})
it('does not re-seed once the stream already has content (stale lock)', async () => {
const a = await newStore()
const token = await a.shouldSeed(NAME)
expect(token).toBeTruthy()
// A seeds and releases its lock.
a.publish(NAME, updateFor('hello'))
await vi.waitFor(async () => expect(await a.getStreamState(NAME)).not.toBeNull())
await a.releaseSeedLock(NAME, token as string)
// A different task must NOT seed again — the lock is free but the stream is non-empty.
const b = await newStore()
expect(await b.shouldSeed(NAME)).toBeNull()
})
it('getStreamState reconstructs the shared document from the stream', async () => {
const a = await newStore()
a.publish(NAME, updateFor('shared content'))
let state: Uint8Array | null = null
await vi.waitFor(async () => {
state = await a.getStreamState(NAME)
expect(state).not.toBeNull()
})
const doc = new Y.Doc()
Y.applyUpdate(doc, state!)
expect(doc.getText('body').toString()).toBe('shared content')
doc.destroy()
})
it('attachRoom catches a fresh task up to the current shared state', async () => {
const a = await newStore()
a.publish(NAME, updateFor('already here'))
await vi.waitFor(async () => expect(await a.getStreamState(NAME)).not.toBeNull())
// A second task opens the same file: its doc must load the existing content, not start empty.
const b = await newStore()
const doc = new Y.Doc()
await b.attachRoom(NAME, doc)
expect(doc.getText('body').toString()).toBe('already here')
doc.destroy()
})
it('converges a peer task via the tailer after attach', async () => {
const a = await newStore()
const b = await newStore()
const bDoc = new Y.Doc()
await b.attachRoom(NAME, bDoc)
// A publishes an edit; B's multiplexed reader must apply it to B's attached doc.
a.publish(NAME, updateFor('from task A'))
await vi.waitFor(() => expect(bDoc.getText('body').toString()).toBe('from task A'), {
timeout: 2000,
})
bDoc.destroy()
})
it('compaction never trims peer entries the compacting task has not yet integrated', async () => {
const streamKey = `filedoc:stream:${NAME}`
const noop = Buffer.from(Y.encodeStateAsUpdate(new Y.Doc())).toString('base64')
// Two peer edits published by ANOTHER task that this task's tailer has not read yet.
const peerDoc = new Y.Doc()
const peerUpdates: Uint8Array[] = []
peerDoc.on('update', (u: Uint8Array) => peerUpdates.push(u))
peerDoc.getText('body').insert(0, 'PEER1')
peerDoc.getText('body').insert(5, 'PEER2')
// Backing: 400 already-integrated (no-op) entries this task's doc reflects, then the 2 un-integrated
// peer entries. Enough entries to cross COMPACT_THRESHOLD.
const entries = Array.from({ length: 400 }, (_, i) => ({
id: `${i + 1}-0`,
message: { u: noop },
}))
entries.push({ id: '401-0', message: { u: Buffer.from(peerUpdates[0]).toString('base64') } })
entries.push({ id: '402-0', message: { u: Buffer.from(peerUpdates[1]).toString('base64') } })
state.backing!.streams.set(streamKey, entries)
state.backing!.seq = 402
const a = await newStore()
// This task has integrated only up to entry 400 (all no-ops) — its local doc is empty and lags the
// two peer entries. Inject that lagging room directly (a real edit was integrated → realEdited).
;(a as any).rooms.set(NAME, {
doc: new Y.Doc(),
lastId: '400-0',
publishes: 0,
seededObserved: true,
realEdited: true,
})
await (a as any).maybeCompact(NAME)
// A fresh catch-up must still reconstruct the peer content — compaction must not have trimmed 401/402.
const doc = new Y.Doc()
Y.applyUpdate(doc, (await a.getStreamState(NAME))!)
expect(doc.getText('body').toString()).toBe('PEER1PEER2')
doc.destroy()
// The appended snapshot entry must carry the snapshot marker, so a fresh catch-up task treats it as
// edited content (not a bare seed) and persists on last-disconnect.
const stream = state.backing!.streams.get(streamKey)!
expect(stream[stream.length - 1].message.s).toBe('1')
})
it('tags an agent-streamed frame so a peer tailer applies it as REDIS_AGENT_ORIGIN (never persisted)', async () => {
const streamKey = `filedoc:stream:${NAME}`
const a = await newStore()
const b = await newStore()
const bDoc = new Y.Doc()
// Capture the origin the tailer stamps each applied entry with — the persistence gate keys off it.
const origins: unknown[] = []
bDoc.on('update', (_u: Uint8Array, origin: unknown) => origins.push(origin))
await b.attachRoom(NAME, bDoc)
// A normal edit tails as REDIS_ORIGIN (a peer edit that CAN be persisted).
a.publish(NAME, updateFor('user edit'))
await vi.waitFor(() => expect(origins).toContain(REDIS_ORIGIN), { timeout: 2000 })
// An agent-streamed frame is published WITH the agent flag: the stream entry carries the marker, and
// the peer tailer applies it as REDIS_AGENT_ORIGIN — excluded from the relay's edited/persist gate.
a.publish(NAME, updateFor('agent frame'), true)
await vi.waitFor(() => expect(origins).toContain(REDIS_AGENT_ORIGIN), { timeout: 2000 })
const stream = state.backing!.streams.get(streamKey)!
expect(stream.some((e) => e.message.a === '1')).toBe(true)
// The normal edit's entry carries no agent marker.
expect(stream.filter((e) => e.message.a === '1')).toHaveLength(1)
bDoc.destroy()
})
it('latches realEdited synchronously so a concurrent compaction can never mislabel a real edit', async () => {
// The data-loss race: a real edit sits in room.doc synchronously, but if realEdited were set only
// AFTER appendUpdate's awaits, a concurrent agent-triggered compaction could snapshot that content and
// stamp it an agent (no-persist) frame — losing the edit. The latch must be set in the same tick.
const a = await newStore()
const doc = new Y.Doc()
await a.attachRoom(NAME, doc)
const room = (a as any).rooms.get(NAME)
expect(room.realEdited).toBe(false)
// Kick off a real (non-agent) append but do NOT await it: realEdited must already be true before the
// xAdd/expire awaits resolve, so any compaction racing on the awaits sees the real edit.
const pending = (a as any).appendUpdate(NAME, updateFor('real user edit'))
expect(room.realEdited).toBe(true)
await pending
doc.destroy()
})
it('stamps a compaction snapshot of an agent-ONLY stream as an agent frame (never persisted)', async () => {
const streamKey = `filedoc:stream:${NAME}`
const noop = Buffer.from(Y.encodeStateAsUpdate(new Y.Doc())).toString('base64')
// A doc whose content is purely agent preview (no real edit integrated) — realEdited stays false.
const agentDoc = docWithText('agent-only preview body')
const entries = Array.from({ length: 400 }, (_, i) => ({
id: `${i + 1}-0`,
message: { u: noop },
}))
state.backing!.streams.set(streamKey, entries)
state.backing!.seq = 400
const a = await newStore()
;(a as any).rooms.set(NAME, {
doc: agentDoc,
lastId: '400-0',
publishes: 0,
seededObserved: true,
realEdited: false,
})
await (a as any).maybeCompact(NAME)
// The snapshot must carry the AGENT marker, NOT the snapshot marker, so a peer catch-up applies it as
// REDIS_AGENT_ORIGIN and never marks the doc edited — the no-persist guarantee survives compaction.
const stream = state.backing!.streams.get(streamKey)!
const last = stream[stream.length - 1].message
expect(last.a).toBe('1')
expect(last.s).toBeUndefined()
// Content is still fully reconstructable from the compacted stream.
const doc = new Y.Doc()
Y.applyUpdate(doc, (await a.getStreamState(NAME))!)
expect(doc.getText('body').toString()).toBe('agent-only preview body')
doc.destroy()
agentDoc.destroy()
})
it('retries a transient append failure so the edit is not lost from the shared log', async () => {
const a = await newStore()
state.backing!.failXAdd = 2 // first two xAdd attempts throw; the third must succeed
a.publish(NAME, updateFor('resilient'))
await vi.waitFor(
async () => {
const doc = new Y.Doc()
Y.applyUpdate(doc, (await a.getStreamState(NAME))!)
expect(doc.getText('body').toString()).toBe('resilient')
doc.destroy()
},
{ timeout: 2000 }
)
})
it('streamHasContent fences a seed apply against an already-seeded stream', async () => {
const a = await newStore()
expect(await a.streamHasContent(NAME)).toBe(false)
a.publish(NAME, updateFor('seeded'))
await vi.waitFor(async () => expect(await a.streamHasContent(NAME)).toBe(true))
})
it('serializes merges across tasks via the merge lock', async () => {
const a = await newStore()
const b = await newStore()
const aTok = await a.acquireMergeSlot(NAME, 5_000)
expect(aTok).toBeTruthy()
// A holds it → B is refused until A releases.
expect(await b.acquireMergeSlot(NAME, 5_000)).toBeNull()
// A stale-holder release with the WRONG token must NOT free A's lock (compare-and-delete).
await b.releaseMergeSlot(NAME, 'wrong-token')
expect(await b.acquireMergeSlot(NAME, 5_000)).toBeNull()
// A releases with its real token → B can now acquire.
await a.releaseMergeSlot(NAME, aTok as string)
const bTok = await b.acquireMergeSlot(NAME, 5_000)
expect(bTok).toBeTruthy()
await b.releaseMergeSlot(NAME, bTok as string)
})
it('is disabled without a REDIS_URL and behaves single-replica', async () => {
const store = new FileDocStore(undefined)
expect(store.enabled).toBe(false)
// Seeds locally (returns a sentinel token), never touches a stream.
expect(await store.shouldSeed(NAME)).toBeTruthy()
expect(await store.getStreamState(NAME)).toBeNull()
const doc = new Y.Doc()
await store.attachRoom(NAME, doc) // no-op, no throw
expect(doc.getText('body').toString()).toBe('')
doc.destroy()
})
it('seedIfEmpty writes the seed once and reports it, then refuses a non-empty stream', async () => {
const a = await newStore()
expect(await a.seedIfEmpty(NAME, updateFor('first'))).toBe(true)
// A second seed attempt (any task) must be refused — the stream already holds content.
const b = await newStore()
expect(await b.seedIfEmpty(NAME, updateFor('second'))).toBe(false)
const doc = new Y.Doc()
Y.applyUpdate(doc, (await a.getStreamState(NAME))!)
expect(doc.getText('body').toString()).toBe('first')
doc.destroy()
})
it('atomic seed prevents split-brain even when the seed lock expired mid-seed', async () => {
// The exact split-brain precondition from the concurrency audit: the seed lock is only an efficiency
// optimization, so if it lapses (TTL) while the stream is still empty, TWO tasks can both hold a
// token and both try to seed with DIFFERENT docs (distinct Yjs client ids). The atomic seedIfEmpty
// must still let only one land — otherwise the union duplicates content.
const a = await newStore()
const b = await newStore()
const tokenA = await a.shouldSeed(NAME)
expect(tokenA).toBeTruthy()
// Simulate A's lock expiring mid-seed so B also wins the freed lock over a still-empty stream.
state.backing!.kv.delete(`filedoc:seedlock:${NAME}`)
const tokenB = await b.shouldSeed(NAME)
expect(tokenB).toBeTruthy()
// Both tasks now race to seed with distinct client ids.
const [seededA, seededB] = await Promise.all([
a.seedIfEmpty(NAME, updateFor('SEED-A')),
b.seedIfEmpty(NAME, updateFor('SEED-B')),
])
expect([seededA, seededB].filter(Boolean)).toHaveLength(1)
// Exactly one seed is in the stream — the reconstructed text is a single seed, never a duplicated
// union of both (e.g. 'SEED-ASEED-B').
const doc = new Y.Doc()
Y.applyUpdate(doc, (await a.getStreamState(NAME))!)
expect(['SEED-A', 'SEED-B']).toContain(doc.getText('body').toString())
doc.destroy()
})
it('a peer edit published during attachRoom catch-up is not lost', async () => {
// Author two INCREMENTAL edits from one doc so they converge to 'basepeer' (not an independent union).
const author = new Y.Doc()
const updates: Uint8Array[] = []
author.on('update', (u: Uint8Array) => updates.push(u))
author.getText('body').insert(0, 'base')
author.getText('body').insert(4, 'peer')
const a = await newStore()
a.publish(NAME, updates[0]) // 'base'
await vi.waitFor(async () => expect(await a.getStreamState(NAME)).not.toBeNull())
// Task B attaches; while its synchronous catch-up runs, task A publishes the second edit. The tailer
// resumes from the id catch-up stopped at, so the edit converges rather than falling into a gap.
const b = await newStore()
const bDoc = new Y.Doc()
const attach = b.attachRoom(NAME, bDoc)
a.publish(NAME, updates[1]) // 'peer' appended
await attach
await vi.waitFor(() => expect(bDoc.getText('body').toString()).toBe('basepeer'), {
timeout: 2000,
})
bDoc.destroy()
author.destroy()
})
it('concurrent compaction on two tasks preserves the full document', async () => {
const streamKey = `filedoc:stream:${NAME}`
const noop = Buffer.from(Y.encodeStateAsUpdate(new Y.Doc())).toString('base64')
// Two peer edits neither compacting task has integrated (id > each task's lastId).
const peerDoc = new Y.Doc()
const peerUpdates: Uint8Array[] = []
peerDoc.on('update', (u: Uint8Array) => peerUpdates.push(u))
peerDoc.getText('body').insert(0, 'PEER1')
peerDoc.getText('body').insert(5, 'PEER2')
const entries = Array.from({ length: 400 }, (_, i) => ({
id: `${i + 1}-0`,
message: { u: noop },
}))
entries.push({ id: '401-0', message: { u: Buffer.from(peerUpdates[0]).toString('base64') } })
entries.push({ id: '402-0', message: { u: Buffer.from(peerUpdates[1]).toString('base64') } })
state.backing!.streams.set(streamKey, entries)
state.backing!.seq = 402
// Two tasks whose local docs lag at DIFFERENT points (400 and 401): both cross the threshold and
// compact concurrently. Each must only trim what its own snapshot subsumes, so the union of both
// snapshots plus the un-integrated peer entries still reconstructs the whole doc.
const a = await newStore()
const b = await newStore()
const docA = new Y.Doc()
Y.applyUpdate(docA, peerUpdates[0]) // A integrated up to 401
;(a as any).rooms.set(NAME, {
doc: docA,
lastId: '401-0',
publishes: 0,
seededObserved: true,
realEdited: true,
})
;(b as any).rooms.set(NAME, {
doc: new Y.Doc(),
lastId: '400-0',
publishes: 0,
seededObserved: true,
realEdited: true,
})
await Promise.all([(a as any).maybeCompact(NAME), (b as any).maybeCompact(NAME)])
const doc = new Y.Doc()
Y.applyUpdate(doc, (await a.getStreamState(NAME))!)
expect(doc.getText('body').toString()).toBe('PEER1PEER2')
doc.destroy()
})
})
@@ -0,0 +1,725 @@
/**
* Shared, multi-replica Yjs backend for the collaborative file-document relay, over Redis Streams.
*
* The relay keeps an in-memory {@link Y.Doc} per open file (for the sync handshake, awareness, and
* copilot merges), but on a horizontally-scaled deployment (multiple ECS tasks, autoscaling) that
* per-process doc is NOT authoritative on its own: two tasks each seeding the same file from markdown
* would mint independent Yjs client ids and union into duplicated content (split-brain), and a task
* only ever sees the edits of ITS OWN clients. This module makes every task converge on ONE CRDT per
* file by treating a Redis Stream as the shared, ordered, replayable log of Yjs updates — the union of
* a stream's entries IS the document. It is the "shared Yjs backend (y-redis / Hocuspocus)" the relay's
* single-replica model always deferred, built natively for our Socket.IO transport on the Redis the
* Socket.IO adapter already runs.
*
* How it fits the relay's message flow (see `file-doc.ts`):
* - Doc-sync messages no longer ride the Socket.IO Redis ADAPTER cross-pod. Instead each applied
* update is {@link publish}ed to the stream; every task's multiplexed reader
* applies it to its local doc (origin {@link REDIS_ORIGIN}) and fans it out to ITS OWN clients. So a
* client receives each update exactly once, from its own task's local broadcast — no adapter
* amplification, and every task's doc stays converged. (Awareness/presence stay on the adapter: they
* are ephemeral and need no convergence or replay.)
* - {@link attachRoom} does a synchronous catch-up read from the head of the stream when a task first
* opens a file, so a late-joining task (the normal case under autoscaling) loads the current shared
* state before its first client syncs. Catch-up + tail are seamless: the tailer resumes from the
* exact id catch-up stopped at.
* - The one-time seed is written via the atomic {@link seedIfEmpty} (append-iff-empty in one Redis
* step), so exactly one task ever writes the seed cluster-wide (the fix for split-brain) — even if two
* tasks race. {@link shouldSeed} is a Redis lock + empty-stream check layered on top ONLY as an
* efficiency gate (so tasks don't all run the seed fetch); correctness does not depend on it.
*
* When `REDIS_URL` is unset (single-pod dev) the store is DISABLED and every method degrades to the
* relay's original single-replica behavior: seed locally, no stream, no tailer.
*
* @module
*/
import { createLogger } from '@sim/logger'
import { FILE_DOC_SEED, FILE_DOC_TIMEOUTS } from '@sim/realtime-protocol/file-doc'
import { getErrorMessage } from '@sim/utils/errors'
import { sleep } from '@sim/utils/helpers'
import { generateId } from '@sim/utils/id'
import { backoffWithJitter } from '@sim/utils/retry'
import { createClient, type RedisClientType } from 'redis'
import * as Y from 'yjs'
const logger = createLogger('FileDocStore')
/**
* Compare-and-delete: release a lock ONLY if this task still holds it (its token still the value), so a
* lock that expired and was re-acquired by another task is never stolen by the original holder's release.
*/
const RELEASE_LOCK_SCRIPT =
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end"
/**
* Atomic seed: append the seed entry ONLY if the stream is still empty, in one Redis-side step. This is
* the real split-brain guard — two tasks racing (even both past an expired seed lock) can never both
* write a seed (each would mint a distinct Yjs client id → duplicated content), because the emptiness
* check and the append happen atomically with no check-then-append window. The seed lock is only an
* efficiency optimization (avoid two seed fetches); correctness does not depend on it staying held.
* Returns 1 if THIS call wrote the seed, 0 if the stream already had content.
*/
const SEED_IF_EMPTY_SCRIPT =
"if redis.call('xlen', KEYS[1]) == 0 then redis.call('xadd', KEYS[1], '*', ARGV[1], ARGV[2]); return 1 else return 0 end"
/**
* Monotonic set of the synced-version token: overwrite ONLY when the new value is greater than the
* stored one (or none is stored). The token is written fire-and-forget from multiple sites (seed stamp,
* merge, persist) and across tasks, so an out-of-order write must never REGRESS it to a version older
* than the live doc already incorporates — a regressed token causes spurious If-Match conflicts and, on a
* last-leave flush with no live room to reconcile into, a lost persist. Refreshes the TTL on both paths
* so a write skipped as older still keeps the (higher) value alive. Versions are monotonic epoch-ms,
* comfortably within a Lua double, so the numeric compare is exact.
*/
const SET_VERSION_IF_NEWER_SCRIPT =
"local c = redis.call('get', KEYS[1]); if c == false or tonumber(c) < tonumber(ARGV[1]) then redis.call('set', KEYS[1], ARGV[1], 'EX', ARGV[2]) else redis.call('expire', KEYS[1], ARGV[2]) end; return 1"
/**
* The transaction origin the store stamps on updates it applies from the stream. The relay's
* `doc.on('update')` handler uses it to distinguish an update that ARRIVED from a peer (fan out to
* local clients, but do NOT re-publish — it is already in the stream) from a local edit (fan out AND
* publish). It must be a non-string sentinel so it is never mistaken for a socket id.
*/
export const REDIS_ORIGIN = Symbol('file-doc-redis')
/**
* Origin for a COMPACTED SNAPSHOT applied from the stream. A snapshot folds the seed + all prior edits
* into one entry, so a fresh task catching up from it would otherwise never see a separate post-seed
* edit frame and would treat the doc as unedited. The relay's edit-tracker uses this origin to mark the
* doc edited (a snapshot only exists after the stream crossed the compaction threshold, i.e. real edits
* happened). Behaves like {@link REDIS_ORIGIN} otherwise (already in the stream — never re-published).
*/
export const REDIS_SNAPSHOT_ORIGIN = Symbol('file-doc-redis-snapshot')
/**
* Origin for an AGENT-STREAMED frame applied from the stream (a copilot output token relayed via
* {@link FILE_DOC_MESSAGE_TYPE.SYNC_NO_PERSIST}). A peer task tails these to stay live mid-stream, but
* they are transient preview content the copilot's durable `edit_content` write reconciles — so the
* relay's edit-tracker must NOT mark the doc edited on them (a startup-race duplicate between two stream
* leaders would otherwise become eligible for a peer task's persist). Behaves like {@link REDIS_ORIGIN}
* otherwise (already in the stream — never re-published).
*/
export const REDIS_AGENT_ORIGIN = Symbol('file-doc-redis-agent')
const STREAM_PREFIX = 'filedoc:stream:'
/** Cluster-wide "durable version the live doc is synced to" (the persist If-Match token). */
const SYNC_VERSION_PREFIX = 'filedoc:syncver:'
const SEED_LOCK_PREFIX = 'filedoc:seedlock:'
const COMPACT_LOCK_PREFIX = 'filedoc:compactlock:'
const PERSIST_LOCK_PREFIX = 'filedoc:persistlock:'
const MERGE_LOCK_PREFIX = 'filedoc:mergelock:'
/** Cluster-wide "a client is actively streaming an agent edit into this live doc" flag — set (refreshed)
* on every agent frame so a durable {@link applyMarkdownToLiveFileDoc} merge defers to that client
* (which is applying the same content) instead of double-writing it. Short-TTL'd so it self-clears the
* moment streaming stops, after which the final durable merge lands as a near-noop. */
const AGENT_STREAM_PREFIX = 'filedoc:agentstream:'
/** The field each stream entry carries — a base64 Yjs update. */
const UPDATE_FIELD = 'u'
/** Marks a stream entry as a compaction SNAPSHOT (folds seed + edits), so the tailer applies it with
* {@link REDIS_SNAPSHOT_ORIGIN}. Present only on snapshot entries. */
const SNAPSHOT_FIELD = 's'
/** Marks a stream entry as an AGENT-STREAMED preview frame, so the tailer applies it with
* {@link REDIS_AGENT_ORIGIN} (never marks the doc edited). Present only on agent-frame entries. */
const AGENT_FIELD = 'a'
/** Sentinel token a DISABLED store returns from a lock acquire, so single-replica callers proceed
* without special-casing; {@link FileDocStore.releaseLock} treats it as a no-op. Not a real UUID, so it
* can never collide with a {@link generateId} token. */
const DISABLED_LOCK_TOKEN = '__disabled__'
/** How long a blocking multiplexed read waits before re-snapshotting the live room set. Also bounds
* how long a room attached mid-block waits for its first cross-task update (updates are not lost — the
* next read resumes from its last id — only briefly delayed). */
const READ_BLOCK_MS = 1_000
/** Idle poll cadence when NO room is open on this task, so a freshly-attached room is picked up fast
* without busy-spinning an empty task. */
const IDLE_POLL_MS = 250
/** Max entries drained per stream per read. */
const READ_COUNT = 200
/** Compact a stream once it exceeds this many entries (snapshot + trim). */
const COMPACT_THRESHOLD = 400
/** Check whether compaction is due only every Nth local publish, to avoid an XLEN per keystroke. */
const COMPACT_CHECK_EVERY = 64
/** Compaction critical section (snapshot + xAdd + xTrim) is fast; a generous TTL covers a slow Redis
* round-trip without risking expiry mid-compact. Released via compare-and-delete regardless. */
const COMPACT_LOCK_TTL_MS = 10_000
/** Retry a failed stream append this many times before giving up, so a transient Redis blip doesn't
* silently drop an edit from the shared log (which no peer would then ever see). */
const PUBLISH_MAX_RETRIES = 3
/** The seed lock spans the app seed fetch (hard-bounded at `seedRequestMs = 8s`) + the atomic seed
* append. It is only an EFFICIENCY optimization — it stops two tasks both running the seed fetch — and is
* sized to comfortably exceed the fetch bound while staying near the client readiness deadline (12s) so a
* dead seeder's lock frees when clients would recover anyway. Double-seed is prevented even if the lock
* expires mid-seed, because the seed is written via the atomic {@link SEED_IF_EMPTY_SCRIPT}
* (append-iff-empty), NOT the lock — correctness never depends on the lock staying held. */
const SEED_LOCK_TTL_MS = FILE_DOC_TIMEOUTS.seedRequestMs + 4_000
/** How long a stream survives with no heartbeat — long enough that an occupied-but-idle doc never
* loses its shared state (the heartbeat refreshes it while any task holds the room). */
const STREAM_TTL_SEC = 600
/** Refresh every occupied stream's TTL on this cadence, so a live doc's stream never expires. */
const HEARTBEAT_MS = 60_000
const streamKey = (name: string) => `${STREAM_PREFIX}${name}`
/**
* Decode one stream entry's base64 Yjs update and apply it to `doc`. A malformed entry is logged and
* SKIPPED — never thrown — so one bad frame can neither wedge the tailer nor abort a headless
* stream-fold. Shared by the tailer/catch-up (applies with {@link REDIS_ORIGIN}) and the merge-base
* reconstruction (no origin — a throwaway doc), so the two can never diverge on how an entry is read.
*/
function applyEntryToDoc(
doc: Y.Doc,
id: string,
message: Record<string, string>,
origin?: unknown
): void {
const encoded = message[UPDATE_FIELD]
if (!encoded) return
try {
Y.applyUpdate(doc, new Uint8Array(Buffer.from(encoded, 'base64')), origin)
} catch (error) {
logger.warn('FileDocStore dropping malformed stream entry', {
id,
error: getErrorMessage(error),
})
}
}
/** Whether a doc carries the seed flag (mirrors the relay's `isDocSeeded`), so the store can tell the
* one-time seed transition from a real post-seed edit without re-implementing the check divergently. */
function isDocSeeded(doc: Y.Doc): boolean {
return doc.getMap(FILE_DOC_SEED.configMap).get(FILE_DOC_SEED.flag) === true
}
/** One locally-open room the store tracks: its doc and the last stream id applied to it. */
interface StoreRoom {
doc: Y.Doc
/** The id of the last stream entry applied to `doc`; the tailer resumes strictly after it. */
lastId: string
/** Local publish count, to pace compaction checks. */
publishes: number
/** Set once the doc has been observed seeded, so the seed transition itself is never mistaken for an
* edit (mirrors the relay's `seededObserved`). */
seededObserved: boolean
/** Whether the doc has integrated any REAL (non-agent, non-seed) edit. Compaction stamps its snapshot
* as an AGENT snapshot ({@link REDIS_AGENT_ORIGIN}, never persisted) until this is true, so a long
* agent-only stream that crosses the compaction threshold can't fold its preview content into a
* snapshot that marks peers edited. */
realEdited: boolean
}
/**
* The Redis-Streams shared Yjs backend. A single instance per process. `enabled` is false when there
* is no `REDIS_URL`, in which case every method is a no-op and the relay runs single-replica.
*/
export class FileDocStore {
readonly enabled: boolean
/** Command connection: XADD / locks / XLEN / XTRIM / EXPIRE. */
private write: RedisClientType | null = null
/** Dedicated connection for blocking XREAD (a blocking command monopolizes its connection). */
private read: RedisClientType | null = null
private readonly rooms = new Map<string, StoreRoom>()
private running = false
private heartbeat: ReturnType<typeof setInterval> | null = null
constructor(private readonly redisUrl: string | undefined) {
this.enabled = Boolean(redisUrl)
}
/** Connect the two Redis clients and start the multiplexed reader + TTL heartbeat. Idempotent. */
async init(): Promise<void> {
if (!this.enabled || this.running || !this.redisUrl) return
const options = {
url: this.redisUrl,
socket: {
reconnectStrategy: (retries: number) => {
if (retries > 10) return new Error('FileDocStore Redis reconnection failed')
return Math.min(retries * 100, 3000)
},
},
}
this.write = createClient(options)
this.read = this.write.duplicate()
this.write.on('error', (err) => logger.error('FileDocStore write client error:', err))
this.read.on('error', (err) => logger.error('FileDocStore read client error:', err))
await Promise.all([this.write.connect(), this.read.connect()])
this.running = true
void this.runReader()
this.heartbeat = setInterval(() => void this.refreshTtls(), HEARTBEAT_MS)
logger.info('FileDocStore ready — shared Yjs backend over Redis Streams enabled')
}
/** Stop the reader/heartbeat and close both clients. */
async shutdown(): Promise<void> {
this.running = false
if (this.heartbeat) clearInterval(this.heartbeat)
this.heartbeat = null
await Promise.all([this.write?.quit().catch(() => {}), this.read?.quit().catch(() => {})])
this.write = null
this.read = null
}
/**
* Register a locally-opened room and load the shared state into its doc: read the whole stream from
* the head, apply every entry (origin {@link REDIS_ORIGIN}), and remember the last id so the tailer
* resumes exactly after it. A brand-new file has an empty stream and loads nothing (it is seeded
* shortly after, via {@link shouldSeed}). No-op when disabled.
*/
async attachRoom(name: string, doc: Y.Doc): Promise<void> {
if (!this.enabled || !this.write) return
// Register BEFORE the async read so a concurrent publish/tailer for this room can't be missed —
// the tailer resumes from `lastId`, which the catch-up advances.
const room: StoreRoom = {
doc,
lastId: '0',
publishes: 0,
seededObserved: false,
realEdited: false,
}
this.rooms.set(name, room)
try {
const entries = await this.write.xRange(streamKey(name), '-', '+')
for (const entry of entries) {
// The room can be detached + its doc destroyed while catch-up is in flight (a fast open→close);
// stop touching it the moment that happens.
if (this.rooms.get(name) !== room) return
this.applyEntry(room, entry.id, entry.message)
}
await this.write.expire(streamKey(name), STREAM_TTL_SEC)
} catch (error) {
logger.warn(`FileDocStore catch-up failed for ${name}`, { error: getErrorMessage(error) })
}
}
/** Deregister a room the relay is destroying, so the tailer stops touching its (about-to-be-destroyed) doc. */
detachRoom(name: string): void {
this.rooms.delete(name)
}
/**
* Append a locally-applied update to the shared stream so every task converges, AWAITING the write
* and retrying a transient failure ({@link PUBLISH_MAX_RETRIES}) so a Redis blip can't silently drop
* an edit from the shared log. Only the `xAdd` is retried; the TTL refresh + compaction check are
* post-write best-effort and never re-trigger the append. Throws if the append ultimately fails.
*/
private async appendUpdate(name: string, update: Uint8Array, agent = false): Promise<void> {
if (!this.write) return
// Latch realEdited SYNCHRONOUSLY — before the first await — for a real (non-agent) publish. The edit
// already sits in room.doc (applied in doc.on('update') before publish was called), so if this set
// were deferred past the xAdd/expire awaits a CONCURRENT agent-frame-triggered maybeCompact could read
// realEdited=false, snapshot the doc (which already holds this real edit), and stamp it an agent
// (no-persist) snapshot — a lost edit. Setting it in the same synchronous tick as the doc mutation
// makes "room.doc holds a real edit ⇒ realEdited" hold before any compaction (always async) can run.
// Monotonic latch, so an eager set is safe; the seed never flows through here (it uses seedIfEmpty).
if (!agent) {
const editedRoom = this.rooms.get(name)
if (editedRoom) editedRoom.realEdited = true
}
const encoded = Buffer.from(update).toString('base64')
const fields: Record<string, string> = { [UPDATE_FIELD]: encoded }
if (agent) fields[AGENT_FIELD] = '1'
for (let attempt = 0; attempt <= PUBLISH_MAX_RETRIES; attempt++) {
try {
await this.write.xAdd(streamKey(name), '*', fields)
break
} catch (error) {
if (attempt === PUBLISH_MAX_RETRIES) {
logger.error(`FileDocStore append failed for ${name}`, { error: getErrorMessage(error) })
throw error
}
// Snappy backoff — a stream append is a fast op; a transient blip clears in tens of ms.
// `backoffWithJitter` is 1-indexed, so pass the 1-based attempt number.
await sleep(backoffWithJitter(attempt + 1, null, { baseMs: 50, maxMs: 500 }))
}
}
await this.write.expire(streamKey(name), STREAM_TTL_SEC).catch(() => {})
const room = this.rooms.get(name)
if (room && ++room.publishes % COMPACT_CHECK_EVERY === 0) void this.maybeCompact(name)
}
/**
* Fire-and-forget append for the hot keystroke path (`doc.on('update')`): converges peers without
* blocking the relay. Retries internally; never throws. No-op when disabled. Pass `agent: true` for
* a copilot preview frame so peer tasks tail it as {@link REDIS_AGENT_ORIGIN} and never persist it.
*/
publish(name: string, update: Uint8Array, agent = false): void {
if (!this.enabled || !this.write) return
void this.appendUpdate(name, update, agent).catch(() => {}) // already logged inside appendUpdate
}
/**
* Awaitable append for callers that must know the update is durably in the stream before proceeding
* — the copilot merge, so the cross-task merge lock is not released before the diff is committed
* (else the next task would diff a stale base). Throws on ultimate failure. No-op when disabled.
*/
async publishAndWait(name: string, update: Uint8Array): Promise<void> {
if (!this.enabled || !this.write) return
await this.appendUpdate(name, update)
}
/**
* Atomically seed the stream iff it is still empty (see {@link SEED_IF_EMPTY_SCRIPT}). This is what
* actually prevents split-brain double-seeding: the emptiness check and the append happen in one
* Redis-side step, so — unlike a separate {@link streamHasContent} fence + {@link publishAndWait} —
* there is no check-then-append window, and two tasks racing (even both past an expired seed lock) can
* never both write a seed. Returns true if THIS call wrote the seed (apply it locally), false if the
* stream was already seeded (the tailer will deliver the peer's seed — do NOT apply a second one).
* Retries a transient Redis error like {@link appendUpdate}; throws if it ultimately fails. Disabled →
* true (single-replica: seed locally, no stream).
*/
async seedIfEmpty(name: string, update: Uint8Array): Promise<boolean> {
if (!this.enabled || !this.write) return true
const encoded = Buffer.from(update).toString('base64')
for (let attempt = 0; attempt <= PUBLISH_MAX_RETRIES; attempt++) {
try {
const wrote = await this.write.eval(SEED_IF_EMPTY_SCRIPT, {
keys: [streamKey(name)],
arguments: [UPDATE_FIELD, encoded],
})
await this.write.expire(streamKey(name), STREAM_TTL_SEC).catch(() => {})
return wrote === 1
} catch (error) {
if (attempt === PUBLISH_MAX_RETRIES) {
logger.error(`FileDocStore seed failed for ${name}`, { error: getErrorMessage(error) })
throw error
}
await sleep(backoffWithJitter(attempt + 1, null, { baseMs: 50, maxMs: 500 }))
}
}
return false
}
/**
* Whether the file's stream already holds content — an EFFICIENCY recheck in {@link shouldSeed} that
* skips the seed fetch when a prior holder already seeded (the split-brain guard itself is the atomic
* {@link SEED_IF_EMPTY_SCRIPT}, not this check). Treats `true` as "already seeded", so it fails CLOSED:
* a Redis `xLen` error returns `true` (cannot confirm empty → skip the redundant fetch; the atomic seed
* would no-op anyway). `false` only when genuinely empty, or when disabled (single-replica).
*/
async streamHasContent(name: string): Promise<boolean> {
if (!this.enabled || !this.write) return false
try {
return (await this.write.xLen(streamKey(name))) > 0
} catch (error) {
logger.warn(`FileDocStore streamHasContent failed for ${name}`, {
error: getErrorMessage(error),
})
return true
}
}
/**
* Acquire a distributed lock with a unique ownership TOKEN (`SET key <token> NX PX`). Returns the
* token to release with, or `null` if not won. Fails CLOSED (null) on a Redis error — a lock we can't
* prove we hold must not be treated as held. The special sentinel {@link DISABLED_LOCK_TOKEN} lets a
* disabled store return a truthy token so callers proceed single-replica without special-casing.
*/
private async acquireLock(key: string, ttlMs: number): Promise<string | null> {
if (!this.enabled || !this.write) return DISABLED_LOCK_TOKEN
const token = generateId()
try {
return (await this.write.set(key, token, { NX: true, PX: ttlMs })) === 'OK' ? token : null
} catch (error) {
logger.warn(`FileDocStore lock ${key} failed`, { error: getErrorMessage(error) })
return null
}
}
/** Release a lock via compare-and-delete, so it is only dropped if we still hold our token. */
private async releaseLock(key: string, token: string): Promise<void> {
if (!this.write || token === DISABLED_LOCK_TOKEN) return
await this.write.eval(RELEASE_LOCK_SCRIPT, { keys: [key], arguments: [token] }).catch(() => {})
}
/**
* Decide whether THIS task should run the (expensive) seed fetch + write for a file. Returns a lock
* TOKEN when this task wins the seed lock and the stream still looks empty; `null` otherwise. This is an
* EFFICIENCY gate — it stops every task that opens the file at once from each fetching the seed. It does
* NOT by itself guarantee a single seed: exactly-once is enforced by the atomic {@link seedIfEmpty} the
* token-holder then calls (the split-brain guard), so a lock that expires mid-seed cannot cause a
* double-seed. Release the token with {@link releaseSeedLock}. Disabled → always a token (single-replica:
* seed locally).
*/
async shouldSeed(name: string): Promise<string | null> {
const token = await this.acquireLock(`${SEED_LOCK_PREFIX}${name}`, SEED_LOCK_TTL_MS)
if (!token || token === DISABLED_LOCK_TOKEN) return token
// The lock could be free yet the stream already seeded (a prior holder seeded then its lock
// expired). Re-check so we skip the redundant seed fetch — the atomic seedIfEmpty would no-op anyway,
// but this avoids the wasted app round-trip.
if (await this.streamHasContent(name)) {
await this.releaseSeedLock(name, token)
return null
}
return token
}
/**
* Build the file's current shared state from the stream, headless (no registered room), for a merge
* that must reach the live doc regardless of which task holds it. Returns the encoded Yjs state, or
* `null` when the stream is empty — i.e. no doc is (or was recently) live, so there is nothing to
* merge into and the caller should fall back to a direct file write. Disabled → always null.
*/
async getStreamState(name: string): Promise<Uint8Array | null> {
if (!this.enabled || !this.write) return null
const entries = await this.write.xRange(streamKey(name), '-', '+')
if (entries.length === 0) return null
const doc = new Y.Doc()
try {
for (const entry of entries) applyEntryToDoc(doc, entry.id, entry.message)
return Y.encodeStateAsUpdate(doc)
} finally {
doc.destroy()
}
}
/** Release the seed lock (compare-and-delete) once the seed has been published or a seed attempt failed. */
async releaseSeedLock(name: string, token: string): Promise<void> {
await this.releaseLock(`${SEED_LOCK_PREFIX}${name}`, token)
}
/**
* A best-effort TTL dedup WINDOW (NOT a lock): claim the right to run a debounced persist for the next
* `ttlMs`, so concurrent tasks editing the same file don't each write a redundant blob version. It is
* never released — it simply expires after `ttlMs`, gating the debounced persist to ~once per window
* cluster-wide. Fails OPEN (returns true on a Redis error): a redundant persist is a harmless
* idempotent write, so it must never block a real one. The final last-collaborator flush does NOT gate
* on this — it must always write.
*/
async tryClaimPersistWindow(name: string, ttlMs: number): Promise<boolean> {
if (!this.enabled || !this.write) return true
try {
const won = await this.write.set(`${PERSIST_LOCK_PREFIX}${name}`, '1', {
NX: true,
PX: ttlMs,
})
return won === 'OK'
} catch {
return true
}
}
/**
* Try to claim the cross-task right to merge new content into this file. The relay already serializes
* merges per task; this extends that across tasks so two copilot edits to the same file landing on
* different tasks don't each diff the SAME shared base and publish conflicting full-document rewrites.
* The loser waits and retries so it diffs against the winner's RESULT (correct sequential merge).
* Returns a lock TOKEN (proceed) when disabled or once won; `null` otherwise (fails CLOSED on error, so
* a merge never races when exclusivity can't be proven). Release with {@link releaseMergeSlot}.
*/
async acquireMergeSlot(name: string, ttlMs: number): Promise<string | null> {
return this.acquireLock(`${MERGE_LOCK_PREFIX}${name}`, ttlMs)
}
/**
* The durable file version (its `updatedAt`, epoch ms) the shared live doc is synced to — the
* cluster-wide {@link https://www.rfc-editor.org/rfc/rfc7232 `If-Match`} token for persistence. Held
* in Redis (not per-task room state) so whichever task runs a debounced/last-leave persist reads the
* SAME version, even though the write that advanced it (a seed or a merged edit) may have run on
* another task. Returns `null` when unset/expired (persist then falls back to the local room's value).
*/
async getSyncedVersion(name: string): Promise<number | null> {
if (!this.enabled || !this.write) return null
try {
const value = await this.write.get(`${SYNC_VERSION_PREFIX}${name}`)
const parsed = value === null ? Number.NaN : Number(value)
return Number.isFinite(parsed) ? parsed : null
} catch (error) {
logger.warn(`FileDocStore getSyncedVersion failed for ${name}`, {
error: getErrorMessage(error),
})
return null
}
}
/** Record the durable version the shared live doc is now synced to. MONOTONIC — writes only when the
* new value exceeds the stored one ({@link SET_VERSION_IF_NEWER_SCRIPT}), so an out-of-order
* fire-and-forget write can't regress the token. Best-effort; TTL-bounded like the stream so an idle
* file's key can't outlive its room. No-op when disabled (single-pod fallback). */
async setSyncedVersion(name: string, version: number): Promise<void> {
if (!this.enabled || !this.write) return
// Retry a transient failure (bounded) rather than swallow it: this token is the ONLY way a
// peer-seeded task learns the durable version, so a dropped write would leave that peer's persists
// deferring forever with the session's edits stranded in the TTL'd stream. The monotonic script makes
// a retry that races a newer value a no-op, never a regression.
for (let attempt = 0; attempt <= PUBLISH_MAX_RETRIES; attempt++) {
try {
await this.write.eval(SET_VERSION_IF_NEWER_SCRIPT, {
keys: [`${SYNC_VERSION_PREFIX}${name}`],
arguments: [String(version), String(STREAM_TTL_SEC)],
})
return
} catch (error) {
if (attempt === PUBLISH_MAX_RETRIES) {
logger.warn(`FileDocStore setSyncedVersion failed for ${name}`, {
error: getErrorMessage(error),
})
return
}
await sleep(backoffWithJitter(attempt + 1, null, { baseMs: 50, maxMs: 500 }))
}
}
}
/** Mark (or refresh) that a client is actively streaming an agent edit into this live doc — a plain
* `SET key "1" PX ttl`, so it self-clears when streaming stops. Best-effort; no-op when disabled. */
async markAgentStreaming(name: string, ttlMs: number): Promise<void> {
if (!this.enabled || !this.write) return
try {
await this.write.set(`${AGENT_STREAM_PREFIX}${name}`, '1', { PX: ttlMs })
} catch (error) {
logger.warn(`FileDocStore markAgentStreaming failed for ${name}`, {
error: getErrorMessage(error),
})
}
}
/** Whether a client is currently streaming an agent edit into this live doc (see
* {@link markAgentStreaming}). Best-effort; treats an error/disabled store as "not streaming" so a
* merge never blocks on this check. */
async isAgentStreaming(name: string): Promise<boolean> {
if (!this.enabled || !this.write) return false
try {
return (await this.write.exists(`${AGENT_STREAM_PREFIX}${name}`)) === 1
} catch (error) {
logger.warn(`FileDocStore isAgentStreaming failed for ${name}`, {
error: getErrorMessage(error),
})
return false
}
}
async releaseMergeSlot(name: string, token: string): Promise<void> {
await this.releaseLock(`${MERGE_LOCK_PREFIX}${name}`, token)
}
private applyEntry(room: StoreRoom, id: string, message: Record<string, string>): void {
room.lastId = id
// A compaction snapshot folds seed + edits into one frame; stamp it so the relay's edit-tracker
// treats a fresh catch-up from it as edited (a snapshot only exists once real edits accumulated). An
// agent-streamed preview frame is stamped separately so the tracker NEVER marks it edited.
const origin = message[SNAPSHOT_FIELD]
? REDIS_SNAPSHOT_ORIGIN
: message[AGENT_FIELD]
? REDIS_AGENT_ORIGIN
: REDIS_ORIGIN
const seededBefore = room.seededObserved
applyEntryToDoc(room.doc, id, message, origin)
if (isDocSeeded(room.doc)) room.seededObserved = true
// Track a real edit integrated from the stream so compaction knows whether its snapshot represents
// real content or agent-only preview: a real snapshot (folds real edits), or a markerless edit
// applied AFTER the doc was already seeded (the seed transition itself never counts). Agent frames
// and agent snapshots (REDIS_AGENT_ORIGIN) never count.
if (origin === REDIS_SNAPSHOT_ORIGIN || (origin === REDIS_ORIGIN && seededBefore)) {
room.realEdited = true
}
}
/**
* The single multiplexed tail loop: block-read every locally-open room's stream from its last id and
* apply new entries. One blocking connection for the whole process regardless of open-file count.
*/
private async runReader(): Promise<void> {
while (this.running && this.read) {
const snapshot = new Map(this.rooms)
if (snapshot.size === 0) {
await sleep(IDLE_POLL_MS)
continue
}
try {
const res = await this.read.xRead(
[...snapshot].map(([name, room]) => ({ key: streamKey(name), id: room.lastId })),
{ BLOCK: READ_BLOCK_MS, COUNT: READ_COUNT }
)
if (!res) continue
for (const stream of res) {
const name = stream.name.slice(STREAM_PREFIX.length)
const room = this.rooms.get(name)
// Skip if detached mid-read, OR replaced by a close→reopen (a DIFFERENT StoreRoom): applying
// entries read against the OLD room's lastId to the new one could regress its lastId (harmless
// but wasteful re-delivery). The new room caught itself up via xRange already.
if (!room || room !== snapshot.get(name)) continue
for (const entry of stream.messages) this.applyEntry(room, entry.id, entry.message)
}
} catch (error) {
if (!this.running) break
logger.warn('FileDocStore reader error; retrying', { error: getErrorMessage(error) })
await sleep(500)
}
}
}
/**
* Snapshot-then-trim compaction: append a full-state snapshot and drop the older deltas it subsumes,
* so the stream stays bounded while a fresh task can still catch up from the head. Lock-guarded so
* only one task compacts a given stream at a time (concurrent snapshot+trim would race). Trims only up
* to what the snapshot provably contains — never un-integrated peer entries (see below).
*/
private async maybeCompact(name: string): Promise<void> {
if (!this.write) return
const room = this.rooms.get(name)
if (!room) return
try {
if ((await this.write.xLen(streamKey(name))) < COMPACT_THRESHOLD) return
const key = `${COMPACT_LOCK_PREFIX}${name}`
const token = await this.acquireLock(key, COMPACT_LOCK_TTL_MS)
if (!token) return
try {
// Capture the snapshot AND the id it covers in one synchronous step (no await between): the
// snapshot is `room.doc`, which holds exactly what this task's tailer has integrated — every
// entry up to `room.lastId`. Entries a peer task published AFTER that (id > lastId) are NOT in
// the snapshot and this task's blocking reader may not have seen them yet, so we must NOT trim
// them — only entries the snapshot provably subsumes (id <= lastId). Trimming to the freshly
// appended snapshot id instead would silently drop those un-integrated peer entries.
const upTo = room.lastId
const snapshot = Buffer.from(Y.encodeStateAsUpdate(room.doc)).toString('base64')
// Stamp the snapshot by what it folds: a real edit → SNAPSHOT_FIELD (a fresh catch-up treats it
// as edited content, not a bare seed). An agent-ONLY stream (no real edit yet) → AGENT_FIELD, so a
// peer catching up applies it as REDIS_AGENT_ORIGIN and never marks the doc edited — preserving
// the no-persist guarantee even when a long copilot stream alone crosses the compaction threshold.
const marker = room.realEdited ? SNAPSHOT_FIELD : AGENT_FIELD
await this.write.xAdd(streamKey(name), '*', {
[UPDATE_FIELD]: snapshot,
[marker]: '1',
})
// MINID keeps entries with id >= upTo: the snapshot, any un-integrated peer entries, and
// `upTo` itself (redundant with the snapshot, harmless); it drops only the folded older deltas.
await this.write.xTrim(streamKey(name), 'MINID', upTo)
} finally {
await this.releaseLock(key, token)
}
} catch (error) {
logger.warn(`FileDocStore compaction failed for ${name}`, { error: getErrorMessage(error) })
}
}
private async refreshTtls(): Promise<void> {
if (!this.write) return
for (const name of this.rooms.keys()) {
await this.write.expire(streamKey(name), STREAM_TTL_SEC).catch(() => {})
// Keep the synced-version key alive as long as its stream, so an open-but-idle doc's persist
// If-Match token can't expire out from under it (which would force a needless reconcile).
await this.write.expire(`${SYNC_VERSION_PREFIX}${name}`, STREAM_TTL_SEC).catch(() => {})
}
}
}
let store: FileDocStore | null = null
/**
* Initialize the process-wide store from the realtime server bootstrap (alongside the socket adapter).
* Authoritative: if a disabled placeholder was lazily created by an early {@link getFileDocStore} call,
* this REPLACES it with the real, connected store — so the bootstrap can never silently no-op. A second
* call once already initialized is a no-op.
*/
export async function initFileDocStore(redisUrl: string | undefined): Promise<FileDocStore> {
if (store?.enabled) return store
store = new FileDocStore(redisUrl)
await store.init()
return store
}
/** The process-wide store. Returns a disabled instance if init was never called (e.g. in unit tests). */
export function getFileDocStore(): FileDocStore {
if (!store) store = new FileDocStore(undefined)
return store
}
@@ -0,0 +1,110 @@
/**
* @vitest-environment node
*
* Multi-replica (store-enabled) coverage for the copilot live-merge stale-check. The main
* `file-doc.test.ts` runs with the store DISABLED (single-replica fallback); this file mocks an ENABLED
* store so the cross-process branch of `mergeMarkdownIntoRoom` — staleness against the SHARED synced
* version under the merge lock, and `recordVersion` writing `setSyncedVersion` — is exercised directly.
* The enabled merge path reads its base from the shared store (not an in-memory room), so no JOIN/seed
* is needed: calling `applyMarkdownToLiveFileDoc` against the fake store drives the branch on its own.
*/
import { beforeEach, describe, expect, it, vi } from 'vitest'
import * as Y from 'yjs'
const { mockFetchFileDocMerge } = vi.hoisted(() => ({
mockFetchFileDocMerge: vi.fn(),
}))
/**
* A minimal ENABLED store: in-memory monotonic synced version (mirrors SET_VERSION_IF_NEWER_SCRIPT), a
* non-null stream state so the merge has a base, and no-op locks/publish. Only the surface the
* store-enabled merge path touches is implemented.
*/
const fakeStore = {
enabled: true,
versions: new Map<string, number>(),
acquireMergeSlot: vi.fn(async () => 'token'),
releaseMergeSlot: vi.fn(async () => {}),
getStreamState: vi.fn(async () => new Uint8Array([1])),
publishAndWait: vi.fn(async () => {}),
getSyncedVersion: vi.fn(async (name: string) => fakeStore.versions.get(name) ?? null),
setSyncedVersion: vi.fn(async (name: string, version: number) => {
fakeStore.versions.set(name, Math.max(fakeStore.versions.get(name) ?? 0, version))
}),
markAgentStreaming: vi.fn(async () => {}),
isAgentStreaming: vi.fn(async () => false),
}
vi.mock('@sim/platform-authz/rooms', () => ({ authorizeRoom: vi.fn() }))
vi.mock('@/handlers/file-doc-app', () => ({
fetchFileDocSeed: vi.fn(),
fetchFileDocMerge: mockFetchFileDocMerge,
fetchFileDocPersist: vi.fn(),
}))
vi.mock('@/handlers/file-doc-store', () => ({
getFileDocStore: () => fakeStore,
REDIS_ORIGIN: Symbol('redis'),
REDIS_SNAPSHOT_ORIGIN: Symbol('redis-snapshot'),
}))
import { applyMarkdownToLiveFileDoc } from '@/handlers/file-doc'
const ROOM_NAME = 'workspace-file-doc:file-1'
describe('applyMarkdownToLiveFileDoc — multi-replica (store-enabled) ordering', () => {
beforeEach(() => {
vi.clearAllMocks()
fakeStore.versions.clear()
fakeStore.acquireMergeSlot.mockResolvedValue('token')
fakeStore.getStreamState.mockResolvedValue(new Uint8Array([1]))
mockFetchFileDocMerge.mockResolvedValue(Y.encodeStateAsUpdate(new Y.Doc()))
})
it('drops a stale durable write against the SHARED synced version', async () => {
// A durable write (e.g. a concurrent human save on another process) records the shared synced version.
expect(await applyMarkdownToLiveFileDoc('file-1', '# durable', { version: 100 })).toBe(
'applied'
)
expect(fakeStore.setSyncedVersion).toHaveBeenCalledWith(ROOM_NAME, 100)
mockFetchFileDocMerge.mockClear()
// A durable write with an OLDER version than the SHARED synced version is stale — rejected under the
// lock before any diff is built, so it can't regress the doc across replicas.
expect(await applyMarkdownToLiveFileDoc('file-1', '# older durable', { version: 50 })).toBe(
'stale'
)
expect(mockFetchFileDocMerge).not.toHaveBeenCalled()
// A newer durable write applies and advances the shared synced version.
expect(await applyMarkdownToLiveFileDoc('file-1', '# durable again', { version: 150 })).toBe(
'applied'
)
expect(fakeStore.setSyncedVersion).toHaveBeenCalledWith(ROOM_NAME, 150)
// setSyncedVersion fired only for the two applied durable writes, never for the stale one.
expect(fakeStore.setSyncedVersion).toHaveBeenCalledTimes(2)
})
it('defers the content merge cluster-wide when a client is streaming (records version, no diff)', async () => {
// The cluster-wide counterpart of the single-replica deferral: while the shared `isAgentStreaming`
// flag is set (a client on ANY replica is streaming this agent edit), the durable merge must record
// the version but skip the content diff — the streaming client owns the bytes, so a whole-document
// merge here would double-write them.
fakeStore.isAgentStreaming.mockResolvedValue(true)
expect(
await applyMarkdownToLiveFileDoc('file-1', '# streamed by a client', { version: 100 })
).toBe('applied')
expect(mockFetchFileDocMerge).not.toHaveBeenCalled() // content deferred to the client
expect(fakeStore.publishAndWait).not.toHaveBeenCalled()
expect(fakeStore.setSyncedVersion).toHaveBeenCalledWith(ROOM_NAME, 100) // version still recorded
// Once streaming stops the flag clears and the (now near-noop) durable merge resumes normally.
fakeStore.isAgentStreaming.mockResolvedValue(false)
expect(await applyMarkdownToLiveFileDoc('file-1', '# final durable', { version: 150 })).toBe(
'applied'
)
expect(mockFetchFileDocMerge).toHaveBeenCalledTimes(1)
})
})
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+9
View File
@@ -1,9 +1,13 @@
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { setupConnectionHandlers } from '@/handlers/connection'
import { setupWorkspaceFileDocHandlers } from '@/handlers/file-doc'
import { setupOperationsHandlers } from '@/handlers/operations'
import { setupPresenceHandlers } from '@/handlers/presence'
import { setupSubblocksHandlers } from '@/handlers/subblocks'
import { setupTablesHandlers } from '@/handlers/tables'
import { setupVariablesHandlers } from '@/handlers/variables'
import { setupWorkflowHandlers } from '@/handlers/workflow'
import { setupWorkspaceInvalidationRoom } from '@/handlers/workspace-invalidation-room'
import type { AuthenticatedSocket } from '@/middleware/auth'
import type { IRoomManager } from '@/rooms'
@@ -13,5 +17,10 @@ export function setupAllHandlers(socket: AuthenticatedSocket, roomManager: IRoom
setupSubblocksHandlers(socket, roomManager)
setupVariablesHandlers(socket, roomManager)
setupPresenceHandlers(socket, roomManager)
// Presence-free, workspace-scoped live-list rooms (share one implementation).
setupWorkspaceInvalidationRoom(socket, roomManager, ROOM_TYPES.WORKSPACE_FILES)
setupWorkspaceInvalidationRoom(socket, roomManager, ROOM_TYPES.WORKSPACE_TABLES)
setupWorkspaceFileDocHandlers(socket, roomManager)
setupTablesHandlers(socket, roomManager)
setupConnectionHandlers(socket, roomManager)
}
+23 -18
View File
@@ -9,6 +9,7 @@ import {
type VariableOperation,
WORKFLOW_OPERATIONS,
} from '@sim/realtime-protocol/constants'
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { WorkflowOperationSchema } from '@sim/realtime-protocol/schemas'
import { getErrorMessage } from '@sim/utils/errors'
import { generateId } from '@sim/utils/id'
@@ -16,7 +17,7 @@ import { ZodError } from 'zod'
import { persistWorkflowOperation } from '@/database/operations'
import type { AuthenticatedSocket } from '@/middleware/auth'
import { checkWorkflowOperationPermission } from '@/middleware/permissions'
import type { IRoomManager, UserSession } from '@/rooms'
import { type IRoomManager, type UserSession, workflowRoom as wf } from '@/rooms'
const logger = createLogger('OperationsHandlers')
@@ -44,7 +45,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
let session: UserSession | null = null
try {
workflowId = await roomManager.getWorkflowIdForSocket(socket.id)
workflowId = (await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.WORKFLOW))?.id ?? null
session = await roomManager.getUserSession(socket.id)
} catch (error) {
logger.error('Error loading session for workflow operation:', error)
@@ -65,7 +66,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
let hasRoom = false
try {
hasRoom = await roomManager.hasWorkflowRoom(workflowId)
hasRoom = await roomManager.hasRoom(wf(workflowId))
} catch (error) {
logger.error('Error checking workflow room:', error)
emitOperationError(
@@ -98,14 +99,16 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
const operationTimestamp = isPositionUpdate ? timestamp : Date.now()
// Get user presence for permission checking
const users = await roomManager.getWorkflowUsers(workflowId)
const users = await roomManager.getRoomUsers(wf(workflowId))
const userPresence = users.find((u) => u.socketId === socket.id)
// Skip permission checks for non-committed position updates (broadcasts only, no persistence)
if (isPositionUpdate && !commitPositionUpdate) {
// Update last activity
if (userPresence) {
await roomManager.updateUserActivity(workflowId, socket.id, { lastActivity: Date.now() })
await roomManager.updateUserActivity(wf(workflowId), socket.id, {
lastActivity: Date.now(),
})
}
} else {
// Check permissions from cached role for all other operations
@@ -123,7 +126,9 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
return
}
await roomManager.updateUserActivity(workflowId, socket.id, { lastActivity: Date.now() })
await roomManager.updateUserActivity(wf(workflowId), socket.id, {
lastActivity: Date.now(),
})
// Re-validate the workspace role against the DB (cached per pod for a short
// window) so revoked or downgraded collaborators lose write access live.
@@ -198,7 +203,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
timestamp: operationTimestamp,
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
if (operationId) {
socket.emit('operation-confirmed', {
@@ -244,7 +249,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
timestamp: operationTimestamp,
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
if (operationId) {
socket.emit('operation-confirmed', { operationId, serverTimestamp: Date.now() })
@@ -277,7 +282,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
const broadcastData = {
operation,
@@ -317,7 +322,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
const broadcastData = {
operation,
@@ -354,7 +359,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
socket.to(workflowId).emit('workflow-operation', {
operation,
@@ -386,7 +391,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
socket.to(workflowId).emit('workflow-operation', {
operation,
@@ -415,7 +420,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
socket.to(workflowId).emit('workflow-operation', {
operation,
@@ -447,7 +452,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
socket.to(workflowId).emit('workflow-operation', {
operation,
@@ -479,7 +484,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
socket.to(workflowId).emit('workflow-operation', {
operation,
@@ -511,7 +516,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
socket.to(workflowId).emit('workflow-operation', {
operation,
@@ -540,7 +545,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
socket.to(workflowId).emit('workflow-operation', {
operation,
@@ -569,7 +574,7 @@ export function setupOperationsHandlers(socket: AuthenticatedSocket, roomManager
userId: session.userId,
})
await roomManager.updateRoomLastModified(workflowId)
await roomManager.updateRoomLastModified(wf(workflowId))
const broadcastData = {
operation,
+11 -10
View File
@@ -1,4 +1,5 @@
import { createLogger } from '@sim/logger'
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import type { AuthenticatedSocket } from '@/middleware/auth'
import type { IRoomManager } from '@/rooms'
@@ -7,16 +8,16 @@ const logger = createLogger('PresenceHandlers')
export function setupPresenceHandlers(socket: AuthenticatedSocket, roomManager: IRoomManager) {
socket.on('cursor-update', async ({ cursor }) => {
try {
const workflowId = await roomManager.getWorkflowIdForSocket(socket.id)
const room = await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.WORKFLOW)
const session = await roomManager.getUserSession(socket.id)
if (!workflowId || !session) return
if (!room || !session) return
// Update cursor in room state
await roomManager.updateUserActivity(workflowId, socket.id, { cursor })
await roomManager.updateUserActivity(room, socket.id, { cursor })
// Broadcast to other users in the room
socket.to(workflowId).emit('cursor-update', {
// Broadcast to other users in the room (workflow room name is the bare id)
socket.to(room.id).emit('cursor-update', {
socketId: socket.id,
userId: session.userId,
userName: session.userName,
@@ -30,16 +31,16 @@ export function setupPresenceHandlers(socket: AuthenticatedSocket, roomManager:
socket.on('selection-update', async ({ selection }) => {
try {
const workflowId = await roomManager.getWorkflowIdForSocket(socket.id)
const room = await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.WORKFLOW)
const session = await roomManager.getUserSession(socket.id)
if (!workflowId || !session) return
if (!room || !session) return
// Update selection in room state
await roomManager.updateUserActivity(workflowId, socket.id, { selection })
await roomManager.updateUserActivity(room, socket.id, { selection })
// Broadcast to other users in the room
socket.to(workflowId).emit('selection-update', {
// Broadcast to other users in the room (workflow room name is the bare id)
socket.to(room.id).emit('selection-update', {
socketId: socket.id,
userId: session.userId,
userName: session.userName,
@@ -0,0 +1,55 @@
import type { createLogger } from '@sim/logger'
import { authorizeRoom } from '@sim/platform-authz/rooms'
import type { RoomRef } from '@sim/realtime-protocol/rooms'
type Authorized = Awaited<ReturnType<typeof authorizeRoom>>
interface ResolveRoomJoinAuthParams {
userId: string
room: RoomRef
action: 'read' | 'write'
logger: ReturnType<typeof createLogger>
/** Included in the warn log on an authorize throw, e.g. `table room for ${userId}`. */
logLabel: string
messages: { verifyFailed: string; notFound: string; accessDenied: string }
/** Emits the handler's own JOIN_ERROR shape (event name + id key differ per handler). */
emitError: (args: { error: string; code: string; retryable: boolean }) => void
}
/**
* Runs the shared authorize→allowed slice of a room join: authorizes the room and checks
* the result, emitting the handler-specific JOIN_ERROR on failure. Returns the authorized
* result on success, or `null` when it has already emitted an error and the caller must return.
*
* Deliberately excludes the auth/readiness/id-validation preamble and the join-generation
* capture/recheck — those differ per handler, and for file-doc the generation capture sits
* mid-preamble. This helper is always invoked strictly between a handler's generation capture
* and its post-authorize recheck; it contains exactly the one `await authorizeRoom` that the
* recheck was designed to cover and returns before any state mutation, so it never straddles
* that seam. Pass each handler's own `logger` so the log namespace/request-id context is kept.
*/
export async function resolveRoomJoinAuth(
params: ResolveRoomJoinAuthParams
): Promise<Authorized | null> {
const { userId, room, action, logger, logLabel, messages, emitError } = params
let authorized: Authorized
try {
authorized = await authorizeRoom({ userId, room, action })
} catch (error) {
logger.warn(`Error authorizing ${logLabel}:`, error)
emitError({ error: messages.verifyFailed, code: 'VERIFY_ACCESS_FAILED', retryable: true })
return null
}
if (!authorized.allowed) {
emitError({
error: authorized.status === 404 ? messages.notFound : messages.accessDenied,
code: authorized.status === 404 ? 'NOT_FOUND' : 'ACCESS_DENIED',
retryable: false,
})
return null
}
return authorized
}
+7 -5
View File
@@ -3,12 +3,13 @@ import { workflow, workflowBlocks } from '@sim/db/schema'
import { createLogger } from '@sim/logger'
import { assertWorkflowMutable, WorkflowLockedError } from '@sim/platform-authz/workflow'
import { SUBBLOCK_OPERATIONS } from '@sim/realtime-protocol/constants'
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { getErrorMessage } from '@sim/utils/errors'
import { isWorkflowBlockProtected } from '@sim/workflow-types/workflow'
import { and, eq } from 'drizzle-orm'
import type { AuthenticatedSocket } from '@/middleware/auth'
import { checkWorkflowOperationPermission } from '@/middleware/permissions'
import type { IRoomManager } from '@/rooms'
import { type IRoomManager, workflowRoom as wf } from '@/rooms'
const logger = createLogger('SubblocksHandlers')
@@ -69,7 +70,8 @@ export function setupSubblocksHandlers(socket: AuthenticatedSocket, roomManager:
}
try {
const sessionWorkflowId = await roomManager.getWorkflowIdForSocket(socket.id)
const sessionWorkflowId =
(await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.WORKFLOW))?.id ?? null
const session = await roomManager.getUserSession(socket.id)
if (!sessionWorkflowId || !session) {
@@ -106,7 +108,7 @@ export function setupSubblocksHandlers(socket: AuthenticatedSocket, roomManager:
return
}
const hasRoom = await roomManager.hasWorkflowRoom(workflowId)
const hasRoom = await roomManager.hasRoom(wf(workflowId))
if (!hasRoom) {
logger.debug(`Ignoring subblock update: workflow room not found`, {
socketId: socket.id,
@@ -117,7 +119,7 @@ export function setupSubblocksHandlers(socket: AuthenticatedSocket, roomManager:
return
}
const users = await roomManager.getWorkflowUsers(workflowId)
const users = await roomManager.getRoomUsers(wf(workflowId))
const userPresence = users.find((user) => user.socketId === socket.id)
if (!userPresence) {
socket.emit('operation-forbidden', {
@@ -182,7 +184,7 @@ export function setupSubblocksHandlers(socket: AuthenticatedSocket, roomManager:
}
// Update user activity
await roomManager.updateUserActivity(workflowId, socket.id, { lastActivity: Date.now() })
await roomManager.updateUserActivity(wf(workflowId), socket.id, { lastActivity: Date.now() })
// Server-side debounce/coalesce by workflowId+blockId+subblockId
const debouncedKey = `${workflowId}:${blockId}:${subblockId}`
+351
View File
@@ -0,0 +1,351 @@
/**
* @vitest-environment node
*/
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { TABLE_PRESENCE_EVENTS } from '@sim/realtime-protocol/table-presence'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import type { IRoomManager } from '@/rooms'
const { mockAuthorizeRoom } = vi.hoisted(() => ({
mockAuthorizeRoom: vi.fn(),
}))
vi.mock('@sim/db', () => ({
db: { select: vi.fn() },
user: { image: 'image' },
}))
vi.mock('@sim/platform-authz/rooms', () => ({
authorizeRoom: mockAuthorizeRoom,
}))
import { setupTablesHandlers } from '@/handlers/tables'
const TABLE_ROOM = { type: ROOM_TYPES.TABLE, id: 'table-1' }
function createSocket(overrides?: Record<string, unknown>) {
const handlers: Record<string, (payload: unknown) => Promise<void> | void> = {}
const toEmit = vi.fn()
const socket = {
id: 'socket-1',
userId: 'user-1',
userName: 'Test User',
userImage: 'avatar.png',
on: vi.fn((event: string, handler: (payload: unknown) => Promise<void> | void) => {
handlers[event] = handler
}),
emit: vi.fn(),
join: vi.fn(),
leave: vi.fn(),
to: vi.fn().mockReturnValue({ emit: toEmit }),
...overrides,
}
return { handlers, socket, toEmit }
}
function createRoomManager(overrides?: Partial<IRoomManager>): IRoomManager {
return {
isReady: vi.fn().mockReturnValue(true),
getRoomForSocket: vi.fn().mockResolvedValue(null),
getRoomsForSocket: vi.fn().mockResolvedValue([]),
removeUserFromRoom: vi.fn().mockResolvedValue(false),
removeSocketFromAllRooms: vi.fn().mockResolvedValue([]),
broadcastPresenceUpdate: vi.fn().mockResolvedValue(undefined),
getRoomUsers: vi.fn().mockResolvedValue([]),
hasRoom: vi.fn().mockResolvedValue(false),
deleteRoom: vi.fn().mockResolvedValue(undefined),
addUserToRoom: vi.fn().mockResolvedValue(undefined),
getUserSession: vi.fn().mockResolvedValue(null),
updateUserActivity: vi.fn().mockResolvedValue(undefined),
updateRoomLastModified: vi.fn().mockResolvedValue(undefined),
emitToRoom: vi.fn(),
getUniqueUserCount: vi.fn().mockResolvedValue(1),
getTotalActiveConnections: vi.fn().mockResolvedValue(0),
shutdown: vi.fn().mockResolvedValue(undefined),
initialize: vi.fn().mockResolvedValue(undefined),
io: {
in: vi.fn().mockReturnValue({ socketsLeave: vi.fn().mockResolvedValue(undefined) }),
},
...overrides,
} as unknown as IRoomManager
}
type SetupArg = Parameters<typeof setupTablesHandlers>[0]
describe('setupTablesHandlers', () => {
beforeEach(() => {
vi.clearAllMocks()
mockAuthorizeRoom.mockResolvedValue({
allowed: true,
status: 200,
workspaceId: 'ws-1',
workspacePermission: 'admin',
})
})
it('rejects join when the socket is not authenticated', async () => {
const { socket, handlers } = createSocket({ userId: undefined, userName: undefined })
setupTablesHandlers(socket as unknown as SetupArg, createRoomManager())
await handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-1' })
expect(socket.emit).toHaveBeenCalledWith(TABLE_PRESENCE_EVENTS.JOIN_ERROR, {
tableId: 'table-1',
error: 'Authentication required',
code: 'AUTHENTICATION_REQUIRED',
retryable: false,
})
})
it('rejects join with a retryable error when realtime is unavailable', async () => {
const { socket, handlers } = createSocket()
setupTablesHandlers(
socket as unknown as SetupArg,
createRoomManager({ isReady: vi.fn().mockReturnValue(false) })
)
await handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-1' })
expect(socket.emit).toHaveBeenCalledWith(
TABLE_PRESENCE_EVENTS.JOIN_ERROR,
expect.objectContaining({ code: 'ROOM_MANAGER_UNAVAILABLE', retryable: true })
)
})
it('rejects join when table access is denied', async () => {
mockAuthorizeRoom.mockResolvedValue({
allowed: false,
status: 403,
workspaceId: 'ws-1',
workspacePermission: null,
})
const { socket, handlers } = createSocket()
setupTablesHandlers(socket as unknown as SetupArg, createRoomManager())
await handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-1' })
expect(socket.emit).toHaveBeenCalledWith(
TABLE_PRESENCE_EVENTS.JOIN_ERROR,
expect.objectContaining({ code: 'ACCESS_DENIED', retryable: false })
)
})
it('joins the table room and broadcasts presence on success', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
await handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-1', tabSessionId: 'tab-1' })
expect(socket.join).toHaveBeenCalledWith('table:table-1')
expect(roomManager.addUserToRoom).toHaveBeenCalledWith(
TABLE_ROOM,
'socket-1',
expect.objectContaining({ userId: 'user-1', role: 'admin' })
)
expect(socket.emit).toHaveBeenCalledWith(
TABLE_PRESENCE_EVENTS.JOIN_SUCCESS,
expect.objectContaining({ tableId: 'table-1', socketId: 'socket-1' })
)
expect(roomManager.broadcastPresenceUpdate).toHaveBeenCalledWith(TABLE_ROOM)
})
it('persists and relays a cell selection to the namespaced room (id + cell only)', async () => {
const { socket, handlers, toEmit } = createSocket()
const roomManager = createRoomManager({
getRoomForSocket: vi.fn().mockResolvedValue(TABLE_ROOM),
})
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
const cell = {
anchor: { rowId: 'row-1', columnId: 'col-a' },
focus: { rowId: 'row-1', columnId: 'col-a' },
editing: true,
}
await handlers[TABLE_PRESENCE_EVENTS.CELL_SELECTION]({ cell })
expect(roomManager.updateUserActivity).toHaveBeenCalledWith(TABLE_ROOM, 'socket-1', { cell })
// Namespaced room → broadcast targets roomName(room), not the bare id.
expect(socket.to).toHaveBeenCalledWith('table:table-1')
// The delta carries only the socket id + cell — identity comes from the roster.
expect(toEmit).toHaveBeenCalledWith(TABLE_PRESENCE_EVENTS.CELL_SELECTION, {
socketId: 'socket-1',
cell,
})
})
it('drops a malformed cell selection without storing or relaying it', async () => {
const { socket, handlers, toEmit } = createSocket()
const roomManager = createRoomManager({
getRoomForSocket: vi.fn().mockResolvedValue(TABLE_ROOM),
})
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
await handlers[TABLE_PRESENCE_EVENTS.CELL_SELECTION]({ cell: { anchor: 'x"]' } })
expect(roomManager.updateUserActivity).not.toHaveBeenCalled()
expect(toEmit).not.toHaveBeenCalled()
})
it('strips unknown/oversized fields from an otherwise-valid selection before storing or relaying', async () => {
const { socket, handlers, toEmit } = createSocket()
const roomManager = createRoomManager({
getRoomForSocket: vi.fn().mockResolvedValue(TABLE_ROOM),
})
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
await handlers[TABLE_PRESENCE_EVENTS.CELL_SELECTION]({
cell: {
anchor: { rowId: 'row-1', columnId: 'col-a', junk: 'x'.repeat(10_000) },
focus: { rowId: 'row-1', columnId: 'col-a' },
editing: true,
bloat: 'x'.repeat(100_000),
},
})
// Only the whitelisted fields survive — a hostile peer can't amplify an oversized object.
const expected = {
anchor: { rowId: 'row-1', columnId: 'col-a' },
focus: { rowId: 'row-1', columnId: 'col-a' },
editing: true,
}
expect(roomManager.updateUserActivity).toHaveBeenCalledWith(TABLE_ROOM, 'socket-1', {
cell: expected,
})
expect(toEmit).toHaveBeenCalledWith(TABLE_PRESENCE_EVENTS.CELL_SELECTION, {
socketId: 'socket-1',
cell: expected,
})
})
it('skips a superseded queued join on a fast table switch', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
mockAuthorizeRoom.mockResolvedValue({
allowed: true,
status: 200,
workspaceId: 'ws-1',
workspacePermission: 'admin',
})
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
// Two joins enqueued back-to-back (A then B). B bumps the generation synchronously, so A's
// queued run no-ops at its start check — only B commits. Because JOINs are serialized on one
// op chain, A's and B's Redis writes can never interleave (no map-clobber, no stranding).
handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-A' })
await handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-B' })
expect(socket.join).toHaveBeenCalledWith('table:table-B')
expect(socket.join).not.toHaveBeenCalledWith('table:table-A')
expect(roomManager.addUserToRoom).toHaveBeenCalledWith(
{ type: ROOM_TYPES.TABLE, id: 'table-B' },
'socket-1',
expect.anything()
)
expect(roomManager.addUserToRoom).not.toHaveBeenCalledWith(
{ type: ROOM_TYPES.TABLE, id: 'table-A' },
expect.anything(),
expect.anything()
)
})
it('aborts an in-flight join when a leave for that table arrives during authorize', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
let releaseAuth: (value: unknown) => void = () => {}
const pending = new Promise((resolve) => {
releaseAuth = resolve
})
mockAuthorizeRoom.mockReturnValueOnce(pending)
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
const joinPromise = handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-1' })
// Client navigates away while the join is still awaiting authorization.
await handlers[TABLE_PRESENCE_EVENTS.LEAVE]({ tableId: 'table-1' })
releaseAuth({ allowed: true, status: 200, workspaceId: 'ws-1', workspacePermission: 'admin' })
await joinPromise
// The cancelled join must touch no room state — the socket is not stranded.
expect(socket.join).not.toHaveBeenCalled()
expect(roomManager.addUserToRoom).not.toHaveBeenCalled()
expect(roomManager.broadcastPresenceUpdate).not.toHaveBeenCalledWith(TABLE_ROOM)
})
it('aborts an in-flight join when an unscoped leave arrives during authorize', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
let releaseAuth: (value: unknown) => void = () => {}
const pending = new Promise((resolve) => {
releaseAuth = resolve
})
mockAuthorizeRoom.mockReturnValueOnce(pending)
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
const joinPromise = handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-1' })
// A leave with no table id (view teardown) must also cancel the in-flight join.
await handlers[TABLE_PRESENCE_EVENTS.LEAVE](undefined)
releaseAuth({ allowed: true, status: 200, workspaceId: 'ws-1', workspacePermission: 'admin' })
await joinPromise
expect(socket.join).not.toHaveBeenCalled()
expect(roomManager.addUserToRoom).not.toHaveBeenCalled()
})
it('does not abort an in-flight join when a leave targets a different table', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
let releaseAuth: (value: unknown) => void = () => {}
const pending = new Promise((resolve) => {
releaseAuth = resolve
})
mockAuthorizeRoom.mockReturnValueOnce(pending)
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
const joinPromise = handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-B' })
// A stale/deferred leave for a table the client already left must NOT cancel the B join.
await handlers[TABLE_PRESENCE_EVENTS.LEAVE]({ tableId: 'table-A' })
releaseAuth({ allowed: true, status: 200, workspaceId: 'ws-1', workspacePermission: 'admin' })
await joinPromise
expect(socket.join).toHaveBeenCalledWith('table:table-B')
expect(roomManager.addUserToRoom).toHaveBeenCalledWith(
{ type: ROOM_TYPES.TABLE, id: 'table-B' },
'socket-1',
expect.anything()
)
})
it('leaves the table room on leave', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager({
getRoomForSocket: vi.fn().mockResolvedValue(TABLE_ROOM),
})
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
await handlers[TABLE_PRESENCE_EVENTS.LEAVE]({ tableId: 'table-1' })
expect(socket.leave).toHaveBeenCalledWith('table:table-1')
expect(roomManager.removeUserFromRoom).toHaveBeenCalledWith(TABLE_ROOM, 'socket-1')
expect(roomManager.broadcastPresenceUpdate).toHaveBeenCalledWith(TABLE_ROOM, 'socket-1')
})
it('rolls back the Socket.IO membership when a join fails mid-commit', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager({
// socket.join lands first, then the presence write throws — the socket is now in the
// Socket.IO room with no matching socket→room map entry, unreclaimable by any later op.
addUserToRoom: vi.fn().mockRejectedValue(new Error('redis down')),
})
setupTablesHandlers(socket as unknown as SetupArg, roomManager)
await handlers[TABLE_PRESENCE_EVENTS.JOIN]({ tableId: 'table-1' })
// The catch must always roll back the partial membership, not skip it.
expect(socket.leave).toHaveBeenCalledWith('table:table-1')
expect(roomManager.removeUserFromRoom).toHaveBeenCalledWith(TABLE_ROOM, 'socket-1')
expect(socket.emit).toHaveBeenCalledWith(
TABLE_PRESENCE_EVENTS.JOIN_ERROR,
expect.objectContaining({ code: 'JOIN_FAILED' })
)
})
})
+314
View File
@@ -0,0 +1,314 @@
import { createLogger } from '@sim/logger'
import { ROOM_TYPES, type RoomRef, roomName } from '@sim/realtime-protocol/rooms'
import {
type JoinTablePayload,
TABLE_PRESENCE_EVENTS,
type TableCellRef,
type TableCellSelection,
} from '@sim/realtime-protocol/table-presence'
import { resolveAvatarUrl } from '@/handlers/avatar'
import { resolveRoomJoinAuth } from '@/handlers/room-join-auth'
import type { AuthenticatedSocket } from '@/middleware/auth'
import type { IRoomManager, UserPresence } from '@/rooms'
import { filterVisiblePresence, sweepStalePresence } from '@/rooms/presence-visibility'
const logger = createLogger('TablePresenceHandlers')
/** Longest accepted row/column id — real ids are UUIDs/short ids; this bounds a hostile payload. */
const MAX_CELL_ID_LENGTH = 200
/** The table presence room ref for a table id. */
const tableRoom = (tableId: string): RoomRef => ({ type: ROOM_TYPES.TABLE, id: tableId })
function isCellRef(value: unknown): value is TableCellRef {
if (typeof value !== 'object' || value === null) return false
const ref = value as { rowId?: unknown; columnId?: unknown }
return (
typeof ref.rowId === 'string' &&
ref.rowId.length <= MAX_CELL_ID_LENGTH &&
typeof ref.columnId === 'string' &&
ref.columnId.length <= MAX_CELL_ID_LENGTH
)
}
/**
* Validate + whitelist an untrusted peer's selection before it is stored and
* rebroadcast (it ultimately flows into a DOM query on every viewer). Returns the
* normalized selection — `null` for a legitimately cleared selection — or `undefined`
* for anything malformed, so the caller drops it. Only the known fields survive, so a
* hostile client can't amplify an oversized object through the room.
*/
function normalizeCellSelection(cell: unknown): TableCellSelection | undefined {
if (cell === null) return null
if (typeof cell !== 'object') return undefined
const candidate = cell as { anchor?: unknown; focus?: unknown; editing?: unknown }
if (!isCellRef(candidate.anchor) || !isCellRef(candidate.focus)) return undefined
return {
anchor: { rowId: candidate.anchor.rowId, columnId: candidate.anchor.columnId },
focus: { rowId: candidate.focus.rowId, columnId: candidate.focus.columnId },
...(candidate.editing === true ? { editing: true } : {}),
}
}
/**
* Live cell-selection presence for the table grid. Mirrors the workspace-files
* join flow but is table-scoped (room id = tableId) with a bidirectional
* cell-selection channel — the grid analog of the workflow cursor/selection
* relay. Table *data* still flows through the one-way durable event stream
* (`lib/table/events.ts`); this socket carries only ephemeral presence.
*
* Table rooms are namespaced (`table:${id}`), so every broadcast targets
* `roomName(room)`, never the bare `room.id` (which the workflow handler can use
* only because a workflow room's name equals its id).
*/
export function setupTablesHandlers(socket: AuthenticatedSocket, roomManager: IRoomManager) {
// Monotonic per-socket generation: each JOIN/LEAVE bumps it synchronously on arrival, and a
// queued or in-flight op that finds a newer generation aborts — a fast table switch A→B thus
// cancels A the instant B arrives.
let joinGeneration = 0
// The table the socket currently intends to be in (set when a join is enqueued). A leave
// targeting it — or an unscoped leave — bumps the generation to cancel that join; a leave for a
// DIFFERENT table must NOT (a table switch), mirroring workspace-files.
let currentTableId: string | null = null
// Serialize this socket's room mutations (JOIN + LEAVE) so their multi-step async Redis commits
// can never interleave: two concurrent joins would otherwise race on the single-valued
// socket→room map (a late addUserToRoom clobbering a newer join's entry). This restores the
// atomic-commit property the synchronous sibling handlers (file-doc, workspace-files) get for
// free. CELL_SELECTION is NOT chained — it only touches presence activity, never the map.
let opChain: Promise<void> = Promise.resolve()
socket.on(TABLE_PRESENCE_EVENTS.JOIN, ({ tableId, tabSessionId }: JoinTablePayload) => {
// Validate the id BEFORE claiming a generation, so a malformed join can't advance
// joinGeneration and cancel a legitimate in-flight join for another table.
if (typeof tableId !== 'string' || tableId.length === 0) {
socket.emit(TABLE_PRESENCE_EVENTS.JOIN_ERROR, {
tableId: typeof tableId === 'string' ? tableId : '',
error: 'Invalid table id',
code: 'INVALID_PAYLOAD',
retryable: false,
})
return
}
const joinAttempt = (joinGeneration += 1)
currentTableId = tableId
opChain = opChain
.then(() => runJoin(tableId, tabSessionId, joinAttempt))
.catch((error) => logger.error('Error joining table room:', error))
// Returned so callers awaiting this op (e.g. tests) can await its completion; Socket.IO
// ignores a handler's return value.
return opChain
})
async function runJoin(tableId: string, tabSessionId: string | undefined, joinAttempt: number) {
// True once this JOIN has been superseded — a newer JOIN/LEAVE bumped joinGeneration, or the
// socket disconnected. Because ops are serialized, no other op mutates room state while this
// one runs, so only two checks are needed: skip a superseded queued op (here), and one final
// check right before the membership commit.
const superseded = () => joinGeneration !== joinAttempt || socket.disconnected
if (superseded()) return
try {
const userId = socket.userId
const userName = socket.userName
if (!userId || !userName) {
socket.emit(TABLE_PRESENCE_EVENTS.JOIN_ERROR, {
tableId,
error: 'Authentication required',
code: 'AUTHENTICATION_REQUIRED',
retryable: false,
})
return
}
if (!roomManager.isReady()) {
socket.emit(TABLE_PRESENCE_EVENTS.JOIN_ERROR, {
tableId,
error: 'Realtime unavailable',
code: 'ROOM_MANAGER_UNAVAILABLE',
retryable: true,
})
return
}
const room = tableRoom(tableId)
const authorized = await resolveRoomJoinAuth({
userId,
room,
action: 'read',
logger,
logLabel: `table room for ${userId}`,
messages: {
verifyFailed: 'Failed to verify table access',
notFound: 'Table not found',
accessDenied: 'Access denied to table',
},
emitError: ({ error, code, retryable }) =>
socket.emit(TABLE_PRESENCE_EVENTS.JOIN_ERROR, { tableId, error, code, retryable }),
})
if (!authorized) return
// Server-authenticated avatar for the presence roster.
const avatarUrl = await resolveAvatarUrl(socket, userId)
// Leave a previously-joined table room if switching tables. No generation guard is needed
// around this: serialization guarantees no concurrent op committed to a different room
// during the lookup, so `currentRoom` is the socket's genuine prior room, safe to leave.
const currentRoom = await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.TABLE)
if (currentRoom && currentRoom.id !== tableId) {
socket.leave(roomName(currentRoom))
await roomManager.removeUserFromRoom(currentRoom, socket.id)
await roomManager.broadcastPresenceUpdate(currentRoom)
}
// Reclaim presence orphaned by an ungraceful disconnect (no `disconnecting`
// event fires on a pod crash; the room hashes have no TTL). Returns the roster it
// read so the same-tab dedup below reuses it instead of issuing a second read.
const roster = await sweepStalePresence(roomManager, room)
// Clean up the same user's stale socket from the same tab (a reconnect that raced
// the old socket's disconnect), so presence shows one entry. Reuses the sweep's
// roster snapshot; re-removing an already-swept entry is a harmless no-op.
if (tabSessionId) {
for (const existing of roster) {
if (
existing.socketId !== socket.id &&
existing.userId === userId &&
existing.tabSessionId === tabSessionId
) {
await roomManager.removeUserFromRoom(room, existing.socketId)
await roomManager.io.in(existing.socketId).socketsLeave(roomName(room))
}
}
}
// Final re-check before the membership commit: a LEAVE or a newer JOIN enqueued during the
// awaits above bumped the generation, or the socket disconnected. Abort before registering.
if (superseded()) return
socket.join(roomName(room))
const presence: UserPresence = {
userId,
room,
userName,
socketId: socket.id,
tabSessionId,
joinedAt: Date.now(),
lastActivity: Date.now(),
role: authorized.workspacePermission ?? 'read',
avatarUrl,
}
// If the socket disconnects during this commit (disconnect cleanup runs off the op chain),
// this write can land after it, leaving a stale presence entry. Benign and self-correcting:
// filterVisiblePresence hides it and sweepStalePresence reclaims it (same as the siblings).
await roomManager.addUserToRoom(room, socket.id, presence)
// Filter the join ack to live members so a new joiner never briefly sees a
// ghost from an entry the sweep hasn't reclaimed yet.
const presenceUsers = await filterVisiblePresence(
roomManager.io,
room,
await roomManager.getRoomUsers(room)
)
socket.emit(TABLE_PRESENCE_EVENTS.JOIN_SUCCESS, {
tableId,
socketId: socket.id,
presenceUsers,
})
// Post-success, purely decorative: notify peers. The user is already joined and acked, so a
// Redis blip here must not surface as a join failure — swallow it (the next healthy broadcast
// reconciles peers). Kept OUT of the rollback catch below, which is only for pre-success failures.
try {
await roomManager.broadcastPresenceUpdate(room)
} catch (error) {
logger.warn(`Post-join presence broadcast failed for table room ${tableId}`, error)
}
logger.info(`User ${userId} (${userName}) joined table room ${tableId}`)
} catch (error) {
logger.error('Error joining table room:', error)
// Roll back a partial join: cleanup keys off the socket→room map, so a `socket.join` that
// landed without a matching `addUserToRoom` (a throw in between) would otherwise leave the
// socket stranded in the Socket.IO room, unreclaimable by any later op. A failure between the
// commit and the success ack rolls back too and surfaces a retryable error, so the client
// retries rather than hanging. Safe to run even when superseded — serialization means the
// newer op hasn't committed yet, so this touches only this join's own (this-table) state.
try {
const room = tableRoom(tableId)
socket.leave(roomName(room))
await roomManager.removeUserFromRoom(room, socket.id)
} catch {
// Best-effort rollback — the original join failure is the one surfaced below, so a
// secondary cleanup error must not mask it or throw out of the error handler.
}
// Suppress the client-facing error when this join was already superseded: the client has moved
// to a newer table, and a retryable error naming the abandoned one could make it re-join and
// supersede the newer join. The rollback above still runs.
if (superseded()) return
socket.emit(TABLE_PRESENCE_EVENTS.JOIN_ERROR, {
tableId,
error: 'Failed to join table',
code: 'JOIN_FAILED',
retryable: true,
})
}
}
socket.on(TABLE_PRESENCE_EVENTS.LEAVE, (payload?: { tableId?: string }) => {
// Cancel an in-flight/queued join whose table the client is now leaving (or an unscoped
// leave). Scope to the current table intent so a stale/deferred leave for a DIFFERENT table
// can't cancel the join the client has since switched to. Bumped synchronously here — before
// the teardown is enqueued — so it cancels a running join at its next generation check.
if (!payload?.tableId || payload.tableId === currentTableId) {
joinGeneration += 1
currentTableId = null
}
opChain = opChain
.then(() => runLeave(payload))
.catch((error) => logger.error('Error leaving table room:', error))
return opChain
})
async function runLeave(payload?: { tableId?: string }) {
try {
if (!roomManager.isReady()) return
const room = await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.TABLE)
if (!room) return
// Scope the leave to a specific table when the client provides one: a deferred leave from a
// prior view must not evict the socket from a room it has since switched into.
if (payload?.tableId && payload.tableId !== room.id) return
socket.leave(roomName(room))
await roomManager.removeUserFromRoom(room, socket.id)
await roomManager.broadcastPresenceUpdate(room, socket.id)
} catch (error) {
logger.error('Error leaving table room:', error)
}
}
socket.on(TABLE_PRESENCE_EVENTS.CELL_SELECTION, async ({ cell }: { cell: unknown }) => {
try {
// Drop a malformed/oversized selection from an untrusted peer before it is stored
// or rebroadcast (`undefined` = invalid; `null` = a legitimately cleared selection).
const selection = normalizeCellSelection(cell)
if (selection === undefined) return
const room = await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.TABLE)
if (!room) return
// Persist so a later joiner sees this viewer's current selection in the join ack.
await roomManager.updateUserActivity(room, socket.id, { cell: selection })
// Relay to peers (namespaced room → roomName, not room.id). Peers already know this
// socket's identity from the presence roster, so the delta carries only id + cell.
socket.to(roomName(room)).emit(TABLE_PRESENCE_EVENTS.CELL_SELECTION, {
socketId: socket.id,
cell: selection,
})
} catch (error) {
logger.error(`Error handling table cell selection for socket ${socket.id}:`, error)
}
})
}
+7 -5
View File
@@ -4,11 +4,12 @@ import { workflow } from '@sim/db/schema'
import { createLogger } from '@sim/logger'
import { assertWorkflowMutable, WorkflowLockedError } from '@sim/platform-authz/workflow'
import { VARIABLE_OPERATIONS } from '@sim/realtime-protocol/constants'
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { getErrorMessage } from '@sim/utils/errors'
import { eq } from 'drizzle-orm'
import type { AuthenticatedSocket } from '@/middleware/auth'
import { checkWorkflowOperationPermission } from '@/middleware/permissions'
import type { IRoomManager } from '@/rooms'
import { type IRoomManager, workflowRoom as wf } from '@/rooms'
const logger = createLogger('VariablesHandlers')
@@ -61,7 +62,8 @@ export function setupVariablesHandlers(socket: AuthenticatedSocket, roomManager:
}
try {
const sessionWorkflowId = await roomManager.getWorkflowIdForSocket(socket.id)
const sessionWorkflowId =
(await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.WORKFLOW))?.id ?? null
const session = await roomManager.getUserSession(socket.id)
if (!sessionWorkflowId || !session) {
@@ -98,7 +100,7 @@ export function setupVariablesHandlers(socket: AuthenticatedSocket, roomManager:
return
}
const hasRoom = await roomManager.hasWorkflowRoom(workflowId)
const hasRoom = await roomManager.hasRoom(wf(workflowId))
if (!hasRoom) {
logger.debug(`Ignoring variable update: workflow room not found`, {
socketId: socket.id,
@@ -109,7 +111,7 @@ export function setupVariablesHandlers(socket: AuthenticatedSocket, roomManager:
return
}
const users = await roomManager.getWorkflowUsers(workflowId)
const users = await roomManager.getRoomUsers(wf(workflowId))
const userPresence = users.find((user) => user.socketId === socket.id)
if (!userPresence) {
socket.emit('operation-forbidden', {
@@ -174,7 +176,7 @@ export function setupVariablesHandlers(socket: AuthenticatedSocket, roomManager:
}
// Update user activity
await roomManager.updateUserActivity(workflowId, socket.id, { lastActivity: Date.now() })
await roomManager.updateUserActivity(wf(workflowId), socket.id, { lastActivity: Date.now() })
const debouncedKey = `${workflowId}:${variableId}:${field}`
const existing = pendingVariableUpdates.get(debouncedKey)
+229 -19
View File
@@ -4,12 +4,21 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import type { IRoomManager } from '@/rooms'
const { mockGetWorkflowState, mockVerifyWorkflowAccess, mockResolveCurrentWorkflowRole } =
vi.hoisted(() => ({
mockGetWorkflowState: vi.fn(),
mockVerifyWorkflowAccess: vi.fn(),
mockResolveCurrentWorkflowRole: vi.fn(),
}))
const {
mockGetWorkflowState,
mockVerifyWorkflowAccess,
mockResolveCurrentWorkflowRole,
mockResolveAvatarUrl,
} = vi.hoisted(() => ({
mockGetWorkflowState: vi.fn(),
mockVerifyWorkflowAccess: vi.fn(),
mockResolveCurrentWorkflowRole: vi.fn(),
mockResolveAvatarUrl: vi.fn(),
}))
vi.mock('@/handlers/avatar', () => ({
resolveAvatarUrl: mockResolveAvatarUrl,
}))
vi.mock('@sim/db', () => ({
db: { select: vi.fn() },
@@ -33,13 +42,14 @@ interface JoinWorkflowPayload {
}
function createSocket(overrides?: Partial<Record<string, unknown>>) {
const handlers: Record<string, (payload: JoinWorkflowPayload) => Promise<void> | void> = {}
// leave-workflow takes no payload; join-workflow takes one — so the stored handler's arg is optional.
const handlers: Record<string, (payload?: JoinWorkflowPayload) => Promise<void> | void> = {}
const socket = {
id: 'socket-1',
userId: 'user-1',
userName: 'Test User',
userImage: 'avatar.png',
on: vi.fn((event: string, handler: (payload: JoinWorkflowPayload) => Promise<void> | void) => {
on: vi.fn((event: string, handler: (payload?: JoinWorkflowPayload) => Promise<void> | void) => {
handlers[event] = handler
}),
emit: vi.fn(),
@@ -57,21 +67,21 @@ function createSocket(overrides?: Partial<Record<string, unknown>>) {
function createRoomManager(overrides?: Partial<IRoomManager>): IRoomManager {
return {
isReady: vi.fn().mockReturnValue(true),
getWorkflowIdForSocket: vi.fn().mockResolvedValue(null),
removeUserFromRoom: vi.fn().mockResolvedValue(null),
getRoomForSocket: vi.fn().mockResolvedValue(null),
getRoomsForSocket: vi.fn().mockResolvedValue([]),
removeUserFromRoom: vi.fn().mockResolvedValue(false),
removeSocketFromAllRooms: vi.fn().mockResolvedValue([]),
broadcastPresenceUpdate: vi.fn().mockResolvedValue(undefined),
getWorkflowUsers: vi.fn().mockResolvedValue([]),
hasWorkflowRoom: vi.fn().mockResolvedValue(false),
getRoomUsers: vi.fn().mockResolvedValue([]),
hasRoom: vi.fn().mockResolvedValue(false),
deleteRoom: vi.fn().mockResolvedValue(undefined),
addUserToRoom: vi.fn().mockResolvedValue(undefined),
getUserSession: vi.fn().mockResolvedValue(null),
updateUserActivity: vi.fn().mockResolvedValue(undefined),
updateRoomLastModified: vi.fn().mockResolvedValue(undefined),
emitToWorkflow: vi.fn(),
emitToRoom: vi.fn(),
getUniqueUserCount: vi.fn().mockResolvedValue(1),
getTotalActiveConnections: vi.fn().mockResolvedValue(0),
handleWorkflowDeletion: vi.fn().mockResolvedValue(undefined),
handleWorkflowRevert: vi.fn().mockResolvedValue(undefined),
handleWorkflowUpdate: vi.fn().mockResolvedValue(undefined),
shutdown: vi.fn().mockResolvedValue(undefined),
initialize: vi.fn().mockResolvedValue(undefined),
io: {
@@ -90,6 +100,36 @@ describe('setupWorkflowHandlers', () => {
mockGetWorkflowState.mockResolvedValue({ id: 'workflow-1', state: {} })
mockVerifyWorkflowAccess.mockResolvedValue({ hasAccess: true, role: 'admin' })
mockResolveCurrentWorkflowRole.mockResolvedValue('admin')
mockResolveAvatarUrl.mockResolvedValue('avatar.png')
})
it('resolves the avatar before joining so no await sits between socket.join and addUserToRoom', async () => {
const order: string[] = []
mockResolveAvatarUrl.mockImplementation(async () => {
order.push('avatar')
return 'avatar.png'
})
const { socket, handlers } = createSocket({
join: vi.fn(() => {
order.push('join')
}),
})
const roomManager = createRoomManager({
addUserToRoom: vi.fn(async () => {
order.push('add')
}),
})
setupWorkflowHandlers(
socket as unknown as Parameters<typeof setupWorkflowHandlers>[0],
roomManager
)
await handlers['join-workflow']({ workflowId: 'workflow-1', tabSessionId: 'tab-1' })
// The avatar await must complete before socket.join; reintroducing it between
// join and addUserToRoom reopens the revoke-race ghost-presence window.
expect(order).toEqual(['avatar', 'join', 'add'])
})
it('includes workflowId when authentication is missing', async () => {
@@ -193,7 +233,7 @@ describe('setupWorkflowHandlers', () => {
expect(mockResolveCurrentWorkflowRole).toHaveBeenCalledWith('user-1', 'workflow-1', 'write')
expect(socket.join).toHaveBeenCalledWith('workflow-1')
expect(roomManager.addUserToRoom).toHaveBeenCalledWith(
'workflow-1',
{ type: 'workflow', id: 'workflow-1' },
'socket-1',
expect.objectContaining({ role: 'read' })
)
@@ -223,8 +263,8 @@ describe('setupWorkflowHandlers', () => {
it('includes workflowId when an unexpected join failure occurs', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager({
getWorkflowIdForSocket: vi.fn().mockRejectedValue(new Error('boom')),
removeUserFromRoom: vi.fn().mockResolvedValue(null),
getRoomForSocket: vi.fn().mockRejectedValue(new Error('boom')),
removeUserFromRoom: vi.fn().mockResolvedValue(false),
})
setupWorkflowHandlers(
@@ -241,4 +281,174 @@ describe('setupWorkflowHandlers', () => {
retryable: true,
})
})
it('cancels a superseded queued join on a fast workflow switch', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
setupWorkflowHandlers(
socket as unknown as Parameters<typeof setupWorkflowHandlers>[0],
roomManager
)
// Enqueue A without awaiting, then B: B bumps the generation synchronously, so A is superseded
// before its queued op runs and must never commit.
handlers['join-workflow']({ workflowId: 'workflow-a', tabSessionId: 'tab-1' })
await handlers['join-workflow']({ workflowId: 'workflow-b', tabSessionId: 'tab-1' })
expect(socket.join).toHaveBeenCalledWith('workflow-b')
expect(socket.join).not.toHaveBeenCalledWith('workflow-a')
expect(roomManager.addUserToRoom).toHaveBeenCalledWith(
{ type: 'workflow', id: 'workflow-b' },
'socket-1',
expect.anything()
)
expect(roomManager.addUserToRoom).not.toHaveBeenCalledWith(
{ type: 'workflow', id: 'workflow-a' },
'socket-1',
expect.anything()
)
})
it('does not let a malformed join cancel a valid in-flight join', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
setupWorkflowHandlers(
socket as unknown as Parameters<typeof setupWorkflowHandlers>[0],
roomManager
)
const validJoin = handlers['join-workflow']({ workflowId: 'workflow-a', tabSessionId: 'tab-1' })
// A malformed join arrives mid-flight — it must be rejected WITHOUT advancing the generation,
// so it can't supersede the valid join already in flight.
handlers['join-workflow']({ workflowId: '', tabSessionId: 'tab-1' })
await validJoin
expect(socket.emit).toHaveBeenCalledWith(
'join-workflow-error',
expect.objectContaining({ code: 'INVALID_PAYLOAD' })
)
// The valid join still committed — not superseded by the malformed one.
expect(socket.join).toHaveBeenCalledWith('workflow-a')
expect(roomManager.addUserToRoom).toHaveBeenCalledWith(
{ type: 'workflow', id: 'workflow-a' },
'socket-1',
expect.anything()
)
})
it('cancels an in-flight join when a leave is enqueued before it commits', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
setupWorkflowHandlers(
socket as unknown as Parameters<typeof setupWorkflowHandlers>[0],
roomManager
)
handlers['join-workflow']({ workflowId: 'workflow-1', tabSessionId: 'tab-1' })
await handlers['leave-workflow']()
expect(socket.join).not.toHaveBeenCalled()
expect(roomManager.addUserToRoom).not.toHaveBeenCalled()
})
it('rolls back the workflow membership when addUserToRoom fails mid-commit', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager({
addUserToRoom: vi.fn().mockRejectedValue(new Error('redis down')),
})
setupWorkflowHandlers(
socket as unknown as Parameters<typeof setupWorkflowHandlers>[0],
roomManager
)
await handlers['join-workflow']({ workflowId: 'workflow-1', tabSessionId: 'tab-1' })
expect(socket.leave).toHaveBeenCalledWith('workflow-1')
expect(roomManager.removeUserFromRoom).toHaveBeenCalledWith(
{ type: 'workflow', id: 'workflow-1' },
'socket-1'
)
expect(socket.emit).toHaveBeenCalledWith(
'join-workflow-error',
expect.objectContaining({ code: 'JOIN_WORKFLOW_FAILED' })
)
})
it('does not roll back a committed join when a post-success step fails', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager({
// Trailing broadcast (post-addUserToRoom, post-success-ack) fails on a Redis blip.
broadcastPresenceUpdate: vi.fn().mockRejectedValue(new Error('redis blip')),
})
setupWorkflowHandlers(
socket as unknown as Parameters<typeof setupWorkflowHandlers>[0],
roomManager
)
await handlers['join-workflow']({ workflowId: 'workflow-1', tabSessionId: 'tab-1' })
// The user is genuinely joined and was acked; the trailing failure must NOT tear them out.
expect(socket.emit).toHaveBeenCalledWith(
'join-workflow-success',
expect.objectContaining({ workflowId: 'workflow-1' })
)
expect(socket.leave).not.toHaveBeenCalled()
expect(roomManager.removeUserFromRoom).not.toHaveBeenCalled()
expect(socket.emit).not.toHaveBeenCalledWith('join-workflow-error', expect.anything())
})
it('rolls back and surfaces a retryable error when a pre-success step fails after commit', async () => {
// getWorkflowState runs after addUserToRoom but before the success ack — its failure must roll
// back and emit a retryable error so the client retries, never hanging committed-but-unacked.
mockGetWorkflowState.mockRejectedValue(new Error('db blip'))
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
setupWorkflowHandlers(
socket as unknown as Parameters<typeof setupWorkflowHandlers>[0],
roomManager
)
await handlers['join-workflow']({ workflowId: 'workflow-1', tabSessionId: 'tab-1' })
expect(socket.emit).not.toHaveBeenCalledWith('join-workflow-success', expect.anything())
expect(roomManager.removeUserFromRoom).toHaveBeenCalledWith(
{ type: 'workflow', id: 'workflow-1' },
'socket-1'
)
expect(socket.emit).toHaveBeenCalledWith(
'join-workflow-error',
expect.objectContaining({ code: 'JOIN_WORKFLOW_FAILED', retryable: true })
)
})
it('leaves the workflow room even when the session key has expired', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager({
getRoomForSocket: vi.fn().mockResolvedValue({ type: 'workflow', id: 'workflow-1' }),
getUserSession: vi.fn().mockResolvedValue(null),
})
setupWorkflowHandlers(
socket as unknown as Parameters<typeof setupWorkflowHandlers>[0],
roomManager
)
await handlers['leave-workflow']()
expect(socket.leave).toHaveBeenCalledWith('workflow-1')
expect(roomManager.removeUserFromRoom).toHaveBeenCalledWith(
{ type: 'workflow', id: 'workflow-1' },
'socket-1'
)
expect(roomManager.broadcastPresenceUpdate).toHaveBeenCalledWith({
type: 'workflow',
id: 'workflow-1',
})
})
})
+132 -61
View File
@@ -1,15 +1,58 @@
import { db, user } from '@sim/db'
import { createLogger } from '@sim/logger'
import { eq } from 'drizzle-orm'
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { getWorkflowState } from '@/database/operations'
import { resolveAvatarUrl } from '@/handlers/avatar'
import type { AuthenticatedSocket } from '@/middleware/auth'
import { resolveCurrentWorkflowRole, verifyWorkflowAccess } from '@/middleware/permissions'
import type { IRoomManager, UserPresence } from '@/rooms'
import { type IRoomManager, type UserPresence, workflowRoom as wf } from '@/rooms'
import { filterVisiblePresence } from '@/rooms/presence-visibility'
const logger = createLogger('WorkflowHandlers')
export function setupWorkflowHandlers(socket: AuthenticatedSocket, roomManager: IRoomManager) {
socket.on('join-workflow', async ({ workflowId, tabSessionId }) => {
// Monotonic per-socket generation: each JOIN/LEAVE bumps it synchronously on arrival, and a
// queued or in-flight op that finds a newer generation aborts — a fast workflow switch A→B thus
// cancels A the instant B arrives.
let joinGeneration = 0
// Serialize this socket's room mutations (JOIN + LEAVE) so their multi-step async Redis commits
// can never interleave: two concurrent joins would otherwise race on the single-valued
// socket→room map (a late addUserToRoom clobbering a newer join's entry, leaving the socket a
// ghost in the old room and receiving its operation broadcasts). This matches the sibling
// handlers (tables, file-doc, workspace-files).
let opChain: Promise<void> = Promise.resolve()
socket.on('join-workflow', ({ workflowId, tabSessionId }) => {
// Validate the id BEFORE claiming a generation, so a malformed join can't advance joinGeneration
// and cancel a legitimate in-flight switch (matches tables/workspace-files).
if (typeof workflowId !== 'string' || workflowId.length === 0) {
socket.emit('join-workflow-error', {
workflowId: typeof workflowId === 'string' ? workflowId : '',
error: 'Invalid workflow id',
code: 'INVALID_PAYLOAD',
retryable: false,
})
return
}
const joinAttempt = (joinGeneration += 1)
opChain = opChain
.then(() => runJoin(workflowId, tabSessionId, joinAttempt))
.catch((error) => logger.error('Error joining workflow:', error))
// Returned so callers awaiting this op (e.g. tests) can await its completion; Socket.IO
// ignores a handler's return value.
return opChain
})
async function runJoin(
workflowId: string,
tabSessionId: string | undefined,
joinAttempt: number
) {
// True once this JOIN has been superseded — a newer JOIN/LEAVE bumped joinGeneration, or the
// socket disconnected. Because ops are serialized, no other op mutates room state while this
// one runs, so only two checks are needed: skip a superseded queued op (here), and one final
// check right before the membership commit.
const superseded = () => joinGeneration !== joinAttempt || socket.disconnected
if (superseded()) return
try {
const userId = socket.userId
const userName = socket.userName
@@ -64,18 +107,19 @@ export function setupWorkflowHandlers(socket: AuthenticatedSocket, roomManager:
return
}
// Leave current room if in one
const currentWorkflowId = await roomManager.getWorkflowIdForSocket(socket.id)
if (currentWorkflowId) {
socket.leave(currentWorkflowId)
await roomManager.removeUserFromRoom(socket.id, currentWorkflowId)
await roomManager.broadcastPresenceUpdate(currentWorkflowId)
// Leave a previously-joined workflow room if switching workflows. Guard on a DIFFERENT id so a
// re-join of the SAME workflow doesn't leave→re-add and flicker presence for peers.
const currentRoom = await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.WORKFLOW)
if (currentRoom && currentRoom.id !== workflowId) {
socket.leave(currentRoom.id)
await roomManager.removeUserFromRoom(currentRoom, socket.id)
await roomManager.broadcastPresenceUpdate(currentRoom)
}
// Keep this above Redis socket key TTL (1h) so a normal idle user is not evicted too aggressively.
const STALE_THRESHOLD_MS = 75 * 60 * 1000
const now = Date.now()
const existingUsers = await roomManager.getWorkflowUsers(workflowId)
const existingUsers = await roomManager.getRoomUsers(wf(workflowId))
let liveSocketIds = new Set<string>()
let canCheckLiveness = false
@@ -106,7 +150,7 @@ export function setupWorkflowHandlers(socket: AuthenticatedSocket, roomManager:
logger.info(
`Cleaning up socket ${existingUser.socketId} for user ${existingUser.userId} (same tab)`
)
await roomManager.removeUserFromRoom(existingUser.socketId, workflowId)
await roomManager.removeUserFromRoom(wf(workflowId), existingUser.socketId)
await roomManager.io.in(existingUser.socketId).socketsLeave(workflowId)
continue
}
@@ -124,21 +168,30 @@ export function setupWorkflowHandlers(socket: AuthenticatedSocket, roomManager:
logger.info(
`Cleaning up socket ${existingUser.socketId} for user ${existingUser.userId} (stale activity)`
)
await roomManager.removeUserFromRoom(existingUser.socketId, workflowId)
await roomManager.removeUserFromRoom(wf(workflowId), existingUser.socketId)
await roomManager.io.in(existingUser.socketId).socketsLeave(workflowId)
} catch (error) {
logger.warn(`Best-effort cleanup failed for socket ${existingUser.socketId}`, error)
}
}
// Resolve the avatar before the critical section below. It is the only
// await that used to sit between socket.join and addUserToRoom, and a sweep
// eviction in that gap would socketsLeave the socket while its presence
// mapping did not yet exist — cleanupEvictedSocket would find nothing to
// remove, then this join would write presence for a socket already out of
// the room (a ghost collaborator until the stale sweep). Hoisting it keeps
// the whole re-auth -> socket.join -> addUserToRoom section await-free.
const avatarUrl = await resolveAvatarUrl(socket, userId)
// Re-authorize immediately before joining: the access-revalidation sweep
// may have evicted this socket while the awaits above were in flight, and
// its eviction is recorded in the shared role cache before it runs — so a
// revoked user resolves to null here. The resolver is single-flighted per
// (user, workflow), so this read cannot race the sweep's and overwrite a
// recorded revocation with a stale role; and no awaits sit between this
// check and socket.join, so a sweep eviction cannot interleave after it
// and be reversed by this join.
// check and addUserToRoom (avatar resolution is hoisted above), so a sweep
// eviction cannot interleave inside the join and be reversed by it.
const currentRole = await resolveCurrentWorkflowRole(userId, workflowId, userRole)
if (currentRole === null) {
logger.warn(
@@ -154,29 +207,19 @@ export function setupWorkflowHandlers(socket: AuthenticatedSocket, roomManager:
}
userRole = currentRole
// Final re-check before the membership commit: a LEAVE or a newer JOIN enqueued during the
// awaits above bumped the generation, or the socket disconnected. Abort before registering.
// (This guards against a superseding op; the avatar hoist above guards against the off-chain
// access-revalidation sweep, which does not bump the generation.)
if (superseded()) return
// Join the new room
socket.join(workflowId)
// Get avatar URL
let avatarUrl = socket.userImage || null
if (!avatarUrl) {
try {
const [userRecord] = await db
.select({ image: user.image })
.from(user)
.where(eq(user.id, userId))
.limit(1)
avatarUrl = userRecord?.image ?? null
} catch (error) {
logger.warn('Failed to load user avatar for presence', { userId, error })
}
}
// Create presence entry
const userPresence: UserPresence = {
userId,
workflowId,
room: wf(workflowId),
userName,
socketId: socket.id,
tabSessionId,
@@ -186,11 +229,16 @@ export function setupWorkflowHandlers(socket: AuthenticatedSocket, roomManager:
avatarUrl,
}
// Add user to room
await roomManager.addUserToRoom(workflowId, socket.id, userPresence)
// Add user to room — the membership commit.
await roomManager.addUserToRoom(wf(workflowId), socket.id, userPresence)
// Get current presence list for the join acknowledgment
const presenceUsers = await roomManager.getWorkflowUsers(workflowId)
// Get current presence list for the join acknowledgment, filtered to live members so a new
// joiner never sees a ghost from an entry the stale sweep hasn't reclaimed yet.
const presenceUsers = await filterVisiblePresence(
roomManager.io,
wf(workflowId),
await roomManager.getRoomUsers(wf(workflowId))
)
// Get workflow state
const workflowState = await getWorkflowState(workflowId)
@@ -205,18 +253,34 @@ export function setupWorkflowHandlers(socket: AuthenticatedSocket, roomManager:
// Send workflow state
socket.emit('workflow-state', workflowState)
// Broadcast presence update to all users in the room
await roomManager.broadcastPresenceUpdate(workflowId)
const uniqueUserCount = await roomManager.getUniqueUserCount(workflowId)
logger.info(
`User ${userId} (${userName}) joined workflow ${workflowId}. Room now has ${uniqueUserCount} unique users.`
)
// Post-success, purely decorative: notify peers and log the count. The user is already joined
// and acked, so a Redis blip here must not surface as a join failure — swallow it (the next
// healthy presence broadcast reconciles peers). It must stay OUT of the rollback catch below,
// which is only for pre-success failures.
try {
await roomManager.broadcastPresenceUpdate(wf(workflowId))
const uniqueUserCount = await roomManager.getUniqueUserCount(wf(workflowId))
logger.info(
`User ${userId} (${userName}) joined workflow ${workflowId}. Room now has ${uniqueUserCount} unique users.`
)
} catch (error) {
logger.warn(`Post-join presence broadcast failed for workflow ${workflowId}`, error)
}
} catch (error) {
logger.error('Error joining workflow:', error)
// Undo socket.join and room manager entry if any operation failed
// Roll back a partial join: cleanup keys off the socket→room map, so a `socket.join` that
// landed without a matching `addUserToRoom` (a throw in between) would otherwise strand the
// socket in the Socket.IO room, unreclaimable by any later op. A failure between the commit
// and the success ack rolls back too and surfaces a retryable error — so the client retries
// rather than hanging (never left committed-but-unacked). Safe even when superseded —
// serialization means the newer op hasn't committed yet, so this touches only this join's own
// room state, never the newer op's.
socket.leave(workflowId)
await roomManager.removeUserFromRoom(socket.id, workflowId)
await roomManager.removeUserFromRoom(wf(workflowId), socket.id)
// Suppress the client-facing error when this join was already superseded: the client has moved
// to a newer workflow, and a retryable error naming the abandoned one could make it re-join and
// supersede the newer join (an A/B flicker). The rollback above still runs.
if (superseded()) return
const isReady = roomManager.isReady()
socket.emit('join-workflow-error', {
workflowId,
@@ -225,26 +289,33 @@ export function setupWorkflowHandlers(socket: AuthenticatedSocket, roomManager:
retryable: true,
})
}
}
socket.on('leave-workflow', () => {
// A leave always cancels any in-flight/queued join for this socket (the client emits it with no
// payload — there is no partial-switch case as there is for tables). Bumped synchronously here,
// before the teardown is enqueued, so it cancels a running join at its next generation check.
joinGeneration += 1
opChain = opChain
.then(() => runLeave())
.catch((error) => logger.error('Error leaving workflow:', error))
return opChain
})
socket.on('leave-workflow', async () => {
async function runLeave() {
try {
if (!roomManager.isReady()) {
return
}
const workflowId = await roomManager.getWorkflowIdForSocket(socket.id)
const session = await roomManager.getUserSession(socket.id)
if (workflowId && session) {
socket.leave(workflowId)
await roomManager.removeUserFromRoom(socket.id, workflowId)
await roomManager.broadcastPresenceUpdate(workflowId)
logger.info(`User ${session.userId} (${session.userName}) left workflow ${workflowId}`)
}
if (!roomManager.isReady()) return
const room = await roomManager.getRoomForSocket(socket.id, ROOM_TYPES.WORKFLOW)
// The room ref alone is sufficient to leave; no session lookup is gated in front of it, so an
// idle user whose 1h session key expired (while the 24h room mapping is still live) can still
// leave cleanly instead of being stranded as a ghost until disconnect.
if (!room) return
socket.leave(room.id)
await roomManager.removeUserFromRoom(room, socket.id)
await roomManager.broadcastPresenceUpdate(room)
logger.info(`User ${socket.userId} (${socket.userName}) left workflow ${room.id}`)
} catch (error) {
logger.error('Error leaving workflow:', error)
}
})
}
}
@@ -0,0 +1,223 @@
/**
* @vitest-environment node
*/
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import type { IRoomManager } from '@/rooms'
const { mockAuthorizeRoom } = vi.hoisted(() => ({
mockAuthorizeRoom: vi.fn(),
}))
vi.mock('@sim/db', () => ({
db: { select: vi.fn() },
user: { image: 'image' },
}))
vi.mock('@sim/platform-authz/rooms', () => ({
authorizeRoom: mockAuthorizeRoom,
}))
import { setupWorkspaceInvalidationRoom } from '@/handlers/workspace-invalidation-room'
type Payload = { workspaceId?: string }
function createSocket(overrides?: Record<string, unknown>) {
const handlers: Record<string, (payload?: Payload) => Promise<void> | void> = {}
// Live Set so the handler's native `socket.rooms` membership tracking works in tests.
const rooms = new Set<string>()
const socket = {
id: 'socket-1',
userId: 'user-1',
userName: 'Test User',
userImage: 'avatar.png',
rooms,
on: vi.fn((event: string, handler: (payload?: Payload) => Promise<void> | void) => {
handlers[event] = handler
}),
emit: vi.fn(),
join: vi.fn((room: string) => rooms.add(room)),
leave: vi.fn((room: string) => rooms.delete(room)),
to: vi.fn().mockReturnValue({ emit: vi.fn() }),
...overrides,
}
return { handlers, socket, rooms }
}
function createRoomManager(overrides?: Partial<IRoomManager>): IRoomManager {
return {
isReady: vi.fn().mockReturnValue(true),
getRoomForSocket: vi.fn().mockResolvedValue(null),
getRoomsForSocket: vi.fn().mockResolvedValue([]),
removeUserFromRoom: vi.fn().mockResolvedValue(false),
removeSocketFromAllRooms: vi.fn().mockResolvedValue([]),
broadcastPresenceUpdate: vi.fn().mockResolvedValue(undefined),
getRoomUsers: vi.fn().mockResolvedValue([]),
hasRoom: vi.fn().mockResolvedValue(false),
deleteRoom: vi.fn().mockResolvedValue(undefined),
addUserToRoom: vi.fn().mockResolvedValue(undefined),
getUserSession: vi.fn().mockResolvedValue(null),
updateUserActivity: vi.fn().mockResolvedValue(undefined),
updateRoomLastModified: vi.fn().mockResolvedValue(undefined),
emitToRoom: vi.fn(),
getUniqueUserCount: vi.fn().mockResolvedValue(1),
getTotalActiveConnections: vi.fn().mockResolvedValue(0),
shutdown: vi.fn().mockResolvedValue(undefined),
initialize: vi.fn().mockResolvedValue(undefined),
io: {
in: vi.fn().mockReturnValue({ socketsLeave: vi.fn().mockResolvedValue(undefined) }),
},
...overrides,
} as unknown as IRoomManager
}
// The two presence-free live-list rooms share one implementation; run the whole suite against both
// so files and tables can never drift. Event names and room names derive from the room type.
describe.each([ROOM_TYPES.WORKSPACE_FILES, ROOM_TYPES.WORKSPACE_TABLES] as const)(
'setupWorkspaceInvalidationRoom(%s)',
(roomType) => {
const joinEvent = `join-${roomType}`
const successEvent = `${joinEvent}-success`
const errorEvent = `${joinEvent}-error`
const leaveEvent = `leave-${roomType}`
const roomOf = (workspaceId: string) => `${roomType}:${workspaceId}`
const setup = (socket: ReturnType<typeof createSocket>['socket'], roomManager: IRoomManager) =>
setupWorkspaceInvalidationRoom(
socket as unknown as Parameters<typeof setupWorkspaceInvalidationRoom>[0],
roomManager,
roomType
)
beforeEach(() => {
vi.clearAllMocks()
mockAuthorizeRoom.mockResolvedValue({
allowed: true,
status: 200,
workspaceId: 'ws-1',
workspacePermission: 'admin',
})
})
it('rejects join when the socket is not authenticated', async () => {
const { socket, handlers } = createSocket({ userId: undefined, userName: undefined })
setup(socket, createRoomManager())
await handlers[joinEvent]({ workspaceId: 'ws-1' })
expect(socket.emit).toHaveBeenCalledWith(errorEvent, {
workspaceId: 'ws-1',
error: 'Authentication required',
code: 'AUTHENTICATION_REQUIRED',
retryable: false,
})
})
it('rejects join with a retryable error when realtime is unavailable', async () => {
const { socket, handlers } = createSocket()
setup(socket, createRoomManager({ isReady: vi.fn().mockReturnValue(false) }))
await handlers[joinEvent]({ workspaceId: 'ws-1' })
expect(socket.emit).toHaveBeenCalledWith(
errorEvent,
expect.objectContaining({ code: 'ROOM_MANAGER_UNAVAILABLE', retryable: true })
)
})
it('rejects join when workspace access is denied', async () => {
mockAuthorizeRoom.mockResolvedValue({
allowed: false,
status: 403,
workspaceId: 'ws-1',
workspacePermission: null,
})
const { socket, handlers } = createSocket()
setup(socket, createRoomManager())
await handlers[joinEvent]({ workspaceId: 'ws-1' })
expect(socket.emit).toHaveBeenCalledWith(
errorEvent,
expect.objectContaining({ code: 'ACCESS_DENIED', retryable: false })
)
})
it('joins the room on success without any presence bookkeeping', async () => {
const { socket, handlers } = createSocket()
const roomManager = createRoomManager()
setup(socket, roomManager)
await handlers[joinEvent]({ workspaceId: 'ws-1' })
expect(socket.join).toHaveBeenCalledWith(roomOf('ws-1'))
expect(socket.emit).toHaveBeenCalledWith(successEvent, { workspaceId: 'ws-1' })
// The room is live-list-only: no room-manager presence is tracked or broadcast.
expect(roomManager.addUserToRoom).not.toHaveBeenCalled()
expect(roomManager.broadcastPresenceUpdate).not.toHaveBeenCalled()
})
it('leaves a previously-joined room when switching workspaces', async () => {
const { socket, handlers, rooms } = createSocket()
rooms.add(roomOf('ws-old'))
setup(socket, createRoomManager())
await handlers[joinEvent]({ workspaceId: 'ws-1' })
expect(socket.leave).toHaveBeenCalledWith(roomOf('ws-old'))
expect(socket.join).toHaveBeenCalledWith(roomOf('ws-1'))
})
it('leaves the scoped room on leave', () => {
const { socket, handlers, rooms } = createSocket()
rooms.add(roomOf('ws-1'))
setup(socket, createRoomManager())
handlers[leaveEvent]({ workspaceId: 'ws-1' })
expect(socket.leave).toHaveBeenCalledWith(roomOf('ws-1'))
})
it('cancels an in-flight join when the user leaves that workspace mid-authorize', async () => {
const { socket, handlers } = createSocket()
let resolveAuth: (value: unknown) => void = () => {}
mockAuthorizeRoom.mockReturnValue(
new Promise((resolve) => {
resolveAuth = resolve
})
)
setup(socket, createRoomManager())
// Join ws-1 is awaiting authorization when the view unmounts and leaves ws-1.
const joinPromise = handlers[joinEvent]({ workspaceId: 'ws-1' })
handlers[leaveEvent]({ workspaceId: 'ws-1' })
resolveAuth({ allowed: true, status: 200, workspaceId: 'ws-1', workspacePermission: 'admin' })
await joinPromise
// The stale join must NOT join the room the client has since left (no stranded membership).
expect(socket.join).not.toHaveBeenCalled()
expect(socket.emit).not.toHaveBeenCalledWith(successEvent, { workspaceId: 'ws-1' })
})
it('does not cancel an in-flight join when a deferred leave targets a different workspace', async () => {
const { socket, handlers } = createSocket()
let resolveAuth: (value: unknown) => void = () => {}
mockAuthorizeRoom.mockReturnValue(
new Promise((resolve) => {
resolveAuth = resolve
})
)
setup(socket, createRoomManager())
// The client has switched to ws-2 (join in-flight) when a stale leave for the prior ws-1 lands.
const joinPromise = handlers[joinEvent]({ workspaceId: 'ws-2' })
handlers[leaveEvent]({ workspaceId: 'ws-1' })
resolveAuth({ allowed: true, status: 200, workspaceId: 'ws-2', workspacePermission: 'admin' })
await joinPromise
// The deferred leave for ws-1 must not abort the join the client actually wants (ws-2).
expect(socket.join).toHaveBeenCalledWith(roomOf('ws-2'))
expect(socket.emit).toHaveBeenCalledWith(successEvent, { workspaceId: 'ws-2' })
})
}
)
@@ -0,0 +1,155 @@
import { createLogger } from '@sim/logger'
import { type RoomRef, type RoomType, roomName } from '@sim/realtime-protocol/rooms'
import { resolveRoomJoinAuth } from '@/handlers/room-join-auth'
import type { AuthenticatedSocket } from '@/middleware/auth'
import type { IRoomManager } from '@/rooms'
const logger = createLogger('WorkspaceInvalidationRoom')
interface JoinPayload {
workspaceId: string
}
/**
* Wires a workspace-scoped, presence-free "invalidation room" onto a socket: the client joins a
* room named after its workspace, and a `${roomType}-changed` event — fanned out by the server-side
* mutation path over HTTP — reaches every viewer so they refetch. This is the shared core behind the
* workspace-files and workspace-tables browsers; they differ only in `roomType` (which also derives
* the event names, since each room type's wire token IS its event stem: `join-${roomType}`,
* `leave-${roomType}`, `join-${roomType}-success/-error`, and the `${roomType}-changed` broadcast).
*
* These rooms carry NO presence — "who's in a resource" comes from the per-resource room (file-doc /
* table), and mutations go over HTTP. Membership is tracked natively by Socket.IO (`socket.rooms`),
* so a workspace switch just leaves the prior room — no room-manager presence bookkeeping to sync.
*/
export function setupWorkspaceInvalidationRoom(
socket: AuthenticatedSocket,
roomManager: IRoomManager,
roomType: RoomType
) {
const joinEvent = `join-${roomType}`
const leaveEvent = `leave-${roomType}`
const successEvent = `${joinEvent}-success`
const errorEvent = `${joinEvent}-error`
const roomPrefix = `${roomType}:`
const room = (workspaceId: string): RoomRef => ({ type: roomType, id: workspaceId })
// Monotonic per-socket join counter: each join captures its number and, after the async
// authorize, aborts if a newer intent has superseded it — a fast workspace switch A→B can
// otherwise let A's late completion leave B and strand the socket in A, missing B's
// `${roomType}-changed` invalidations.
let joinGeneration = 0
// The workspace the socket currently intends to be in (set when a join starts). A leave that
// targets this workspace — or an unscoped "leave all" — advances joinGeneration so an in-flight
// join is cancelled instead of completing after the view has closed. A stale/deferred leave for
// a DIFFERENT workspace must NOT advance it, or it would abort the join the client has since
// switched to (the bug that bit the file-doc room in #5941).
let currentWorkspace: string | null = null
socket.on(joinEvent, async ({ workspaceId }: JoinPayload) => {
// Validate synchronously BEFORE claiming a generation, so a rejected/malformed join can't
// advance joinGeneration and cancel a legitimate in-flight join for another workspace.
if (!socket.userId || !socket.userName) {
socket.emit(errorEvent, {
workspaceId,
error: 'Authentication required',
code: 'AUTHENTICATION_REQUIRED',
retryable: false,
})
return
}
if (!roomManager.isReady()) {
socket.emit(errorEvent, {
workspaceId,
error: 'Realtime unavailable',
code: 'ROOM_MANAGER_UNAVAILABLE',
retryable: true,
})
return
}
// Validate the client-supplied id before it reaches the DB query (join payloads are
// otherwise raw client input) and before advancing the generation.
if (typeof workspaceId !== 'string' || workspaceId.length === 0) {
socket.emit(errorEvent, {
workspaceId: typeof workspaceId === 'string' ? workspaceId : '',
error: 'Invalid workspace id',
code: 'INVALID_PAYLOAD',
retryable: false,
})
return
}
const joinAttempt = (joinGeneration += 1)
currentWorkspace = workspaceId
try {
const ref = room(workspaceId)
const authorized = await resolveRoomJoinAuth({
userId: socket.userId,
room: ref,
action: 'read',
logger,
logLabel: `${roomType} room for ${socket.userId}`,
messages: {
verifyFailed: 'Failed to verify workspace access',
notFound: 'Workspace not found',
accessDenied: 'Access denied to workspace',
},
emitError: ({ error, code, retryable }) =>
socket.emit(errorEvent, { workspaceId, error, code, retryable }),
})
if (!authorized) return
// A newer join started on this socket during authorize (or it dropped): abort so a
// stale join can't leave the room the client has since switched to.
if (joinGeneration !== joinAttempt || socket.disconnected) return
// Leave any previously-joined room of this type (workspace switch), read straight from the
// socket's native room membership so there's no presence store to keep in sync.
const target = roomName(ref)
for (const joined of socket.rooms) {
if (joined !== target && joined.startsWith(roomPrefix)) socket.leave(joined)
}
socket.join(target)
socket.emit(successEvent, { workspaceId })
} catch (error) {
logger.error(`Error joining ${roomType} room:`, error)
try {
socket.leave(roomName(room(workspaceId)))
} catch {}
// Suppress the client-facing error when this join was already superseded: the client has
// switched to a newer workspace, and a retryable error naming the abandoned one could make it
// re-join and cancel the newer join. The leave above still runs.
if (joinGeneration !== joinAttempt || socket.disconnected) return
socket.emit(errorEvent, {
workspaceId,
error: `Failed to join ${roomType}`,
code: 'JOIN_FAILED',
retryable: true,
})
}
})
socket.on(leaveEvent, (payload?: { workspaceId?: string }) => {
// Cancel an in-flight join whose target the client is now leaving: a join awaiting
// authorization when the view unmounts would otherwise complete afterwards and strand the
// socket in a room it has left. Only when the leave targets the current join intent (or is
// unscoped) — a deferred leave for a different workspace must not abort the join the client
// has since switched to.
if (!payload?.workspaceId || payload.workspaceId === currentWorkspace) {
joinGeneration += 1
currentWorkspace = null
}
// Scope the leave to a specific workspace when the client provides one: a deferred leave
// from a prior page must not evict a room the socket has since switched into.
const target = payload?.workspaceId ? roomName(room(payload.workspaceId)) : null
for (const joined of socket.rooms) {
if (!joined.startsWith(roomPrefix)) continue
if (target && joined !== target) continue
socket.leave(joined)
}
})
}
+19 -15
View File
@@ -4,11 +4,12 @@
* @vitest-environment node
*/
import { createServer, request as httpRequest } from 'http'
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { createMockLogger } from '@sim/testing'
import { randomInt } from '@sim/utils/random'
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'
import { createSocketIOServer } from '@/config/socket'
import { MemoryRoomManager } from '@/rooms'
import { MemoryRoomManager, workflowRoom } from '@/rooms'
import { createHttpHandler } from '@/routes/http'
vi.mock('@/auth', () => ({
@@ -230,9 +231,10 @@ describe('Socket Server Index Integration', () => {
const workflowId = 'test-workflow-123'
const socketId = 'test-socket-123'
const room = workflowRoom(workflowId)
const presence = {
userId: 'user-123',
workflowId,
room,
userName: 'Test User',
socketId,
joinedAt: Date.now(),
@@ -240,10 +242,10 @@ describe('Socket Server Index Integration', () => {
role: 'admin',
}
await roomManager.addUserToRoom(workflowId, socketId, presence)
await roomManager.addUserToRoom(room, socketId, presence)
expect(await roomManager.hasWorkflowRoom(workflowId)).toBe(true)
const users = await roomManager.getWorkflowUsers(workflowId)
expect(await roomManager.hasRoom(room)).toBe(true)
const users = await roomManager.getRoomUsers(room)
expect(users).toHaveLength(1)
expect(users[0].socketId).toBe(socketId)
})
@@ -252,9 +254,10 @@ describe('Socket Server Index Integration', () => {
const socketId = 'test-socket-123'
const workflowId = 'test-workflow-456'
const room = workflowRoom(workflowId)
const presence = {
userId: 'user-123',
workflowId,
room,
userName: 'Test User',
socketId,
joinedAt: Date.now(),
@@ -262,9 +265,9 @@ describe('Socket Server Index Integration', () => {
role: 'admin',
}
await roomManager.addUserToRoom(workflowId, socketId, presence)
await roomManager.addUserToRoom(room, socketId, presence)
expect(await roomManager.getWorkflowIdForSocket(socketId)).toBe(workflowId)
expect(await roomManager.getRoomForSocket(socketId, ROOM_TYPES.WORKFLOW)).toEqual(room)
const session = await roomManager.getUserSession(socketId)
expect(session).toBeDefined()
expect(session?.userId).toBe('user-123')
@@ -274,9 +277,10 @@ describe('Socket Server Index Integration', () => {
const workflowId = 'test-workflow-789'
const socketId = 'test-socket-789'
const room = workflowRoom(workflowId)
const presence = {
userId: 'user-789',
workflowId,
room,
userName: 'Test User',
socketId,
joinedAt: Date.now(),
@@ -284,16 +288,16 @@ describe('Socket Server Index Integration', () => {
role: 'admin',
}
await roomManager.addUserToRoom(workflowId, socketId, presence)
await roomManager.addUserToRoom(room, socketId, presence)
expect(await roomManager.hasWorkflowRoom(workflowId)).toBe(true)
expect(await roomManager.hasRoom(room)).toBe(true)
// Remove user
await roomManager.removeUserFromRoom(socketId)
await roomManager.removeUserFromRoom(room, socketId)
// Room should be cleaned up since it's now empty
expect(await roomManager.hasWorkflowRoom(workflowId)).toBe(false)
expect(await roomManager.getWorkflowIdForSocket(socketId)).toBeNull()
expect(await roomManager.hasRoom(room)).toBe(false)
expect(await roomManager.getRoomForSocket(socketId, ROOM_TYPES.WORKFLOW)).toBeNull()
})
})
@@ -324,7 +328,7 @@ describe('Socket Server Index Integration', () => {
expect(typeof roomManager.addUserToRoom).toBe('function')
expect(typeof roomManager.removeUserFromRoom).toBe('function')
expect(typeof roomManager.handleWorkflowDeletion).toBe('function')
expect(typeof roomManager.removeSocketFromAllRooms).toBe('function')
expect(typeof roomManager.broadcastPresenceUpdate).toBe('function')
})
})
+36
View File
@@ -6,6 +6,8 @@ import { createSocketIOServer, shutdownSocketIOAdapter } from '@/config/socket'
import { assertSchemaCompatibility } from '@/database/preflight'
import { env } from '@/env'
import { setupAllHandlers } from '@/handlers'
import { flushAllFileDocRooms } from '@/handlers/file-doc'
import { getFileDocStore, initFileDocStore } from '@/handlers/file-doc-store'
import { type AuthenticatedSocket, authenticateSocket } from '@/middleware/auth'
import { type IRoomManager, MemoryRoomManager, RedisRoomManager } from '@/rooms'
import { createHttpHandler } from '@/routes/http'
@@ -55,6 +57,10 @@ async function main() {
// Initialize room manager (Redis or in-memory based on config)
const roomManager = await createRoomManager(io)
// Initialize the shared Yjs backend for collaborative file docs (Redis Streams). Enabled only when
// REDIS_URL is set; otherwise the relay runs its original single-replica in-memory doc path.
await initFileDocStore(env.REDIS_URL)
// Set up authentication middleware
io.use(authenticateSocket)
@@ -106,11 +112,26 @@ async function main() {
logger.info(`Health check available at: http://localhost:${PORT}/health`)
})
let shuttingDown = false
const shutdown = async () => {
// SIGINT and SIGTERM both bind this; a double signal (or SIGTERM then SIGINT during the drain)
// must not run the whole teardown twice — that means a second forced-exit timer and a second
// Redis quit (which throws "The client is closed").
if (shuttingDown) return
shuttingDown = true
logger.info('Shutting down Socket.IO server...')
accessRevalidation.stop()
// Flush open collaborative docs to durable markdown BEFORE tearing down Redis/the store — the
// per-socket disconnect flush is fire-and-forget and would race process exit.
try {
await flushAllFileDocRooms()
logger.info('Flushed open collaborative documents')
} catch (error) {
logger.error('Error flushing collaborative documents on shutdown:', error)
}
try {
await roomManager.shutdown()
logger.info('RoomManager shutdown complete')
@@ -124,6 +145,21 @@ async function main() {
logger.error('Error during Socket.IO adapter shutdown:', error)
}
try {
await getFileDocStore().shutdown()
} catch (error) {
logger.error('Error during FileDocStore shutdown:', error)
}
// Close local client connections so `httpServer.close()` can complete its callback and exit
// gracefully — otherwise open websockets keep it hanging until the forced-exit timer below.
// Local-only: a rolling deploy must not disconnect clients pinned to other pods.
try {
io.local.disconnectSockets(true)
} catch (error) {
logger.error('Error disconnecting sockets on shutdown:', error)
}
httpServer.close(() => {
logger.info('Socket.IO server closed')
process.exit(0)
+2 -1
View File
@@ -1,3 +1,4 @@
export { MemoryRoomManager } from '@/rooms/memory-manager'
export { RedisRoomManager } from '@/rooms/redis-manager'
export type { IRoomManager, UserPresence, UserSession, WorkflowRoom } from '@/rooms/types'
export type { IRoomManager, RoomState, UserPresence, UserSession } from '@/rooms/types'
export { WorkflowRoomService, workflowRoom } from '@/rooms/workflow-room-service'
@@ -0,0 +1,209 @@
/**
* Multi-room semantics for the room manager. Exercises the invariants the
* single-room → multi-room migration must preserve: a socket in two rooms,
* refcounted session cleanup, presence isolation, and full-disconnect cleanup.
*
* @vitest-environment node
*/
import { ROOM_TYPES, type RoomRef } from '@sim/realtime-protocol/rooms'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { MemoryRoomManager } from '@/rooms/memory-manager'
import { sweepStalePresence } from '@/rooms/presence-visibility'
import type { UserPresence } from '@/rooms/types'
function fakeIo(liveSocketIds: string[] = []) {
const emit = vi.fn()
return {
emit,
io: {
to: vi.fn().mockReturnValue({ emit }),
in: vi.fn().mockReturnValue({
fetchSockets: vi.fn().mockResolvedValue(liveSocketIds.map((id) => ({ id }))),
}),
} as never,
}
}
function presence(room: RoomRef, socketId: string, userId: string): UserPresence {
return {
userId,
room,
userName: `user-${userId}`,
socketId,
joinedAt: Date.now(),
lastActivity: Date.now(),
role: 'admin',
}
}
const WORKFLOW: RoomRef = { type: ROOM_TYPES.WORKFLOW, id: 'wf-1' }
const FILES: RoomRef = { type: ROOM_TYPES.WORKSPACE_FILES, id: 'ws-1' }
describe('MemoryRoomManager multi-room', () => {
let manager: MemoryRoomManager
beforeEach(async () => {
manager = new MemoryRoomManager(fakeIo().io)
await manager.initialize()
})
it('tracks a single socket in two rooms of different types', async () => {
await manager.addUserToRoom(WORKFLOW, 'socket-1', presence(WORKFLOW, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
const rooms = await manager.getRoomsForSocket('socket-1')
expect(rooms).toHaveLength(2)
expect(rooms).toContainEqual(WORKFLOW)
expect(rooms).toContainEqual(FILES)
expect(await manager.getRoomForSocket('socket-1', ROOM_TYPES.WORKFLOW)).toEqual(WORKFLOW)
expect(await manager.getRoomForSocket('socket-1', ROOM_TYPES.WORKSPACE_FILES)).toEqual(FILES)
})
it('keeps the shared session alive when leaving one of two rooms (refcount)', async () => {
await manager.addUserToRoom(WORKFLOW, 'socket-1', presence(WORKFLOW, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
const removed = await manager.removeUserFromRoom(WORKFLOW, 'socket-1')
expect(removed).toBe(true)
// The files room and the shared session must survive.
expect(await manager.hasRoom(WORKFLOW)).toBe(false)
expect(await manager.hasRoom(FILES)).toBe(true)
expect(await manager.getUserSession('socket-1')).not.toBeNull()
expect(await manager.getRoomForSocket('socket-1', ROOM_TYPES.WORKFLOW)).toBeNull()
expect(await manager.getRoomForSocket('socket-1', ROOM_TYPES.WORKSPACE_FILES)).toEqual(FILES)
})
it('drops the shared session only when the last room is left', async () => {
await manager.addUserToRoom(WORKFLOW, 'socket-1', presence(WORKFLOW, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
await manager.removeUserFromRoom(WORKFLOW, 'socket-1')
expect(await manager.getUserSession('socket-1')).not.toBeNull()
await manager.removeUserFromRoom(FILES, 'socket-1')
expect(await manager.getUserSession('socket-1')).toBeNull()
expect(await manager.getRoomsForSocket('socket-1')).toHaveLength(0)
})
it('isolates presence between rooms of different types', async () => {
await manager.addUserToRoom(WORKFLOW, 'socket-1', presence(WORKFLOW, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-2', presence(FILES, 'socket-2', 'user-2'))
expect(await manager.getRoomUsers(WORKFLOW)).toHaveLength(1)
expect(await manager.getRoomUsers(FILES)).toHaveLength(2)
})
it('removes a socket from every room on disconnect and reports them', async () => {
await manager.addUserToRoom(WORKFLOW, 'socket-1', presence(WORKFLOW, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-2', presence(FILES, 'socket-2', 'user-2'))
const removed = await manager.removeSocketFromAllRooms('socket-1')
expect(removed).toHaveLength(2)
expect(removed).toContainEqual(WORKFLOW)
expect(removed).toContainEqual(FILES)
expect(await manager.hasRoom(WORKFLOW)).toBe(false)
// The files room still has socket-2.
expect(await manager.getRoomUsers(FILES)).toHaveLength(1)
expect(await manager.getUserSession('socket-1')).toBeNull()
})
it('does not clobber another type when two sockets share a room', async () => {
await manager.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-2', presence(FILES, 'socket-2', 'user-2'))
await manager.removeUserFromRoom(FILES, 'socket-1')
expect(await manager.hasRoom(FILES)).toBe(true)
expect(await manager.getUserSession('socket-2')).not.toBeNull()
})
it('sweepStalePresence reclaims not-live stale entries but keeps live and fresh ones', async () => {
const { io } = fakeIo(['socket-live'])
const m = new MemoryRoomManager(io)
await m.initialize()
const staleMs = 76 * 60 * 1000
await m.addUserToRoom(FILES, 'socket-live', presence(FILES, 'socket-live', 'u1'))
await m.addUserToRoom(FILES, 'socket-dead', {
...presence(FILES, 'socket-dead', 'u2'),
joinedAt: Date.now() - staleMs,
lastActivity: Date.now() - staleMs,
})
await m.addUserToRoom(FILES, 'socket-recent', presence(FILES, 'socket-recent', 'u3'))
await sweepStalePresence(m, FILES)
const remaining = (await m.getRoomUsers(FILES)).map((u) => u.socketId).sort()
// socket-dead: not live + stale → removed. socket-live: live → kept.
// socket-recent: not live but fresh (transient) → kept.
expect(remaining).toEqual(['socket-live', 'socket-recent'])
})
it('deleteRoom unconditionally drops all room state', async () => {
await manager.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
await manager.addUserToRoom(FILES, 'socket-2', presence(FILES, 'socket-2', 'user-2'))
expect(await manager.hasRoom(FILES)).toBe(true)
await manager.deleteRoom(FILES)
expect(await manager.hasRoom(FILES)).toBe(false)
expect(await manager.getRoomUsers(FILES)).toHaveLength(0)
})
it('ignores removal of a room the socket is not in (id-guarded)', async () => {
await manager.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
// Removing a workflow room the socket never joined must be a no-op — it must
// not wipe the files mapping or the shared session.
const removed = await manager.removeUserFromRoom(WORKFLOW, 'socket-1')
expect(removed).toBe(false)
expect(await manager.hasRoom(FILES)).toBe(true)
expect(await manager.getUserSession('socket-1')).not.toBeNull()
expect(await manager.getRoomForSocket('socket-1', ROOM_TYPES.WORKSPACE_FILES)).toEqual(FILES)
})
it('broadcasts presence on the room-type-specific event name', async () => {
const { emit, io } = fakeIo()
const m = new MemoryRoomManager(io)
await m.initialize()
await m.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
await m.broadcastPresenceUpdate(FILES)
expect(emit).toHaveBeenCalledWith('workspace-files:presence-update', expect.any(Array))
await m.broadcastPresenceUpdate(WORKFLOW)
expect(emit).toHaveBeenCalledWith('presence-update', expect.any(Array))
})
it('omits an excluded socket from the presence broadcast (disconnect ghost guard)', async () => {
const { emit, io } = fakeIo()
const m = new MemoryRoomManager(io)
await m.initialize()
await m.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
await m.addUserToRoom(FILES, 'socket-2', presence(FILES, 'socket-2', 'user-2'))
// Broadcast as if socket-1 is disconnecting: even though its presence entry is
// still present, it must not appear in the emitted list.
await m.broadcastPresenceUpdate(FILES, 'socket-1')
const emitted = emit.mock.calls.at(-1)?.[1] as Array<{ socketId: string }>
expect(emitted.map((u) => u.socketId)).toEqual(['socket-2'])
})
it('never emits a presence entry whose socket is no longer live (ghost guard)', async () => {
// Only socket-2 is a live Socket.IO member; socket-1 is an orphaned entry that
// outlived a failed removal.
const { emit, io } = fakeIo(['socket-2'])
const m = new MemoryRoomManager(io)
await m.initialize()
await m.addUserToRoom(FILES, 'socket-1', presence(FILES, 'socket-1', 'user-1'))
await m.addUserToRoom(FILES, 'socket-2', presence(FILES, 'socket-2', 'user-2'))
await m.broadcastPresenceUpdate(FILES)
const emitted = emit.mock.calls.at(-1)?.[1] as Array<{ socketId: string }>
expect(emitted.map((u) => u.socketId)).toEqual(['socket-2'])
})
})
+115 -180
View File
@@ -1,16 +1,30 @@
import { createLogger } from '@sim/logger'
import {
presenceEventName,
type RoomRef,
type RoomType,
roomName,
} from '@sim/realtime-protocol/rooms'
import type { Server } from 'socket.io'
import type { IRoomManager, UserPresence, UserSession, WorkflowRoom } from '@/rooms/types'
import { filterVisiblePresence } from '@/rooms/presence-visibility'
import type { IRoomManager, RoomState, UserPresence, UserSession } from '@/rooms/types'
const logger = createLogger('MemoryRoomManager')
/** Stable string key for a room in the local maps (distinct from the Socket.IO room name). */
function roomKey(room: RoomRef): string {
return `${room.type}:${room.id}`
}
/**
* In-memory room manager for single-pod deployments
* Used as fallback when REDIS_URL is not configured
* In-memory room manager for single-pod deployments. Used when REDIS_URL is not
* configured. Domain-neutral: keyed by {@link RoomRef}, supports a socket in
* multiple rooms (one per {@link RoomType}).
*/
export class MemoryRoomManager implements IRoomManager {
private workflowRooms = new Map<string, WorkflowRoom>()
private socketToWorkflow = new Map<string, string>()
private rooms = new Map<string, RoomState>()
/** socketId -> (roomType -> roomId) */
private socketRooms = new Map<string, Map<RoomType, string>>()
private userSessions = new Map<string, UserSession>()
private _io: Server
@@ -31,235 +45,156 @@ export class MemoryRoomManager implements IRoomManager {
}
async shutdown(): Promise<void> {
this.workflowRooms.clear()
this.socketToWorkflow.clear()
this.rooms.clear()
this.socketRooms.clear()
this.userSessions.clear()
logger.info('MemoryRoomManager shutdown complete')
}
async addUserToRoom(workflowId: string, socketId: string, presence: UserPresence): Promise<void> {
// Create room if it doesn't exist
if (!this.workflowRooms.has(workflowId)) {
this.workflowRooms.set(workflowId, {
workflowId,
users: new Map(),
lastModified: Date.now(),
activeConnections: 0,
})
async addUserToRoom(room: RoomRef, socketId: string, presence: UserPresence): Promise<void> {
const key = roomKey(room)
let state = this.rooms.get(key)
if (!state) {
state = { room, users: new Map(), lastModified: Date.now(), activeConnections: 0 }
this.rooms.set(key, state)
}
const room = this.workflowRooms.get(workflowId)!
room.users.set(socketId, presence)
room.activeConnections++
room.lastModified = Date.now()
state.users.set(socketId, presence)
state.activeConnections++
state.lastModified = Date.now()
// Map socket to workflow
this.socketToWorkflow.set(socketId, workflowId)
let socketRoomMap = this.socketRooms.get(socketId)
if (!socketRoomMap) {
socketRoomMap = new Map()
this.socketRooms.set(socketId, socketRoomMap)
}
socketRoomMap.set(room.type, room.id)
// Store session
this.userSessions.set(socketId, {
userId: presence.userId,
userName: presence.userName,
avatarUrl: presence.avatarUrl,
})
logger.debug(`Added user ${presence.userId} to workflow ${workflowId} (socket: ${socketId})`)
logger.debug(`Added user ${presence.userId} to room ${key} (socket: ${socketId})`)
}
async removeUserFromRoom(socketId: string, workflowIdHint?: string): Promise<string | null> {
const currentWorkflowId = this.socketToWorkflow.get(socketId) ?? null
const workflowId = workflowIdHint ?? currentWorkflowId
async removeUserFromRoom(room: RoomRef, socketId: string): Promise<boolean> {
const key = roomKey(room)
const state = this.rooms.get(key)
let existed = false
if (!workflowId) {
return null
}
const room = this.workflowRooms.get(workflowId)
if (room) {
if (room.users.delete(socketId)) {
room.activeConnections = Math.max(0, room.activeConnections - 1)
}
// Clean up empty rooms
if (room.activeConnections === 0) {
this.workflowRooms.delete(workflowId)
logger.info(`Cleaned up empty workflow room: ${workflowId}`)
if (state?.users.has(socketId)) {
existed = true
state.users.delete(socketId)
state.activeConnections = Math.max(0, state.activeConnections - 1)
if (state.users.size === 0) {
this.rooms.delete(key)
logger.info(`Cleaned up empty room: ${key}`)
}
}
// Only clear the socket's own mappings when it is not mapped to a different
// room — removing a stale room's entry must not destroy the mapping of a
// room the socket has since moved to.
if (currentWorkflowId === null || currentWorkflowId === workflowId) {
this.socketToWorkflow.delete(socketId)
const socketRoomMap = this.socketRooms.get(socketId)
if (socketRoomMap && socketRoomMap.get(room.type) === room.id) {
socketRoomMap.delete(room.type)
// Drop the shared session only when the socket has left its last room.
if (socketRoomMap.size === 0) {
this.socketRooms.delete(socketId)
this.userSessions.delete(socketId)
}
}
return existed
}
async removeSocketFromAllRooms(socketId: string): Promise<RoomRef[]> {
const socketRoomMap = this.socketRooms.get(socketId)
if (!socketRoomMap || socketRoomMap.size === 0) {
this.userSessions.delete(socketId)
return []
}
logger.debug(`Removed socket ${socketId} from workflow ${workflowId}`)
return workflowId
const rooms: RoomRef[] = Array.from(socketRoomMap.entries()).map(([type, id]) => ({ type, id }))
for (const room of rooms) {
await this.removeUserFromRoom(room, socketId)
}
// Belt-and-suspenders: ensure session is gone even if the map drifted.
this.socketRooms.delete(socketId)
this.userSessions.delete(socketId)
return rooms
}
async getWorkflowIdForSocket(socketId: string): Promise<string | null> {
return this.socketToWorkflow.get(socketId) ?? null
async getRoomsForSocket(socketId: string): Promise<RoomRef[]> {
const socketRoomMap = this.socketRooms.get(socketId)
if (!socketRoomMap) return []
return Array.from(socketRoomMap.entries()).map(([type, id]) => ({ type, id }))
}
async getRoomForSocket(socketId: string, type: RoomType): Promise<RoomRef | null> {
const id = this.socketRooms.get(socketId)?.get(type)
return id ? { type, id } : null
}
async getUserSession(socketId: string): Promise<UserSession | null> {
return this.userSessions.get(socketId) ?? null
}
async getWorkflowUsers(workflowId: string): Promise<UserPresence[]> {
const room = this.workflowRooms.get(workflowId)
if (!room) return []
return Array.from(room.users.values())
async getRoomUsers(room: RoomRef): Promise<UserPresence[]> {
const state = this.rooms.get(roomKey(room))
if (!state) return []
return Array.from(state.users.values())
}
async hasWorkflowRoom(workflowId: string): Promise<boolean> {
return this.workflowRooms.has(workflowId)
async hasRoom(room: RoomRef): Promise<boolean> {
return this.rooms.has(roomKey(room))
}
async deleteRoom(room: RoomRef): Promise<void> {
this.rooms.delete(roomKey(room))
}
async updateUserActivity(
workflowId: string,
room: RoomRef,
socketId: string,
updates: Partial<Pick<UserPresence, 'cursor' | 'selection' | 'lastActivity'>>
updates: Partial<Pick<UserPresence, 'cursor' | 'selection' | 'cell' | 'lastActivity'>>
): Promise<void> {
const room = this.workflowRooms.get(workflowId)
if (!room) return
const presence = this.rooms.get(roomKey(room))?.users.get(socketId)
if (!presence) return
const presence = room.users.get(socketId)
if (presence) {
if (updates.cursor !== undefined) presence.cursor = updates.cursor
if (updates.selection !== undefined) presence.selection = updates.selection
presence.lastActivity = updates.lastActivity ?? Date.now()
}
if (updates.cursor !== undefined) presence.cursor = updates.cursor
if (updates.selection !== undefined) presence.selection = updates.selection
if (updates.cell !== undefined) presence.cell = updates.cell
presence.lastActivity = updates.lastActivity ?? Date.now()
}
async updateRoomLastModified(workflowId: string): Promise<void> {
const room = this.workflowRooms.get(workflowId)
if (room) {
room.lastModified = Date.now()
}
async updateRoomLastModified(room: RoomRef): Promise<void> {
const state = this.rooms.get(roomKey(room))
if (state) state.lastModified = Date.now()
}
async broadcastPresenceUpdate(workflowId: string): Promise<void> {
const users = await this.getWorkflowUsers(workflowId)
this._io.to(workflowId).emit('presence-update', users)
async broadcastPresenceUpdate(room: RoomRef, excludeSocketId?: string): Promise<void> {
const users = await this.getRoomUsers(room)
const visible = await filterVisiblePresence(this._io, room, users, excludeSocketId)
this._io.to(roomName(room)).emit(presenceEventName(room.type), visible)
}
emitToWorkflow<T = unknown>(workflowId: string, event: string, payload: T): void {
this._io.to(workflowId).emit(event, payload)
emitToRoom<T = unknown>(room: RoomRef, event: string, payload: T): void {
this._io.to(roomName(room)).emit(event, payload)
}
async getUniqueUserCount(workflowId: string): Promise<number> {
const room = this.workflowRooms.get(workflowId)
if (!room) return 0
async getUniqueUserCount(room: RoomRef): Promise<number> {
const state = this.rooms.get(roomKey(room))
if (!state) return 0
const uniqueUsers = new Set<string>()
room.users.forEach((presence) => {
uniqueUsers.add(presence.userId)
})
state.users.forEach((presence) => uniqueUsers.add(presence.userId))
return uniqueUsers.size
}
async getTotalActiveConnections(): Promise<number> {
let total = 0
for (const room of this.workflowRooms.values()) {
total += room.activeConnections
for (const state of this.rooms.values()) {
total += state.activeConnections
}
return total
}
async handleWorkflowDeletion(workflowId: string): Promise<void> {
logger.info(`Handling workflow deletion notification for ${workflowId}`)
const room = this.workflowRooms.get(workflowId)
if (!room) {
logger.debug(`No active room found for deleted workflow ${workflowId}`)
return
}
this._io.to(workflowId).emit('workflow-deleted', {
workflowId,
message: 'This workflow has been deleted',
timestamp: Date.now(),
})
const socketsToDisconnect: string[] = []
room.users.forEach((_presence, socketId) => {
socketsToDisconnect.push(socketId)
})
for (const socketId of socketsToDisconnect) {
const socket = this._io.sockets.sockets.get(socketId)
if (socket) {
socket.leave(workflowId)
logger.debug(`Disconnected socket ${socketId} from deleted workflow ${workflowId}`)
}
await this.removeUserFromRoom(socketId)
}
this.workflowRooms.delete(workflowId)
logger.info(
`Cleaned up workflow room ${workflowId} after deletion (${socketsToDisconnect.length} users disconnected)`
)
}
async handleWorkflowRevert(workflowId: string, timestamp: number): Promise<void> {
logger.info(`Handling workflow revert notification for ${workflowId}`)
const room = this.workflowRooms.get(workflowId)
if (!room) {
logger.debug(`No active room found for reverted workflow ${workflowId}`)
return
}
this._io.to(workflowId).emit('workflow-reverted', {
workflowId,
message: 'Workflow has been reverted to deployed state',
timestamp,
})
room.lastModified = timestamp
logger.info(`Notified ${room.users.size} users about workflow revert: ${workflowId}`)
}
async handleWorkflowUpdate(workflowId: string): Promise<void> {
logger.info(`Handling workflow update notification for ${workflowId}`)
const room = this.workflowRooms.get(workflowId)
if (!room) {
logger.debug(`No active room found for updated workflow ${workflowId}`)
return
}
const timestamp = Date.now()
this._io.to(workflowId).emit('workflow-updated', {
workflowId,
message: 'Workflow has been updated externally',
timestamp,
})
room.lastModified = timestamp
logger.info(`Notified ${room.users.size} users about workflow update: ${workflowId}`)
}
async handleWorkflowDeployed(workflowId: string): Promise<void> {
logger.info(`Handling workflow deployed notification for ${workflowId}`)
const room = this.workflowRooms.get(workflowId)
if (!room) {
logger.debug(`No active room found for deployed workflow ${workflowId}`)
return
}
this._io.to(workflowId).emit('workflow-deployed', {
workflowId,
timestamp: Date.now(),
})
logger.info(`Notified ${room.users.size} users about workflow deployment change: ${workflowId}`)
}
}
@@ -0,0 +1,84 @@
import { type RoomRef, roomName } from '@sim/realtime-protocol/rooms'
import type { Server } from 'socket.io'
import type { IRoomManager, UserPresence } from '@/rooms/types'
/**
* How stale a not-live presence entry must be before a join-time sweep reclaims
* it. The `liveIds` gate (not this threshold) is what protects an active
* collaborator; the threshold only bounds how long a genuinely-orphaned entry
* (e.g. a crashed pod that never fired `disconnecting`) lingers. Matches the
* workflow join sweep.
*/
const STALE_PRESENCE_THRESHOLD_MS = 75 * 60 * 1000
/**
* Filters a room's stored presence down to what should actually be broadcast:
* drops `excludeSocketId` (e.g. a socket mid-disconnect that is still connected),
* then reconciles against the live Socket.IO membership so an entry orphaned by a
* failed removal (the room hashes have no TTL) is never emitted as a ghost.
*
* Fail-safe: if the liveness lookup throws, or returns an empty set while we still
* hold presence entries (a cross-pod `fetchSockets` timeout, not a truly empty
* room), we emit the unfiltered list rather than hide everyone — a transient
* ghost self-corrects on the next broadcast, but hiding live collaborators would
* be a worse, visible failure.
*/
export async function filterVisiblePresence<T extends { socketId: string }>(
io: Server,
room: RoomRef,
users: T[],
excludeSocketId?: string
): Promise<T[]> {
const candidates = excludeSocketId
? users.filter((user) => user.socketId !== excludeSocketId)
: users
try {
const liveSockets = await io.in(roomName(room)).fetchSockets()
if (liveSockets.length === 0) {
return candidates
}
const liveIds = new Set(liveSockets.map((socket) => socket.id))
return candidates.filter((user) => liveIds.has(user.socketId))
} catch {
return candidates
}
}
/**
* Reclaims orphaned presence entries in a room: any stored socket that is no
* longer a live Socket.IO member AND has been idle past
* {@link STALE_PRESENCE_THRESHOLD_MS} is removed. This is how a room-users hash
* (which has no TTL) is bounded against ungraceful disconnects — a pod crash
* fires no `disconnecting` event, so its entries would otherwise persist forever.
* Run on join, like the workflow room does. No-op when the liveness lookup fails
* (so a transient adapter blip can't evict live collaborators).
*/
export async function sweepStalePresence(
manager: IRoomManager,
room: RoomRef
): Promise<UserPresence[]> {
// Read the roster first so it is returned to the caller (for same-tab dedup) even when the
// liveness probe below fails — a fetchSockets outage must skip only the stale-removal, never
// the caller's dedup.
const users = await manager.getRoomUsers(room)
let liveIds: Set<string>
try {
const liveSockets = await manager.io.in(roomName(room)).fetchSockets()
liveIds = new Set(liveSockets.map((socket) => socket.id))
} catch {
return users
}
const now = Date.now()
for (const user of users) {
if (liveIds.has(user.socketId)) continue
const lastSeen = user.lastActivity || user.joinedAt || 0
if (now - lastSeen > STALE_PRESENCE_THRESHOLD_MS) {
await manager.removeUserFromRoom(room, user.socketId)
}
}
// Return the pre-removal roster so a caller can reuse it (e.g. same-tab dedup) instead
// of re-reading; re-removing an already-swept entry downstream is a harmless no-op.
return users
}
+220 -258
View File
@@ -1,140 +1,154 @@
import { createLogger } from '@sim/logger'
import {
presenceEventName,
type RoomRef,
type RoomType,
roomName,
} from '@sim/realtime-protocol/rooms'
import { createClient, type RedisClientType } from 'redis'
import type { Server } from 'socket.io'
import { filterVisiblePresence } from '@/rooms/presence-visibility'
import type { IRoomManager, UserPresence, UserSession } from '@/rooms/types'
const logger = createLogger('RedisRoomManager')
/**
* Redis key scheme (all room-scoped keys are prefixed by room type):
* {type}:{id}:users HASH socketId -> UserPresence JSON (room membership)
* {type}:{id}:meta HASH room metadata (lastModified)
* socket:{sid}:rooms HASH roomType -> roomId (the socket's rooms, one per type)
* socket:{sid}:session HASH userId/userName/avatarUrl (shared across the socket's rooms)
*
* Workflow rooms keep their historical `workflow:{id}:users`/`:meta` keys (the
* type prefix IS `workflow`), so no presence-state migration is needed for them.
*/
const KEYS = {
workflowUsers: (wfId: string) => `workflow:${wfId}:users`,
workflowMeta: (wfId: string) => `workflow:${wfId}:meta`,
socketWorkflow: (socketId: string) => `socket:${socketId}:workflow`,
roomUsers: (room: RoomRef) => `${room.type}:${room.id}:users`,
roomMeta: (room: RoomRef) => `${room.type}:${room.id}:meta`,
socketRooms: (socketId: string) => `socket:${socketId}:rooms`,
socketSession: (socketId: string) => `socket:${socketId}:session`,
socketPresenceWorkflow: (socketId: string) => `socket:${socketId}:presence-workflow`,
} as const
const SOCKET_KEY_TTL = 3600
const SOCKET_PRESENCE_WORKFLOW_KEY_TTL = 24 * 60 * 60
/** TTL for the socket's room-set. Long enough that an idle-but-connected socket is not evicted. */
const SOCKET_ROOMS_TTL = 24 * 60 * 60
/**
* The shared session key MUST share the room-set TTL. `getRoomForSocket` reads the room-set and the
* workflow handlers gate edits/presence on `room && session`, so a session that expired while the
* room-set is still alive would wedge an active-but-idle collaborator into "session expired" until a
* full reload — the activity update only `EXPIRE`s the session, which cannot resurrect an
* already-gone key (only `addUserToRoom` re-`HSET`s it). Both are refreshed together on every
* activity, so they expire together.
*/
const SESSION_TTL = SOCKET_ROOMS_TTL
/**
* Lua script for atomic user removal from room.
* The hint, when provided, is the target room to remove membership from; the
* socket's current mapping is only the fallback. Socket-level keys are cleared
* only when the socket is not mapped to a different room, so removing a stale
* room cannot destroy the mapping of a room the socket has since moved to.
* Returns the target workflowId, or null when no target could be resolved.
* Handles room cleanup atomically to prevent race conditions.
* Atomic single-room removal. Removes a socket from one room's presence, drops
* the room from the socket's room-set, and — critically — deletes the SHARED
* session key only when the socket has left its LAST room (otherwise a leave from
* one room would break the socket's other rooms). Cleans up empty room state.
*
* KEYS: [socketRooms, socketSession, roomUsers, roomMeta]
* ARGV: [roomType, socketId, roomId]
* Returns 1 if the socket was a member of the room, else 0.
*/
const REMOVE_USER_SCRIPT = `
local socketWorkflowKey = KEYS[1]
const REMOVE_ROOM_SCRIPT = `
local socketRoomsKey = KEYS[1]
local socketSessionKey = KEYS[2]
local socketPresenceWorkflowKey = KEYS[3]
local workflowUsersPrefix = ARGV[1]
local workflowMetaPrefix = ARGV[2]
local socketId = ARGV[3]
local workflowIdHint = ARGV[4]
local roomUsersKey = KEYS[3]
local roomMetaKey = KEYS[4]
local roomType = ARGV[1]
local socketId = ARGV[2]
local roomId = ARGV[3]
local currentWorkflowId = redis.call('GET', socketWorkflowKey)
if not currentWorkflowId then
currentWorkflowId = redis.call('GET', socketPresenceWorkflowKey)
local removed = redis.call('HDEL', roomUsersKey, socketId)
-- Only drop the socket's mapping for this type if it points at THIS room, so
-- removing a room the socket isn't in can't wipe a different room's mapping or
-- spuriously trigger the last-room session cleanup (mirrors the memory manager).
if redis.call('HGET', socketRoomsKey, roomType) == roomId then
redis.call('HDEL', socketRoomsKey, roomType)
if redis.call('HLEN', socketRoomsKey) == 0 then
redis.call('DEL', socketRoomsKey, socketSessionKey)
end
end
local workflowId = currentWorkflowId
if workflowIdHint ~= '' then
workflowId = workflowIdHint
if redis.call('HLEN', roomUsersKey) == 0 then
redis.call('DEL', roomUsersKey, roomMetaKey)
end
if not workflowId then
return nil
end
local workflowUsersKey = workflowUsersPrefix .. workflowId .. ':users'
local workflowMetaKey = workflowMetaPrefix .. workflowId .. ':meta'
redis.call('HDEL', workflowUsersKey, socketId)
if (not currentWorkflowId) or currentWorkflowId == workflowId then
redis.call('DEL', socketWorkflowKey, socketSessionKey, socketPresenceWorkflowKey)
end
local remaining = redis.call('HLEN', workflowUsersKey)
if remaining == 0 then
redis.call('DEL', workflowUsersKey, workflowMetaKey)
end
return workflowId
return removed
`
/**
* Lua script for atomic user activity update.
* Performs read-modify-write atomically to prevent lost updates.
* Also refreshes TTL on socket keys to prevent expiry during long sessions.
* Atomic presence-activity update (read-modify-write) that also refreshes the
* socket key TTLs to keep a long-lived session alive.
*
* KEYS: [roomUsers, socketRooms, socketSession]
* ARGV: [socketId, cursorJson, selectionJson, lastActivity, roomsTtl, sessionTtl, cellJson]
* Returns 1 if the socket had presence in the room, else 0.
*/
const UPDATE_ACTIVITY_SCRIPT = `
local workflowUsersKey = KEYS[1]
local socketWorkflowKey = KEYS[2]
local roomUsersKey = KEYS[1]
local socketRoomsKey = KEYS[2]
local socketSessionKey = KEYS[3]
local socketPresenceWorkflowKey = KEYS[4]
local socketId = ARGV[1]
local cursorJson = ARGV[2]
local selectionJson = ARGV[3]
local lastActivity = ARGV[4]
local ttl = tonumber(ARGV[5])
local presenceWorkflowTtl = tonumber(ARGV[6])
local roomsTtl = tonumber(ARGV[5])
local sessionTtl = tonumber(ARGV[6])
local cellJson = ARGV[7]
local existingJson = redis.call('HGET', workflowUsersKey, socketId)
local existingJson = redis.call('HGET', roomUsersKey, socketId)
if not existingJson then
return 0
end
local existing = cjson.decode(existingJson)
if cursorJson ~= '' then
existing.cursor = cjson.decode(cursorJson)
end
if selectionJson ~= '' then
existing.selection = cjson.decode(selectionJson)
end
if cellJson ~= '' then
existing.cell = cjson.decode(cellJson)
end
existing.lastActivity = tonumber(lastActivity)
redis.call('HSET', workflowUsersKey, socketId, cjson.encode(existing))
redis.call('EXPIRE', socketWorkflowKey, ttl)
redis.call('EXPIRE', socketSessionKey, ttl)
redis.call('EXPIRE', socketPresenceWorkflowKey, presenceWorkflowTtl)
redis.call('HSET', roomUsersKey, socketId, cjson.encode(existing))
redis.call('EXPIRE', socketRoomsKey, roomsTtl)
redis.call('EXPIRE', socketSessionKey, sessionTtl)
return 1
`
/**
* Redis-backed room manager for multi-pod deployments.
* Uses Lua scripts for atomic operations to prevent race conditions.
* Redis-backed room manager for multi-pod deployments. Domain-neutral: keyed by
* {@link RoomRef}, supports a socket in multiple rooms (one per {@link RoomType}).
* Uses Lua scripts for atomic multi-key operations.
*/
export class RedisRoomManager implements IRoomManager {
private redis: RedisClientType
private _io: Server
private isConnected = false
private removeUserScriptSha: string | null = null
private removeRoomScriptSha: string | null = null
private updateActivityScriptSha: string | null = null
constructor(io: Server, redisUrl: string) {
this._io = io
this.redis = createClient({
url: redisUrl,
})
this.redis = createClient({ url: redisUrl })
this.redis.on('error', (err) => {
logger.error('Redis client error:', err)
})
this.redis.on('reconnecting', () => {
logger.warn('Redis client reconnecting...')
this.isConnected = false
})
this.redis.on('ready', () => {
logger.info('Redis client ready')
this.isConnected = true
})
this.redis.on('end', () => {
logger.warn('Redis client connection closed')
this.isConnected = false
@@ -146,7 +160,14 @@ export class RedisRoomManager implements IRoomManager {
}
isReady(): boolean {
return this.isConnected
// Gate on the loaded script SHAs, not just the connection: the `ready` event flips
// `isConnected` true as soon as the socket connects — before `initialize()` loads the Lua
// scripts — so a bare `isConnected` check would report ready while `removeUserFromRoom` /
// `updateUserActivity` would silently no-op on a null SHA. Reporting not-ready here makes the
// POST endpoints return a retryable 503 during that startup window instead.
return (
this.isConnected && this.removeRoomScriptSha !== null && this.updateActivityScriptSha !== null
)
}
async initialize(): Promise<void> {
@@ -154,14 +175,18 @@ export class RedisRoomManager implements IRoomManager {
try {
await this.redis.connect()
this.isConnected = true
// Pre-load Lua scripts for better performance
this.removeUserScriptSha = await this.redis.scriptLoad(REMOVE_USER_SCRIPT)
this.removeRoomScriptSha = await this.redis.scriptLoad(REMOVE_ROOM_SCRIPT)
this.updateActivityScriptSha = await this.redis.scriptLoad(UPDATE_ACTIVITY_SCRIPT)
// Mark ready only after the scripts load — isReady() gates removeUserFromRoom/updateUserActivity,
// which silently no-op without a script SHA. Setting the flag before scriptLoad would make
// isReady() lie if scriptLoad threw.
this.isConnected = true
logger.info('RedisRoomManager connected to Redis and scripts loaded')
} catch (error) {
this.isConnected = false
logger.error('Failed to connect to Redis:', error)
throw error
}
@@ -169,7 +194,6 @@ export class RedisRoomManager implements IRoomManager {
async shutdown(): Promise<void> {
if (!this.isConnected) return
try {
await this.redis.quit()
this.isConnected = false
@@ -179,93 +203,102 @@ export class RedisRoomManager implements IRoomManager {
}
}
async addUserToRoom(workflowId: string, socketId: string, presence: UserPresence): Promise<void> {
async addUserToRoom(room: RoomRef, socketId: string, presence: UserPresence): Promise<void> {
try {
const pipeline = this.redis.multi()
pipeline.hSet(KEYS.workflowUsers(workflowId), socketId, JSON.stringify(presence))
pipeline.hSet(KEYS.workflowMeta(workflowId), 'lastModified', Date.now().toString())
pipeline.set(KEYS.socketWorkflow(socketId), workflowId)
pipeline.expire(KEYS.socketWorkflow(socketId), SOCKET_KEY_TTL)
pipeline.set(KEYS.socketPresenceWorkflow(socketId), workflowId)
pipeline.expire(KEYS.socketPresenceWorkflow(socketId), SOCKET_PRESENCE_WORKFLOW_KEY_TTL)
pipeline.hSet(KEYS.roomUsers(room), socketId, JSON.stringify(presence))
pipeline.hSet(KEYS.roomMeta(room), 'lastModified', Date.now().toString())
pipeline.hSet(KEYS.socketRooms(socketId), room.type, room.id)
pipeline.expire(KEYS.socketRooms(socketId), SOCKET_ROOMS_TTL)
pipeline.hSet(KEYS.socketSession(socketId), {
userId: presence.userId,
userName: presence.userName,
avatarUrl: presence.avatarUrl || '',
})
pipeline.expire(KEYS.socketSession(socketId), SOCKET_KEY_TTL)
pipeline.expire(KEYS.socketSession(socketId), SESSION_TTL)
const results = await pipeline.exec()
// Check if any command failed
const failed = results.some((result) => result instanceof Error)
if (failed) {
logger.error(`Pipeline partially failed when adding user to room`, { workflowId, socketId })
logger.error('Pipeline partially failed when adding user to room', {
room,
socketId,
})
throw new Error('Failed to store user session data in Redis')
}
logger.debug(`Added user ${presence.userId} to workflow ${workflowId} (socket: ${socketId})`)
logger.debug(`Added user ${presence.userId} to room ${room.type}:${room.id} (${socketId})`)
} catch (error) {
logger.error(`Failed to add user to room: ${socketId} -> ${workflowId}`, error)
logger.error(`Failed to add user to room: ${socketId} -> ${room.type}:${room.id}`, error)
throw error
}
}
async removeUserFromRoom(
socketId: string,
workflowIdHint?: string,
retried = false
): Promise<string | null> {
if (!this.removeUserScriptSha) {
async removeUserFromRoom(room: RoomRef, socketId: string, retried = false): Promise<boolean> {
if (!this.removeRoomScriptSha) {
logger.error('removeUserFromRoom called before initialize()')
return null
return false
}
try {
const workflowId = await this.redis.evalSha(this.removeUserScriptSha, {
const removed = await this.redis.evalSha(this.removeRoomScriptSha, {
keys: [
KEYS.socketWorkflow(socketId),
KEYS.socketRooms(socketId),
KEYS.socketSession(socketId),
KEYS.socketPresenceWorkflow(socketId),
KEYS.roomUsers(room),
KEYS.roomMeta(room),
],
arguments: ['workflow:', 'workflow:', socketId, workflowIdHint ?? ''],
arguments: [room.type, socketId, room.id],
})
if (typeof workflowId === 'string' && workflowId.length > 0) {
logger.debug(`Removed socket ${socketId} from workflow ${workflowId}`)
return workflowId
}
return null
return typeof removed === 'number' ? removed > 0 : Number(removed) > 0
} catch (error) {
if ((error as Error).message?.includes('NOSCRIPT') && !retried) {
logger.warn('Lua script not found, reloading...')
this.removeUserScriptSha = await this.redis.scriptLoad(REMOVE_USER_SCRIPT)
return this.removeUserFromRoom(socketId, workflowIdHint, true)
this.removeRoomScriptSha = await this.redis.scriptLoad(REMOVE_ROOM_SCRIPT)
return this.removeUserFromRoom(room, socketId, true)
}
logger.error(`Failed to remove user from room: ${socketId}`, error)
return null
logger.error(`Failed to remove socket ${socketId} from room ${room.type}:${room.id}`, error)
return false
}
}
async getWorkflowIdForSocket(socketId: string): Promise<string | null> {
const workflowId = await this.redis.get(KEYS.socketWorkflow(socketId))
if (workflowId) {
return workflowId
async removeSocketFromAllRooms(socketId: string): Promise<RoomRef[]> {
const rooms = await this.getRoomsForSocket(socketId)
if (rooms.length === 0) {
// Nothing tracked (already cleaned up or TTL-expired); ensure session is gone.
await this.redis.del(KEYS.socketSession(socketId)).catch(() => {})
return []
}
return this.redis.get(KEYS.socketPresenceWorkflow(socketId))
const removed: RoomRef[] = []
for (const room of rooms) {
const wasMember = await this.removeUserFromRoom(room, socketId)
if (wasMember) removed.push(room)
}
return removed
}
async getRoomsForSocket(socketId: string): Promise<RoomRef[]> {
try {
const entries = await this.redis.hGetAll(KEYS.socketRooms(socketId))
return Object.entries(entries).map(([type, id]) => ({ type: type as RoomType, id }))
} catch (error) {
logger.error(`Failed to get rooms for socket ${socketId}:`, error)
return []
}
}
async getRoomForSocket(socketId: string, type: RoomType): Promise<RoomRef | null> {
const id = await this.redis.hGet(KEYS.socketRooms(socketId), type)
return id ? { type, id } : null
}
async getUserSession(socketId: string): Promise<UserSession | null> {
try {
const session = await this.redis.hGetAll(KEYS.socketSession(socketId))
if (!session.userId) {
return null
}
if (!session.userId) return null
return {
userId: session.userId,
userName: session.userName,
@@ -277,34 +310,54 @@ export class RedisRoomManager implements IRoomManager {
}
}
async getWorkflowUsers(workflowId: string): Promise<UserPresence[]> {
/**
* Reads and parses the room roster. Throws on a transport error (so a caller can
* distinguish "genuinely empty" from "read failed"); a single corrupted entry is
* skipped, not fatal.
*/
private async readRoomUsers(room: RoomRef): Promise<UserPresence[]> {
const users = await this.redis.hGetAll(KEYS.roomUsers(room))
return Object.entries(users)
.map(([socketId, json]) => {
try {
return JSON.parse(json) as UserPresence
} catch {
logger.warn(`Corrupted user data for socket ${socketId}, skipping`)
return null
}
})
.filter((u): u is UserPresence => u !== null)
}
async getRoomUsers(room: RoomRef): Promise<UserPresence[]> {
try {
const users = await this.redis.hGetAll(KEYS.workflowUsers(workflowId))
return Object.entries(users)
.map(([socketId, json]) => {
try {
return JSON.parse(json) as UserPresence
} catch {
logger.warn(`Corrupted user data for socket ${socketId}, skipping`)
return null
}
})
.filter((u): u is UserPresence => u !== null)
return await this.readRoomUsers(room)
} catch (error) {
logger.error(`Failed to get workflow users for ${workflowId}:`, error)
logger.error(`Failed to get room users for ${room.type}:${room.id}:`, error)
return []
}
}
async hasWorkflowRoom(workflowId: string): Promise<boolean> {
const exists = await this.redis.exists(KEYS.workflowUsers(workflowId))
async hasRoom(room: RoomRef): Promise<boolean> {
const exists = await this.redis.exists(KEYS.roomUsers(room))
return exists > 0
}
async deleteRoom(room: RoomRef): Promise<void> {
// Log AND rethrow (like addUserToRoom): a failed wipe must not be reported as a
// clean deletion by the caller — the request surfaces it (and can be retried).
try {
await this.redis.del([KEYS.roomUsers(room), KEYS.roomMeta(room)])
} catch (error) {
logger.error(`Failed to delete room ${room.type}:${room.id}:`, error)
throw error
}
}
async updateUserActivity(
workflowId: string,
room: RoomRef,
socketId: string,
updates: Partial<Pick<UserPresence, 'cursor' | 'selection' | 'lastActivity'>>,
updates: Partial<Pick<UserPresence, 'cursor' | 'selection' | 'cell' | 'lastActivity'>>,
retried = false
): Promise<void> {
if (!this.updateActivityScriptSha) {
@@ -314,155 +367,64 @@ export class RedisRoomManager implements IRoomManager {
try {
await this.redis.evalSha(this.updateActivityScriptSha, {
keys: [
KEYS.workflowUsers(workflowId),
KEYS.socketWorkflow(socketId),
KEYS.socketSession(socketId),
KEYS.socketPresenceWorkflow(socketId),
],
keys: [KEYS.roomUsers(room), KEYS.socketRooms(socketId), KEYS.socketSession(socketId)],
arguments: [
socketId,
updates.cursor !== undefined ? JSON.stringify(updates.cursor) : '',
updates.selection !== undefined ? JSON.stringify(updates.selection) : '',
(updates.lastActivity ?? Date.now()).toString(),
SOCKET_KEY_TTL.toString(),
SOCKET_PRESENCE_WORKFLOW_KEY_TTL.toString(),
SOCKET_ROOMS_TTL.toString(),
SESSION_TTL.toString(),
// Trailing arg (ARGV[7]) so existing indices stay stable. `null` (cleared
// selection) serializes to 'null'; `undefined` (no cell change) to '' (skip).
updates.cell !== undefined ? JSON.stringify(updates.cell) : '',
],
})
} catch (error) {
if ((error as Error).message?.includes('NOSCRIPT') && !retried) {
logger.warn('Lua script not found, reloading...')
this.updateActivityScriptSha = await this.redis.scriptLoad(UPDATE_ACTIVITY_SCRIPT)
return this.updateUserActivity(workflowId, socketId, updates, true)
return this.updateUserActivity(room, socketId, updates, true)
}
logger.error(`Failed to update user activity: ${socketId}`, error)
}
}
async updateRoomLastModified(workflowId: string): Promise<void> {
await this.redis.hSet(KEYS.workflowMeta(workflowId), 'lastModified', Date.now().toString())
async updateRoomLastModified(room: RoomRef): Promise<void> {
await this.redis.hSet(KEYS.roomMeta(room), 'lastModified', Date.now().toString())
}
async broadcastPresenceUpdate(workflowId: string): Promise<void> {
const users = await this.getWorkflowUsers(workflowId)
// io.to() with Redis adapter broadcasts to all pods
this._io.to(workflowId).emit('presence-update', users)
async broadcastPresenceUpdate(room: RoomRef, excludeSocketId?: string): Promise<void> {
let users: UserPresence[]
try {
// Read via the throwing variant, NOT getRoomUsers: a transport error there returns `[]`, which
// would broadcast an empty roster and clear every remaining collaborator's presence until the
// next healthy update. Skip instead — the next successful join/activity broadcast (or the
// stale sweep) reconciles peers.
users = await this.readRoomUsers(room)
} catch (error) {
logger.error(
`Skipping presence broadcast for ${room.type}:${room.id} (roster read failed):`,
error
)
return
}
const visible = await filterVisiblePresence(this._io, room, users, excludeSocketId)
// io.to() with the Redis adapter broadcasts to all pods.
this._io.to(roomName(room)).emit(presenceEventName(room.type), visible)
}
emitToWorkflow<T = unknown>(workflowId: string, event: string, payload: T): void {
this._io.to(workflowId).emit(event, payload)
emitToRoom<T = unknown>(room: RoomRef, event: string, payload: T): void {
this._io.to(roomName(room)).emit(event, payload)
}
async getUniqueUserCount(workflowId: string): Promise<number> {
const users = await this.getWorkflowUsers(workflowId)
const uniqueUserIds = new Set(users.map((u) => u.userId))
return uniqueUserIds.size
async getUniqueUserCount(room: RoomRef): Promise<number> {
const users = await this.getRoomUsers(room)
return new Set(users.map((u) => u.userId)).size
}
async getTotalActiveConnections(): Promise<number> {
// This is more complex with Redis - we'd need to scan all workflow:*:users keys
// For now, just count sockets in this server instance
// The true count would require aggregating across all pods
// Local instance only; the true cross-pod count would require aggregation.
return this._io.sockets.sockets.size
}
async handleWorkflowDeletion(workflowId: string): Promise<void> {
logger.info(`Handling workflow deletion notification for ${workflowId}`)
try {
const users = await this.getWorkflowUsers(workflowId)
if (users.length === 0) {
logger.debug(`No active users found for deleted workflow ${workflowId}`)
return
}
// Notify all clients across all pods via Redis adapter
this._io.to(workflowId).emit('workflow-deleted', {
workflowId,
message: 'This workflow has been deleted',
timestamp: Date.now(),
})
// Use Socket.IO's cross-pod socketsLeave() to remove all sockets from the room
// This works across all pods when using the Redis adapter
await this._io.in(workflowId).socketsLeave(workflowId)
logger.debug(`All sockets left workflow room ${workflowId} via socketsLeave()`)
// Remove all users from Redis state
for (const user of users) {
await this.removeUserFromRoom(user.socketId, workflowId)
}
// Clean up room data
await this.redis.del([KEYS.workflowUsers(workflowId), KEYS.workflowMeta(workflowId)])
logger.info(
`Cleaned up workflow room ${workflowId} after deletion (${users.length} users disconnected)`
)
} catch (error) {
logger.error(`Failed to handle workflow deletion for ${workflowId}:`, error)
}
}
async handleWorkflowRevert(workflowId: string, timestamp: number): Promise<void> {
logger.info(`Handling workflow revert notification for ${workflowId}`)
const hasRoom = await this.hasWorkflowRoom(workflowId)
if (!hasRoom) {
logger.debug(`No active room found for reverted workflow ${workflowId}`)
return
}
this._io.to(workflowId).emit('workflow-reverted', {
workflowId,
message: 'Workflow has been reverted to deployed state',
timestamp,
})
await this.updateRoomLastModified(workflowId)
const userCount = await this.getUniqueUserCount(workflowId)
logger.info(`Notified ${userCount} users about workflow revert: ${workflowId}`)
}
async handleWorkflowUpdate(workflowId: string): Promise<void> {
logger.info(`Handling workflow update notification for ${workflowId}`)
const hasRoom = await this.hasWorkflowRoom(workflowId)
if (!hasRoom) {
logger.debug(`No active room found for updated workflow ${workflowId}`)
return
}
const timestamp = Date.now()
this._io.to(workflowId).emit('workflow-updated', {
workflowId,
message: 'Workflow has been updated externally',
timestamp,
})
await this.updateRoomLastModified(workflowId)
const userCount = await this.getUniqueUserCount(workflowId)
logger.info(`Notified ${userCount} users about workflow update: ${workflowId}`)
}
async handleWorkflowDeployed(workflowId: string): Promise<void> {
logger.info(`Handling workflow deployed notification for ${workflowId}`)
const hasRoom = await this.hasWorkflowRoom(workflowId)
if (!hasRoom) {
logger.debug(`No active room found for deployed workflow ${workflowId}`)
return
}
this._io.to(workflowId).emit('workflow-deployed', {
workflowId,
timestamp: Date.now(),
})
const userCount = await this.getUniqueUserCount(workflowId)
logger.info(`Notified ${userCount} users about workflow deployment change: ${workflowId}`)
}
}
+72 -83
View File
@@ -1,11 +1,16 @@
import type { RoomRef, RoomType } from '@sim/realtime-protocol/rooms'
import type { TableCellSelection } from '@sim/realtime-protocol/table-presence'
import type { Server } from 'socket.io'
/**
* User presence data stored in room state
* User presence data stored in room state.
*
* `room` is the generic room address (see `@sim/realtime-protocol/rooms`). A
* socket may hold presence in more than one room, but only one room per type.
*/
export interface UserPresence {
userId: string
workflowId: string
room: RoomRef
userName: string
socketId: string
tabSessionId?: string
@@ -14,11 +19,19 @@ export interface UserPresence {
role: string
cursor?: { x: number; y: number }
selection?: { type: 'block' | 'edge' | 'none'; id?: string }
/** The viewer's current table cell selection, for table presence rooms. */
cell?: TableCellSelection
avatarUrl?: string | null
/**
* The subfolder the user is viewing, recorded at join for room types that track
* a per-viewer location (e.g. the workspace file browser). `null` is the root.
*/
folderId?: string | null
}
/**
* User session data (minimal info for quick lookups)
* User session data (minimal info for quick lookups). Shared across all rooms a
* socket is in — keyed by socket, not by room.
*/
export interface UserSession {
userId: string
@@ -27,125 +40,101 @@ export interface UserSession {
}
/**
* Workflow room state
* Room presence state.
*/
export interface WorkflowRoom {
workflowId: string
export interface RoomState {
room: RoomRef
users: Map<string, UserPresence>
lastModified: number
activeConnections: number
}
/**
* Common interface for room managers (in-memory and Redis)
* All methods that access state are async to support Redis operations
* Common interface for room managers (in-memory and Redis).
*
* The manager is domain-neutral: it tracks room membership and presence keyed by
* {@link RoomRef}, and knows nothing about workflows, files, or any specific
* domain. Domain lifecycle concerns (e.g. workflow deletion/deploy broadcasts)
* live in domain services that compose a manager — see `WorkflowRoomService`.
*
* A socket may occupy multiple rooms, at most one per {@link RoomType}. The
* shared session key is dropped only when a socket leaves its last room.
*
* All state-accessing methods are async to support the Redis implementation.
*/
export interface IRoomManager {
readonly io: Server
/**
* Initialize the room manager (connect to Redis, etc.)
*/
/** Initialize the manager (connect to Redis, load scripts, etc.). */
initialize(): Promise<void>
/**
* Whether the room manager is ready to serve requests
*/
/** Whether the manager is ready to serve requests. */
isReady(): boolean
/**
* Clean shutdown
*/
/** Clean shutdown. */
shutdown(): Promise<void>
/**
* Add a user to a workflow room
*/
addUserToRoom(workflowId: string, socketId: string, presence: UserPresence): Promise<void>
/** Add a socket's presence to a room. */
addUserToRoom(room: RoomRef, socketId: string, presence: UserPresence): Promise<void>
/**
* Remove a user's membership of a workflow room.
* When workflowIdHint is provided it is the target room; the socket's current
* mapping is only the fallback (and covers missing/expired mapping keys).
* Socket-level mappings are cleared only when the socket is not mapped to a
* different room, so removing a stale room cannot destroy the mapping of a
* room the socket has since moved to.
* Returns the target workflowId, or null when no target could be resolved
* (or, for the Redis manager, when the removal failed).
* Remove a socket from a single room. Returns `true` if it was a member. The
* shared session is dropped only if this was the socket's last room.
*/
removeUserFromRoom(socketId: string, workflowIdHint?: string): Promise<string | null>
removeUserFromRoom(room: RoomRef, socketId: string): Promise<boolean>
/**
* Get the workflow ID for a socket
* Remove a socket from every room it occupies (disconnect). Returns the rooms
* it was in, so the caller can rebroadcast presence per room.
*/
getWorkflowIdForSocket(socketId: string): Promise<string | null>
removeSocketFromAllRooms(socketId: string): Promise<RoomRef[]>
/**
* Get user session data for a socket
*/
/** Every room the socket currently occupies. */
getRoomsForSocket(socketId: string): Promise<RoomRef[]>
/** The socket's room of a given type (at most one per type), or `null`. */
getRoomForSocket(socketId: string, type: RoomType): Promise<RoomRef | null>
/** Session data for a socket (shared across its rooms). */
getUserSession(socketId: string): Promise<UserSession | null>
/**
* Get all users in a workflow room
*/
getWorkflowUsers(workflowId: string): Promise<UserPresence[]>
/** All users present in a room. */
getRoomUsers(room: RoomRef): Promise<UserPresence[]>
/** Whether a room currently has any presence. */
hasRoom(room: RoomRef): Promise<boolean>
/**
* Check if a workflow room exists
* Unconditionally drop all state for a room (presence + metadata). Used when a
* room's underlying resource is destroyed (e.g. a deleted workflow) to guarantee
* no state lingers even if per-socket removals failed or a socket joined mid-teardown.
*/
hasWorkflowRoom(workflowId: string): Promise<boolean>
deleteRoom(room: RoomRef): Promise<void>
/**
* Update user activity (cursor, selection, lastActivity)
*/
/** Update a socket's activity (cursor, selection, cell, lastActivity) within a room. */
updateUserActivity(
workflowId: string,
room: RoomRef,
socketId: string,
updates: Partial<Pick<UserPresence, 'cursor' | 'selection' | 'lastActivity'>>
updates: Partial<Pick<UserPresence, 'cursor' | 'selection' | 'cell' | 'lastActivity'>>
): Promise<void>
/**
* Update room's lastModified timestamp
*/
updateRoomLastModified(workflowId: string): Promise<void>
/** Bump a room's lastModified timestamp. */
updateRoomLastModified(room: RoomRef): Promise<void>
/**
* Broadcast presence update to all clients in a workflow room
* Broadcast the room's presence list to all clients in the room. Pass
* `excludeSocketId` (e.g. a disconnecting socket) to omit that socket from the
* broadcast even if its presence entry outlived a failed removal — so it is
* never shown as a ghost collaborator.
*/
broadcastPresenceUpdate(workflowId: string): Promise<void>
broadcastPresenceUpdate(room: RoomRef, excludeSocketId?: string): Promise<void>
/**
* Emit an event to all clients in a workflow room
*/
emitToWorkflow<T = unknown>(workflowId: string, event: string, payload: T): void
/** Emit an event to all clients in a room. */
emitToRoom<T = unknown>(room: RoomRef, event: string, payload: T): void
/**
* Get the number of unique users in a workflow room
*/
getUniqueUserCount(workflowId: string): Promise<number>
/** Number of unique users in a room. */
getUniqueUserCount(room: RoomRef): Promise<number>
/**
* Get total active connections across all rooms
*/
/** Total active connections tracked by this instance. */
getTotalActiveConnections(): Promise<number>
/**
* Handle workflow deletion - notify users and clean up room
*/
handleWorkflowDeletion(workflowId: string): Promise<void>
/**
* Handle workflow revert - notify users
*/
handleWorkflowRevert(workflowId: string, timestamp: number): Promise<void>
/**
* Handle workflow update - notify users
*/
handleWorkflowUpdate(workflowId: string): Promise<void>
/**
* Handle workflow deployment change - notify users to refresh deployment state
*/
handleWorkflowDeployed(workflowId: string): Promise<void>
}
@@ -0,0 +1,126 @@
import { createLogger } from '@sim/logger'
import { ROOM_TYPES, type RoomRef, roomName } from '@sim/realtime-protocol/rooms'
import type { IRoomManager } from '@/rooms/types'
const logger = createLogger('WorkflowRoomService')
/** The workflow room ref for a workflow id. Its Socket.IO room name is the bare id. */
export function workflowRoom(workflowId: string): RoomRef {
return { type: ROOM_TYPES.WORKFLOW, id: workflowId }
}
/**
* Workflow-domain lifecycle broadcasts, composed over a domain-neutral
* {@link IRoomManager}. Keeps workflow-specific concerns (deletion, revert,
* update, deploy notifications) out of the generic manager, mirroring how the
* workflow socket handlers own workflow semantics.
*/
export class WorkflowRoomService {
constructor(private readonly manager: IRoomManager) {}
async handleWorkflowDeletion(workflowId: string): Promise<void> {
logger.info(`Handling workflow deletion notification for ${workflowId}`)
const room = workflowRoom(workflowId)
const name = roomName(room)
// Always notify — reach every socket still in the Socket.IO room so the client
// clears the deleted workflow, even if that socket's Redis presence was evicted
// (in which case it would be missing from getRoomUsers). Emitting to an empty
// room is a harmless no-op.
this.manager.emitToRoom(room, 'workflow-deleted', {
workflowId,
message: 'This workflow has been deleted',
timestamp: Date.now(),
})
// Clean per-socket state for every socket that is either a live Socket.IO member
// OR still has presence — so an evicted/late-joined socket's room mapping and
// session are dropped too, not just the presence-tracked ones.
const socketIds = new Set<string>()
try {
const liveSockets = await this.manager.io.in(name).fetchSockets()
for (const s of liveSockets) socketIds.add(s.id)
} catch (error) {
logger.warn(`Could not enumerate sockets for deleted workflow ${workflowId}`, error)
}
for (const user of await this.manager.getRoomUsers(room)) socketIds.add(user.socketId)
// Remove every socket from the Socket.IO room (cross-pod via the Redis adapter).
await this.manager.io.in(name).socketsLeave(name)
// Independent per-socket removals — run concurrently.
await Promise.all(
Array.from(socketIds, (socketId) => this.manager.removeUserFromRoom(room, socketId))
)
// Final unconditional wipe — the workflow is gone, so no room state may linger
// even if a per-socket removal failed (matches the pre-refactor managers, which
// ended deletion with an unconditional room drop).
await this.manager.deleteRoom(room)
logger.info(
`Cleaned up workflow room ${workflowId} after deletion (${socketIds.size} sockets removed)`
)
}
async handleWorkflowRevert(workflowId: string, timestamp: number): Promise<void> {
logger.info(`Handling workflow revert notification for ${workflowId}`)
const room = workflowRoom(workflowId)
if (!(await this.manager.hasRoom(room))) {
logger.debug(`No active room found for reverted workflow ${workflowId}`)
return
}
this.manager.emitToRoom(room, 'workflow-reverted', {
workflowId,
message: 'Workflow has been reverted to deployed state',
timestamp,
})
await this.manager.updateRoomLastModified(room)
const userCount = await this.manager.getUniqueUserCount(room)
logger.info(`Notified ${userCount} users about workflow revert: ${workflowId}`)
}
async handleWorkflowUpdate(workflowId: string): Promise<void> {
logger.info(`Handling workflow update notification for ${workflowId}`)
const room = workflowRoom(workflowId)
if (!(await this.manager.hasRoom(room))) {
logger.debug(`No active room found for updated workflow ${workflowId}`)
return
}
this.manager.emitToRoom(room, 'workflow-updated', {
workflowId,
message: 'Workflow has been updated externally',
timestamp: Date.now(),
})
await this.manager.updateRoomLastModified(room)
const userCount = await this.manager.getUniqueUserCount(room)
logger.info(`Notified ${userCount} users about workflow update: ${workflowId}`)
}
async handleWorkflowDeployed(workflowId: string): Promise<void> {
logger.info(`Handling workflow deployed notification for ${workflowId}`)
const room = workflowRoom(workflowId)
if (!(await this.manager.hasRoom(room))) {
logger.debug(`No active room found for deployed workflow ${workflowId}`)
return
}
this.manager.emitToRoom(room, 'workflow-deployed', {
workflowId,
timestamp: Date.now(),
})
const userCount = await this.manager.getUniqueUserCount(room)
logger.info(`Notified ${userCount} users about workflow deployment change: ${workflowId}`)
}
}
+84 -5
View File
@@ -1,7 +1,9 @@
import type { IncomingMessage, ServerResponse } from 'http'
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { safeCompare } from '@sim/security/compare'
import { env } from '@/env'
import type { IRoomManager } from '@/rooms'
import { applyMarkdownToLiveFileDoc } from '@/handlers/file-doc'
import { type IRoomManager, WorkflowRoomService } from '@/rooms'
interface Logger {
info: (message: string, ...args: unknown[]) => void
@@ -41,6 +43,10 @@ function readRequestBody(req: IncomingMessage): Promise<string> {
})
}
function isNonEmptyString(value: unknown): value is string {
return typeof value === 'string' && value.length > 0
}
function sendSuccess(res: ServerResponse): void {
res.writeHead(200, { 'Content-Type': 'application/json' })
res.end(JSON.stringify({ success: true }))
@@ -58,6 +64,8 @@ function sendError(res: ServerResponse, message: string, status = 500): void {
* @returns HTTP request handler function
*/
export function createHttpHandler(roomManager: IRoomManager, logger: Logger) {
const workflowRoomService = new WorkflowRoomService(roomManager)
return async (req: IncomingMessage, res: ServerResponse) => {
res.setHeader('X-Robots-Tag', 'noindex, nofollow')
@@ -101,7 +109,8 @@ export function createHttpHandler(roomManager: IRoomManager, logger: Logger) {
try {
const body = await readRequestBody(req)
const { workflowId } = JSON.parse(body)
await roomManager.handleWorkflowDeletion(workflowId)
if (!isNonEmptyString(workflowId)) return sendError(res, 'Invalid workflowId', 400)
await workflowRoomService.handleWorkflowDeletion(workflowId)
sendSuccess(res)
} catch (error) {
logger.error('Error handling workflow deletion notification:', error)
@@ -115,7 +124,8 @@ export function createHttpHandler(roomManager: IRoomManager, logger: Logger) {
try {
const body = await readRequestBody(req)
const { workflowId } = JSON.parse(body)
await roomManager.handleWorkflowUpdate(workflowId)
if (!isNonEmptyString(workflowId)) return sendError(res, 'Invalid workflowId', 400)
await workflowRoomService.handleWorkflowUpdate(workflowId)
sendSuccess(res)
} catch (error) {
logger.error('Error handling workflow update notification:', error)
@@ -129,7 +139,8 @@ export function createHttpHandler(roomManager: IRoomManager, logger: Logger) {
try {
const body = await readRequestBody(req)
const { workflowId } = JSON.parse(body)
await roomManager.handleWorkflowDeployed(workflowId)
if (!isNonEmptyString(workflowId)) return sendError(res, 'Invalid workflowId', 400)
await workflowRoomService.handleWorkflowDeployed(workflowId)
sendSuccess(res)
} catch (error) {
logger.error('Error handling workflow deployed notification:', error)
@@ -143,7 +154,8 @@ export function createHttpHandler(roomManager: IRoomManager, logger: Logger) {
try {
const body = await readRequestBody(req)
const { workflowId, timestamp } = JSON.parse(body)
await roomManager.handleWorkflowRevert(workflowId, timestamp)
if (!isNonEmptyString(workflowId)) return sendError(res, 'Invalid workflowId', 400)
await workflowRoomService.handleWorkflowRevert(workflowId, timestamp)
sendSuccess(res)
} catch (error) {
logger.error('Error handling workflow revert notification:', error)
@@ -152,6 +164,73 @@ export function createHttpHandler(roomManager: IRoomManager, logger: Logger) {
return
}
// Fan out a file-tree change to everyone viewing a workspace's files, so their
// browser refetches. File mutations happen over the HTTP API (not the socket);
// this is the lossy liveness signal — a missed one only means stale-until-refetch.
if (req.method === 'POST' && req.url === '/api/workspace-files-changed') {
try {
const body = await readRequestBody(req)
const { workspaceId } = JSON.parse(body)
if (!isNonEmptyString(workspaceId)) return sendError(res, 'Invalid workspaceId', 400)
roomManager.emitToRoom(
{ type: ROOM_TYPES.WORKSPACE_FILES, id: workspaceId },
'workspace-files-changed',
{ workspaceId, timestamp: Date.now() }
)
sendSuccess(res)
} catch (error) {
logger.error('Error handling workspace files changed notification:', error)
sendError(res, 'Failed to process files change notification')
}
return
}
// Fan out a table-list change to everyone viewing a workspace's tables, so their browser
// refetches. The list-level counterpart to workspace-files-changed; same lossy-signal contract.
if (req.method === 'POST' && req.url === '/api/workspace-tables-changed') {
try {
const body = await readRequestBody(req)
const { workspaceId } = JSON.parse(body)
if (!isNonEmptyString(workspaceId)) return sendError(res, 'Invalid workspaceId', 400)
roomManager.emitToRoom(
{ type: ROOM_TYPES.WORKSPACE_TABLES, id: workspaceId },
'workspace-tables-changed',
{ workspaceId, timestamp: Date.now() }
)
sendSuccess(res)
} catch (error) {
logger.error('Error handling workspace tables changed notification:', error)
sendError(res, 'Failed to process tables change notification')
}
return
}
// Merge a durable file write into a file's LIVE collaborative document so open editors reconcile to
// it (Stage C) — this is the stream-end/durable reconcile, not token-by-token streaming (that is now
// applied client-side by the open editor). Returns `{ applied }`: when false, no seeded live room
// exists and the caller writes the file directly instead. Live user edits are preserved — the app
// builds a minimal CRDT diff.
if (req.method === 'POST' && req.url === '/api/file-doc/apply-edit') {
try {
const body = await readRequestBody(req)
const { fileId, markdown, version } = JSON.parse(body)
if (!isNonEmptyString(fileId) || typeof markdown !== 'string') {
return sendError(res, 'Invalid fileId or markdown', 400)
}
// `version` (the durable updatedAt this markdown was written with) records that the live doc now
// incorporates that durable version, so the persist If-Match guard won't flag it as a conflict.
const result = await applyMarkdownToLiveFileDoc(fileId, markdown, {
version: typeof version === 'number' ? version : undefined,
})
res.writeHead(200, { 'Content-Type': 'application/json' })
res.end(JSON.stringify({ applied: result === 'applied' }))
} catch (error) {
logger.error('Error applying copilot edit to live file-doc:', error)
sendError(res, 'Failed to apply edit to live document')
}
return
}
res.writeHead(404, { 'Content-Type': 'application/json' })
res.end(JSON.stringify({ error: 'Not found' }))
}
@@ -0,0 +1,37 @@
import { createLogger } from '@sim/logger'
import { getErrorMessage } from '@sim/utils/errors'
import type { NextRequest } from 'next/server'
import { NextResponse } from 'next/server'
import { mergeFileDocContract } from '@/lib/api/contracts/file-doc'
import { parseRequest } from '@/lib/api/server'
import { buildFileDocMergeUpdate } from '@/lib/collab-doc/merge'
import { checkInternalApiKey, createUnauthorizedResponse } from '@/lib/copilot/request/http'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
const logger = createLogger('FileDocMergeAPI')
/**
* POST /api/internal/file-doc/merge — merge new markdown into a live collaborative document as a
* minimal Yjs diff (Stage C — copilot writing into an open doc). The realtime relay ships the current
* doc state; the app returns the diff to apply + relay. Internal only: gated on the shared
* `x-api-key: INTERNAL_API_SECRET` secret, matching the seed endpoint and the realtime relay.
*/
export const POST = withRouteHandler(async (request: NextRequest) => {
const auth = checkInternalApiKey(request)
if (!auth.success) return createUnauthorizedResponse()
const parsed = await parseRequest(mergeFileDocContract, request, {})
if (!parsed.success) return parsed.response
const { fileId, docState, markdown } = parsed.data.body
try {
const update = buildFileDocMergeUpdate(Buffer.from(docState, 'base64'), markdown)
return NextResponse.json({ update: Buffer.from(update).toString('base64') })
} catch (error) {
logger.error('Failed to merge markdown into file-doc', { fileId, error })
return NextResponse.json(
{ error: getErrorMessage(error, 'Failed to merge document') },
{ status: 500 }
)
}
})
@@ -0,0 +1,44 @@
import { createLogger } from '@sim/logger'
import { getErrorMessage } from '@sim/utils/errors'
import type { NextRequest } from 'next/server'
import { NextResponse } from 'next/server'
import { persistFileDocContract } from '@/lib/api/contracts/file-doc'
import { parseRequest } from '@/lib/api/server'
import { persistFileDoc } from '@/lib/collab-doc/persist'
import { checkInternalApiKey, createUnauthorizedResponse } from '@/lib/copilot/request/http'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
const logger = createLogger('FileDocPersistAPI')
/**
* POST /api/internal/file-doc/persist — project a live collaborative document back to durable markdown
* (Yjs → markdown, through the exact editor engine) and write it to the file. Internal only: gated on
* the shared `x-api-key: INTERNAL_API_SECRET` secret, matching the header the realtime relay sends.
* The relay owns the live doc but not the conversion engine or blob/DB access, so it ships the current
* doc state here — the server-authoritative durable path that replaces the editor's client autosave.
*/
export const POST = withRouteHandler(async (request: NextRequest) => {
const auth = checkInternalApiKey(request)
if (!auth.success) return createUnauthorizedResponse()
const parsed = await parseRequest(persistFileDocContract, request, {})
if (!parsed.success) return parsed.response
const { workspaceId, fileId, userId, docState, expectedVersion } = parsed.data.body
try {
const result = await persistFileDoc(
workspaceId,
fileId,
userId,
new Uint8Array(Buffer.from(docState, 'base64')),
expectedVersion
)
return NextResponse.json(result)
} catch (error) {
logger.error('Failed to persist file-doc', { workspaceId, fileId, error })
return NextResponse.json(
{ error: getErrorMessage(error, 'Failed to persist document') },
{ status: 500 }
)
}
})
@@ -0,0 +1,70 @@
/**
* @vitest-environment node
*/
import { createMockRequest } from '@sim/testing'
import { NextResponse } from 'next/server'
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockCheckInternalApiKey, mockBuildFileDocSeed } = vi.hoisted(() => ({
mockCheckInternalApiKey: vi.fn(),
mockBuildFileDocSeed: vi.fn(),
}))
vi.mock('@/lib/copilot/request/http', () => ({
checkInternalApiKey: mockCheckInternalApiKey,
createUnauthorizedResponse: () => NextResponse.json({ error: 'Unauthorized' }, { status: 401 }),
}))
vi.mock('@/lib/collab-doc/seed', () => ({
buildFileDocSeed: mockBuildFileDocSeed,
}))
import { POST } from './route'
function seedRequest(body: unknown) {
return createMockRequest('POST', body, { 'x-api-key': 'internal' })
}
describe('POST /api/internal/file-doc/seed', () => {
beforeEach(() => {
vi.clearAllMocks()
mockCheckInternalApiKey.mockReturnValue({ success: true })
})
// Regression guard for the auth-helper choice: the realtime relay authenticates with
// `x-api-key: INTERNAL_API_SECRET`, so this route MUST gate on `checkInternalApiKey`. Wiring the
// Bearer-JWT-only `checkInternalAuth` (which forbids `x-api-key`) 401s every real seed fetch.
it('401s when the internal api key is rejected, without building a seed', async () => {
mockCheckInternalApiKey.mockReturnValue({ success: false })
const res = await POST(seedRequest({ workspaceId: 'ws-1', fileId: 'file-1' }))
expect(res.status).toBe(401)
expect(mockBuildFileDocSeed).not.toHaveBeenCalled()
})
it('returns the seed as base64 for an authorized request', async () => {
mockBuildFileDocSeed.mockResolvedValue({ update: new Uint8Array([1, 2, 3, 4]) })
const res = await POST(seedRequest({ workspaceId: 'ws-1', fileId: 'file-1' }))
expect(res.status).toBe(200)
expect((await res.json()).update).toBe(Buffer.from([1, 2, 3, 4]).toString('base64'))
expect(mockBuildFileDocSeed).toHaveBeenCalledWith('ws-1', 'file-1')
})
it('returns update:null for a genuinely absent file', async () => {
mockBuildFileDocSeed.mockResolvedValue(null)
const res = await POST(seedRequest({ workspaceId: 'ws-1', fileId: 'missing' }))
expect(res.status).toBe(200)
expect((await res.json()).update).toBeNull()
})
it('400s on a body missing required fields (contract validation, after auth)', async () => {
const res = await POST(seedRequest({ workspaceId: 'ws-1' }))
expect(res.status).toBe(400)
expect(mockBuildFileDocSeed).not.toHaveBeenCalled()
})
it('500s when the seed build throws (a read error the relay should retry)', async () => {
mockBuildFileDocSeed.mockRejectedValue(new Error('db down'))
const res = await POST(seedRequest({ workspaceId: 'ws-1', fileId: 'file-1' }))
expect(res.status).toBe(500)
})
})
@@ -0,0 +1,40 @@
import { createLogger } from '@sim/logger'
import { getErrorMessage } from '@sim/utils/errors'
import type { NextRequest } from 'next/server'
import { NextResponse } from 'next/server'
import { buildFileDocSeedContract } from '@/lib/api/contracts/file-doc'
import { parseRequest } from '@/lib/api/server'
import { buildFileDocSeed } from '@/lib/collab-doc/seed'
import { checkInternalApiKey, createUnauthorizedResponse } from '@/lib/copilot/request/http'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
const logger = createLogger('FileDocSeedAPI')
/**
* POST /api/internal/file-doc/seed — build a server-authoritative collaborative-document seed
* (markdown → Yjs) for the realtime relay to apply on room creation. Internal only: gated on the
* shared `x-api-key: INTERNAL_API_SECRET` secret, matching the header the realtime relay sends
* (`apps/realtime/src/handlers/file-doc-app.ts`) and the realtime server's own inbound validator.
*/
export const POST = withRouteHandler(async (request: NextRequest) => {
const auth = checkInternalApiKey(request)
if (!auth.success) return createUnauthorizedResponse()
const parsed = await parseRequest(buildFileDocSeedContract, request, {})
if (!parsed.success) return parsed.response
const { workspaceId, fileId } = parsed.data.body
try {
const seed = await buildFileDocSeed(workspaceId, fileId)
return NextResponse.json({
update: seed ? Buffer.from(seed.update).toString('base64') : null,
version: seed ? seed.version : null,
})
} catch (error) {
logger.error('Failed to build file-doc seed', { workspaceId, fileId, error })
return NextResponse.json(
{ error: getErrorMessage(error, 'Failed to build seed') },
{ status: 500 }
)
}
})
@@ -6,6 +6,7 @@ import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
import { generateRequestId } from '@/lib/core/utils/request'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { TableQueryValidationError } from '@/lib/table/errors'
import { signalTableRowsChanged } from '@/lib/table/events'
import { toLegacyFilter } from '@/lib/table/query-builder/converters'
import { cancelWorkflowGroupRuns } from '@/lib/table/workflow-columns'
import { accessError, checkAccess, tableFilterError } from '@/app/api/table/utils'
@@ -60,6 +61,12 @@ export const POST = withRouteHandler(async (request: NextRequest, { params }: Ro
} cancelled=${cancelled}`
)
// Cancelling clears/tombstones affected rows' exec state in the DB. The `dispatch: cancelled` events
// drop the run overlay, but the client then renders the row's authoritative DB state — so refetch the
// grid to pick up the cleared cells. Unconditional: `cancelled` counts dispatches, but tombstone row
// writes can happen even when that is 0, and a stale-but-harmless refetch beats a missed one.
signalTableRowsChanged(tableId)
return NextResponse.json({ success: true, data: { cancelled } })
} catch (error) {
// A predicate that Zod accepts but the downgrade rejects (hybrid node,
@@ -22,6 +22,7 @@ import {
import { columnMatchesRef, getColumnId } from '@/lib/table/column-keys'
import { columnTypeById } from '@/lib/table/column-types'
import { isSupportedCurrencyCode } from '@/lib/table/currency'
import { signalTableSchemaChanged } from '@/lib/table/events'
import {
accessError,
checkAccess,
@@ -62,6 +63,7 @@ export const POST = withRouteHandler(async (request: NextRequest, context: Colum
}
const updatedTable = await addTableColumn(tableId, validated.column, requestId)
signalTableSchemaChanged(tableId)
return NextResponse.json({
success: true,
@@ -295,6 +297,7 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu
if (!updatedTable) {
return NextResponse.json({ error: 'No updates specified' }, { status: 400 })
}
signalTableSchemaChanged(tableId)
return NextResponse.json({
success: true,
@@ -364,6 +367,7 @@ export const DELETE = withRouteHandler(
{ tableId, columnName: validated.columnName },
requestId
)
signalTableSchemaChanged(tableId)
return NextResponse.json({
success: true,
@@ -6,6 +6,7 @@ import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
import { generateRequestId } from '@/lib/core/utils/request'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { TableQueryValidationError } from '@/lib/table/errors'
import { signalTableRowsChanged } from '@/lib/table/events'
import { toLegacyFilter } from '@/lib/table/query-builder/converters'
import { runWorkflowColumn } from '@/lib/table/workflow-columns'
import { accessError, checkAccess, tableFilterError } from '@/app/api/table/utils'
@@ -59,6 +60,13 @@ export const POST = withRouteHandler(async (request: NextRequest, { params }: Ro
triggeredByUserId: auth.userId,
})
// Starting a run clears the target group's cells to pending (`bulkClearWorkflowGroupCells`) — a DB
// row change. The `dispatch: dispatching` events drive the run overlay, but the cleared cell values
// come from the rows query, so refetch the grid. Unconditional: the bulk clear can run even on a path
// that then returns a null `dispatchId` (dispatch cancelled post-clear), and a stale-but-harmless
// refetch beats a missed one.
signalTableRowsChanged(tableId)
return NextResponse.json({ success: true, data: { dispatchId } })
} catch (error) {
// A predicate that Zod accepts but the downgrade rejects (hybrid node,
@@ -1,26 +1,13 @@
import { createLogger } from '@sim/logger'
import { toError } from '@sim/utils/errors'
import { sleep } from '@sim/utils/helpers'
import { type NextRequest, NextResponse } from 'next/server'
import { tableEventStreamContract } from '@/lib/api/contracts/tables'
import { parseRequest } from '@/lib/api/server'
import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
import { generateRequestId } from '@/lib/core/utils/request'
import { SSE_HEADERS } from '@/lib/core/utils/sse'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import {
getLatestTableEventId,
readTableEventsSince,
type TableEventEntry,
} from '@/lib/table/events'
import { createEventStreamResponse } from '@/lib/realtime/event-stream-route'
import { getLatestTableEventId, readTableEventsSince } from '@/lib/table/events'
import { accessError, checkAccess } from '@/app/api/table/utils'
const logger = createLogger('TableEventStreamAPI')
const POLL_INTERVAL_MS = 500
const HEARTBEAT_INTERVAL_MS = 15_000
const MAX_STREAM_DURATION_MS = 4 * 60 * 60 * 1000 // 4 hours; client reconnects past this
export const runtime = 'nodejs'
export const dynamic = 'force-dynamic'
@@ -30,12 +17,9 @@ interface RouteContext {
/** GET /api/table/[tableId]/events/stream?from=<lastEventId>
*
* SSE stream of cell-state transitions. Replay-on-reconnect via `from`;
* absent `from` tails from the latest event id (fresh mount — the client has
* just fetched current state, so replaying history would rewind it).
* Pruning (buffer cap exceeded or TTL expired) sends a `pruned` event and
* closes; the client responds with a full row-query refetch and reconnects
* tailing from latest. */
* SSE stream of cell-state transitions over the shared durable event log. Auth
* and access are checked here; the replay/tail/poll/heartbeat/prune mechanics
* come from `createEventStreamResponse`. */
export const GET = withRouteHandler(async (req: NextRequest, context: RouteContext) => {
const requestId = generateRequestId()
const parsed = await parseRequest(tableEventStreamContract, req, context)
@@ -51,123 +35,13 @@ export const GET = withRouteHandler(async (req: NextRequest, context: RouteConte
const access = await checkAccess(tableId, auth.userId, 'read')
if (!access.ok) return accessError(access, requestId, tableId)
logger.info(`[${requestId}] Table event stream opened`, { tableId, fromEventId })
const encoder = new TextEncoder()
let closed = false
const stream = new ReadableStream<Uint8Array>({
async start(controller) {
let lastEventId = fromEventId ?? 0
const deadline = Date.now() + MAX_STREAM_DURATION_MS
let nextHeartbeatAt = Date.now() + HEARTBEAT_INTERVAL_MS
const enqueue = (text: string) => {
if (closed) return
try {
controller.enqueue(encoder.encode(text))
} catch {
closed = true
}
}
const sendEvents = (events: TableEventEntry[]) => {
for (const entry of events) {
if (closed) return
enqueue(`data: ${JSON.stringify(entry)}\n\n`)
lastEventId = entry.eventId
}
}
const sendPrunedAndClose = (earliestEventId: number | undefined) => {
enqueue(
`event: pruned\ndata: ${JSON.stringify({ earliestEventId: earliestEventId ?? null })}\n\n`
)
if (!closed) {
closed = true
try {
controller.close()
} catch {}
}
}
const sendHeartbeat = () => {
// SSE comment line — keeps proxies (ALB default 60s idle) from closing
// the connection during quiet periods.
enqueue(`: ping ${Date.now()}\n\n`)
}
try {
// No replay cursor → tail from the latest event id. Resolved inside
// the try so a Redis failure errors the stream (client reconnects
// with backoff) rather than silently replaying the whole buffer.
if (fromEventId === undefined) {
lastEventId = await getLatestTableEventId(tableId)
}
// Initial replay from buffer.
const initial = await readTableEventsSince(tableId, lastEventId)
if (initial.status === 'pruned') {
sendPrunedAndClose(initial.earliestEventId)
return
}
if (initial.status === 'unavailable') {
throw new Error(`Table event buffer unavailable: ${initial.error}`)
}
sendEvents(initial.events)
// Stream loop — poll the buffer and forward new events. Workflow
// execution stream uses the same shape; pub/sub wakeups are an
// optimization we can add later if 500ms polling becomes a problem.
while (!closed && Date.now() < deadline) {
await sleep(POLL_INTERVAL_MS)
if (closed) return
const result = await readTableEventsSince(tableId, lastEventId)
if (result.status === 'pruned') {
sendPrunedAndClose(result.earliestEventId)
return
}
if (result.status === 'unavailable') {
throw new Error(`Table event buffer unavailable: ${result.error}`)
}
if (result.events.length > 0) {
sendEvents(result.events)
}
if (Date.now() >= nextHeartbeatAt) {
sendHeartbeat()
nextHeartbeatAt = Date.now() + HEARTBEAT_INTERVAL_MS
}
}
// Reached the defensive duration ceiling — close cleanly so the client
// reconnects with the latest lastEventId.
if (!closed) {
enqueue(`event: rotate\ndata: {}\n\n`)
closed = true
try {
controller.close()
} catch {}
}
} catch (error) {
logger.error(`[${requestId}] Table event stream error`, {
tableId,
error: toError(error).message,
})
if (!closed) {
try {
controller.error(error)
} catch {}
}
}
},
cancel() {
closed = true
logger.info(`[${requestId}] Client disconnected from table event stream`, { tableId })
},
})
return new NextResponse(stream, {
headers: { ...SSE_HEADERS, 'X-Table-Id': tableId },
return createEventStreamResponse({
requestId,
label: 'table',
streamId: tableId,
fromEventId,
getLatestEventId: getLatestTableEventId,
readEventsSince: readTableEventsSince,
extraHeaders: { 'X-Table-Id': tableId },
})
})
@@ -10,6 +10,7 @@ import { parseRequest } from '@/lib/api/server'
import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
import { generateRequestId } from '@/lib/core/utils/request'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { signalTableSchemaChanged } from '@/lib/table/events'
import {
addWorkflowGroup,
deleteWorkflowGroup,
@@ -106,6 +107,7 @@ export const POST = withRouteHandler(async (request: NextRequest, { params }: Ro
},
requestId
)
signalTableSchemaChanged(tableId)
return NextResponse.json({
success: true,
data: {
@@ -168,6 +170,7 @@ export const PATCH = withRouteHandler(async (request: NextRequest, { params }: R
},
requestId
)
signalTableSchemaChanged(tableId)
return NextResponse.json({
success: true,
data: {
@@ -201,6 +204,7 @@ export const DELETE = withRouteHandler(async (request: NextRequest, { params }:
{ tableId, groupId: validated.groupId },
requestId
)
signalTableSchemaChanged(tableId)
return NextResponse.json({
success: true,
data: {
@@ -37,6 +37,7 @@ import {
wouldExceedRowLimit,
} from '@/lib/table'
import { sniffCsvDelimiterFromStream } from '@/lib/table/csv-delimiter-stream'
import { signalTableSchemaChanged } from '@/lib/table/events'
import { importAppendRows, importReplaceRows } from '@/lib/table/import-data'
import { getUserSettings } from '@/lib/users/queries'
import {
@@ -323,6 +324,7 @@ export const POST = withRouteHandler(async (request: NextRequest, { params }: Ro
mappedColumns: validation.mappedHeaders.length,
skippedHeaders: validation.skippedHeaders.length,
})
signalTableSchemaChanged(tableId)
return NextResponse.json({
success: true,
@@ -385,6 +387,7 @@ export const POST = withRouteHandler(async (request: NextRequest, { params }: Ro
createdColumns: additions.length,
mappedColumns: validation.mappedHeaders.length,
})
signalTableSchemaChanged(tableId)
return NextResponse.json({
success: true,
@@ -7,6 +7,7 @@ import { generateRequestId } from '@/lib/core/utils/request'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import type { TableMetadata } from '@/lib/table'
import { updateTableMetadata } from '@/lib/table'
import { signalTableMetadataChanged, signalTableSchemaChanged } from '@/lib/table/events'
import { accessError, checkAccess } from '@/app/api/table/utils'
const logger = createLogger('TableMetadataAPI')
@@ -43,11 +44,20 @@ export const PUT = withRouteHandler(async (request: NextRequest, context: TableR
return NextResponse.json({ error: 'Invalid workspace ID' }, { status: 400 })
}
const updated = await updateTableMetadata(
const { metadata: updated, schemaChanged } = await updateTableMetadata(
tableId,
validated.metadata,
table.metadata as TableMetadata | null
)
// Signal collaborators to re-apply the new column layout (width/pin/order) live; the
// grid reconciles against its in-progress resize/drag so a peer's change never clobbers
// the local gesture. A reorder that scrubs a workflow-group's dependencies also mutated
// the schema — escalate to the schema signal so peers refresh run-state/enrichment too.
if (schemaChanged) {
signalTableSchemaChanged(tableId)
} else {
signalTableMetadataChanged(tableId)
}
return NextResponse.json({ success: true, data: { metadata: updated } })
} catch (error) {
@@ -19,6 +19,7 @@ import {
updateTableLocks,
} from '@/lib/table'
import { getWorkspaceTableLimits } from '@/lib/table/billing'
import { signalTableSchemaChanged } from '@/lib/table/events'
import { TABLE_LOCK_FLAGS, TABLE_LOCK_KINDS } from '@/lib/table/types'
import { getWorkspaceWithOwner } from '@/lib/workspaces/permissions/utils'
import {
@@ -185,6 +186,8 @@ export const PATCH = withRouteHandler(
if (validated.name !== undefined) {
await renameTable(tableId, validated.name, requestId, authResult.userId)
// Live-collab: tell open viewers the definition changed so they refetch.
signalTableSchemaChanged(tableId)
}
if (validated.folderId !== undefined) {
@@ -15,6 +15,7 @@ import { generateRequestId } from '@/lib/core/utils/request'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import type { RowData, TableSchema } from '@/lib/table'
import { deleteRow, updateRow } from '@/lib/table'
import { signalTableRowsChanged } from '@/lib/table/events'
import { rowWireTranslators } from '@/app/api/table/row-wire'
import {
accessError,
@@ -147,6 +148,9 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: RowR
// Only `null` when a `cancellationGuard` is supplied and the SQL guard
// rejects the write — this route doesn't pass one, so reaching null is a bug.
if (!updatedRow) throw new Error('updateRow returned null without a cancellationGuard')
// An edit that also triggers a dispatch already emits dispatch/cell events; the
// debounced rows refetch on the peer coalesces the two.
signalTableRowsChanged(tableId)
// Auto-dispatch for user edits is handled inside `updateRow` (mode: 'new').
// Firing a second mode: 'incomplete' dispatch here would race with the
// `mode: 'new'` one AND bulk-clear sibling-group outputs (the incomplete
@@ -213,6 +217,7 @@ export const DELETE = withRouteHandler(async (request: NextRequest, context: Row
}
await deleteRow(table, rowId, requestId)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -26,6 +26,7 @@ import {
validateRowSize,
} from '@/lib/table'
import { TableQueryValidationError } from '@/lib/table/errors'
import { signalTableRowsChanged } from '@/lib/table/events'
import { isTablePredicate, predicateToFilter } from '@/lib/table/query-builder/converters'
import {
validatePredicateShape,
@@ -122,6 +123,7 @@ async function handleBatchInsert(
table,
requestId
)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -207,6 +209,7 @@ export const POST = withRouteHandler(
table,
requestId
)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -433,6 +436,7 @@ export const PUT = withRouteHandler(
{ status: 200 }
)
}
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -499,6 +503,7 @@ export const DELETE = withRouteHandler(
{ tableId, rowIds: validated.rowIds, workspaceId: validated.workspaceId },
requestId
)
if (result.deletedCount > 0) signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -528,6 +533,7 @@ export const DELETE = withRouteHandler(
},
requestId
)
if (result.affectedCount > 0) signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -601,6 +607,7 @@ export const PATCH = withRouteHandler(
table,
requestId
)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -8,6 +8,7 @@ import { generateRequestId } from '@/lib/core/utils/request'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import type { RowData, TableSchema } from '@/lib/table'
import { upsertRow } from '@/lib/table'
import { signalTableRowsChanged } from '@/lib/table/events'
import { rowWireTranslators } from '@/app/api/table/row-wire'
import { accessError, checkAccess, rowWriteErrorResponse } from '@/app/api/table/utils'
@@ -54,6 +55,7 @@ export const POST = withRouteHandler(async (request: NextRequest, context: Upser
table,
requestId
)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -21,6 +21,7 @@ import {
import { columnMatchesRef, getColumnId } from '@/lib/table/column-keys'
import { columnTypeById } from '@/lib/table/column-types'
import { isSupportedCurrencyCode } from '@/lib/table/currency'
import { signalTableSchemaChanged } from '@/lib/table/events'
import {
accessError,
checkAccess,
@@ -76,6 +77,7 @@ export const POST = withRouteHandler(async (request: NextRequest, context: Colum
}
const updatedTable = await addTableColumn(tableId, validated.column, requestId)
signalTableSchemaChanged(tableId)
recordAudit({
workspaceId: validated.workspaceId,
@@ -329,6 +331,7 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu
if (!updatedTable) {
return NextResponse.json({ error: 'No updates specified' }, { status: 400 })
}
signalTableSchemaChanged(tableId)
recordAudit({
workspaceId: validated.workspaceId,
@@ -416,6 +419,7 @@ export const DELETE = withRouteHandler(
{ tableId, columnName: validated.columnName },
requestId
)
signalTableSchemaChanged(tableId)
recordAudit({
workspaceId: validated.workspaceId,
@@ -16,6 +16,7 @@ import type { RowData, TableSchema } from '@/lib/table'
import { deleteRow, updateRow } from '@/lib/table'
import { namedRowMapper } from '@/lib/table/cell-format'
import { buildIdByName, rowDataNameToId } from '@/lib/table/column-keys'
import { signalTableRowsChanged } from '@/lib/table/events'
import { accessError, checkAccess, tableLockErrorResponse } from '@/app/api/table/utils'
import {
checkRateLimit,
@@ -159,6 +160,7 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: RowR
if (!updatedRow) {
return NextResponse.json({ error: 'Row not found' }, { status: 404 })
}
signalTableRowsChanged(tableId)
// Auto-dispatch for user edits is handled inside `updateRow` (mode: 'new').
// Firing a second mode: 'incomplete' dispatch here would race with it AND
// bulk-clear sibling-group outputs.
@@ -241,6 +243,7 @@ export const DELETE = withRouteHandler(async (request: NextRequest, context: Row
// Route through the service (not a raw `db.delete`) so the delete lock is
// enforced — the raw path would return 200 on a locked table.
await deleteRow(result.table, rowId, requestId)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -29,6 +29,7 @@ import {
sortNamesToIds,
} from '@/lib/table/column-keys'
import { TableQueryValidationError } from '@/lib/table/errors'
import { signalTableRowsChanged } from '@/lib/table/events'
import { queryRows } from '@/lib/table/rows/service'
import { resolveFilterSelectValues } from '@/lib/table/select-values'
import { accessError, checkAccess, rowWriteErrorResponse } from '@/app/api/table/utils'
@@ -89,6 +90,7 @@ async function handleBatchInsert(
table,
requestId
)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -282,6 +284,7 @@ export const POST = withRouteHandler(
table,
requestId
)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -367,6 +370,7 @@ export const PUT = withRouteHandler(async (request: NextRequest, context: TableR
},
requestId
)
if (result.affectedCount > 0) signalTableRowsChanged(tableId)
if (result.affectedCount === 0) {
return NextResponse.json({
@@ -439,6 +443,7 @@ export const DELETE = withRouteHandler(
{ tableId, rowIds: validated.rowIds, workspaceId: validated.workspaceId },
requestId
)
if (result.deletedCount > 0) signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -467,6 +472,7 @@ export const DELETE = withRouteHandler(
},
requestId
)
if (result.affectedCount > 0) signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -9,6 +9,7 @@ import type { RowData, TableSchema } from '@/lib/table'
import { upsertRow } from '@/lib/table'
import { namedRowMapper } from '@/lib/table/cell-format'
import { buildIdByName, rowDataNameToId } from '@/lib/table/column-keys'
import { signalTableRowsChanged } from '@/lib/table/events'
import { accessError, checkAccess, tableLockErrorResponse } from '@/app/api/table/utils'
import {
checkRateLimit,
@@ -75,6 +76,7 @@ export const POST = withRouteHandler(async (request: NextRequest, context: Upser
table,
requestId
)
signalTableRowsChanged(tableId)
return NextResponse.json({
success: true,
@@ -7,6 +7,7 @@ import { parseRequest } from '@/lib/api/server'
import { getSession } from '@/lib/auth'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { captureServerEvent } from '@/lib/posthog/server'
import { notifyWorkspaceFilesChanged } from '@/lib/realtime/notify'
import {
FileConflictError,
parseWorkspaceFileKey,
@@ -63,6 +64,8 @@ export const POST = withRouteHandler(
if (created) {
logger.info(`Registered direct upload ${name} -> ${key}`)
await notifyWorkspaceFilesChanged(workspaceId)
captureServerEvent(
userId,
'file_uploaded',
@@ -0,0 +1,116 @@
'use client'
import { Avatar, AvatarFallback, AvatarImage, cn, Tooltip } from '@sim/emcn'
import { getUserColor } from '@/lib/workspaces/colors'
/** Minimal presence shape the avatar stack renders — shared by workflow and files. */
export interface PresenceAvatarUser {
/** Unique id per presence entry, used as the render key: a socket id where presence is
* per-connection (workflow), absent where entries are deduped per user (file docs). */
socketId?: string
userId: string
userName?: string
avatarUrl?: string | null
}
interface UserAvatarProps {
user: PresenceAvatarUser
index: number
}
/**
* A single collaborator avatar: their image, falling back to a colored circle
* with their initial. Wrapped in a name tooltip when the name is known.
*/
function UserAvatar({ user, index }: UserAvatarProps) {
const color = getUserColor(user.userId)
const initials = user.userName ? user.userName.charAt(0).toUpperCase() : '?'
const avatarElement = (
<Avatar size='xs' style={{ zIndex: index + 1 }}>
{user.avatarUrl && (
<AvatarImage
src={user.avatarUrl}
alt={user.userName ? `${user.userName}'s avatar` : 'User avatar'}
referrerPolicy='no-referrer'
/>
)}
<AvatarFallback
style={{ background: color }}
className='border-0 font-semibold text-[7px] text-white leading-none'
>
{initials}
</AvatarFallback>
</Avatar>
)
if (user.userName) {
return (
<Tooltip.Root>
<Tooltip.Trigger asChild>{avatarElement}</Tooltip.Trigger>
<Tooltip.Content side='bottom'>
<span>{user.userName}</span>
</Tooltip.Content>
</Tooltip.Root>
)
}
return avatarElement
}
interface PresenceAvatarsProps {
/** Collaborators to show — already filtered to exclude the current socket. */
users: PresenceAvatarUser[]
/** Max avatars before collapsing the remainder into a "+N" chip. */
maxVisible?: number
/** Layout-only classes for the outer stack (e.g. surrounding margin); chrome is owned here. */
className?: string
}
const DEFAULT_MAX_VISIBLE = 5
/**
* Overlapping stack of collaborator avatars for presence. Presentational only
* the caller owns fetching/filtering presence (workflow sidebar item, files
* header, etc.), so the stack looks identical everywhere it appears.
*/
export function PresenceAvatars({
users,
maxVisible = DEFAULT_MAX_VISIBLE,
className,
}: PresenceAvatarsProps) {
// Reverse so the rightmost avatar stays stable as new ones reveal on the left.
// slice() already returns a fresh array, so the in-place reverse is safe.
const visibleUsers = users.slice(0, maxVisible).reverse()
const overflowCount = Math.max(0, users.length - maxVisible)
if (visibleUsers.length === 0) {
return null
}
return (
<div className={cn('-space-x-1 flex flex-shrink-0 items-center', className)}>
{overflowCount > 0 && (
<Tooltip.Root>
<Tooltip.Trigger asChild>
<Avatar
size='xs'
style={{ zIndex: 0 }}
aria-label={`${overflowCount} more ${overflowCount === 1 ? 'user' : 'users'}`}
>
<AvatarFallback className='border-0 bg-[#404040] font-semibold text-[7px] text-white leading-none'>
+{overflowCount}
</AvatarFallback>
</Avatar>
</Tooltip.Trigger>
<Tooltip.Content side='bottom'>
{overflowCount} more user{overflowCount > 1 ? 's' : ''}
</Tooltip.Content>
</Tooltip.Root>
)}
{visibleUsers.map((user, index) => (
<UserAvatar key={user.socketId ?? user.userId} user={user} index={index} />
))}
</div>
)
}
@@ -108,8 +108,20 @@ interface FileViewerProps {
streamingContent?: string
isAgentEditing?: boolean
streamIsIncremental?: boolean
streamOperation?: string
disableStreamingAutoScroll?: boolean
previewContextKey?: string
/**
* Opt this surface into live collaborative editing (markdown files only). Set by the
* Files page; the agent/Chat surface leaves it off so collaboration and agent-streaming
* never target one editor. See {@link RichMarkdownEditorProps.collaborative}.
*/
collaborative?: boolean
/**
* Called (debounced) with the markdown document's leading-heading text while the file is still
* untitled, so the caller can name the file after it. Only wired for the editable markdown editor.
*/
onDeriveTitleFromHeading?: (headingText: string) => void
}
export function FileViewer(props: FileViewerProps) {
@@ -139,8 +151,11 @@ function FileViewerContent({
streamingContent,
isAgentEditing,
streamIsIncremental,
streamOperation,
disableStreamingAutoScroll = false,
previewContextKey,
collaborative,
onDeriveTitleFromHeading,
}: FileViewerProps) {
const category = resolveFileCategory(file.type, file.name)
@@ -183,8 +198,11 @@ function FileViewerContent({
streamingContent={streamingContent}
isAgentEditing={isAgentEditing}
streamIsIncremental={streamIsIncremental}
streamOperation={streamOperation}
disableStreamingAutoScroll={disableStreamingAutoScroll}
previewContextKey={previewContextKey}
collaborative={collaborative}
onDeriveTitleFromHeading={onDeriveTitleFromHeading}
/>
)
}
@@ -0,0 +1,34 @@
import type { JSONContent } from '@tiptap/core'
import { CodeBlock } from '@tiptap/extension-code-block'
/**
* React-free schema half of the code-block node. Lives apart from {@link ./code-block} (its React
* node view) so the shared editor schema `createMarkdownContentExtensions` in `./extensions` can
* be imported by server code (the collab-doc seed converter) without pulling a client component
* (`useEffect`) into a Server Component module. The client editor injects the node-view variant
* ({@link CodeBlockWithLanguage}) via `nodeViews`.
*/
function codeBlockText(node: JSONContent): string {
return (node.content ?? []).map((child) => child.text ?? '').join('')
}
/** Fence sized to one backtick longer than the longest run inside the code (CommonMark rule). */
function fenceFor(text: string): string {
const longestRun = Math.max(0, ...[...text.matchAll(/`+/g)].map((match) => match[0].length))
return '`'.repeat(Math.max(3, longestRun + 1))
}
/**
* Code block whose markdown serializer sizes the fence to the interior backtick runs, so a code
* block that itself contains a ``` line round-trips instead of shattering. Shared by the test
* (plain) and live ({@link CodeBlockWithLanguage}) paths.
*/
export const MarkdownCodeBlock = CodeBlock.extend({
renderMarkdown: (node: JSONContent) => {
const language = typeof node.attrs?.language === 'string' ? node.attrs.language : ''
const text = codeBlockText(node)
const fence = fenceFor(text)
return `${fence}${language}\n${text}\n${fence}`
},
})
@@ -8,12 +8,11 @@ import {
DropdownMenuTrigger,
useCopyToClipboard,
} from '@sim/emcn'
import type { JSONContent } from '@tiptap/core'
import { CodeBlock } from '@tiptap/extension-code-block'
import type { ReactNodeViewProps } from '@tiptap/react'
import { NodeViewContent, NodeViewWrapper, ReactNodeViewRenderer } from '@tiptap/react'
import { Check, ChevronDown, Code, Copy, Eye, WrapText } from 'lucide-react'
import { looksLikeMermaid, MermaidDiagram } from '../mermaid-diagram'
import { MarkdownCodeBlock } from './code-block-schema'
import { detectLanguage } from './detect-language'
import { useEditorEditable } from './use-editor-editable'
@@ -228,30 +227,6 @@ function CodeBlockView({ node, updateAttributes, editor, getPos }: ReactNodeView
)
}
function codeBlockText(node: JSONContent): string {
return (node.content ?? []).map((child) => child.text ?? '').join('')
}
/** Fence sized to one backtick longer than the longest run inside the code (CommonMark rule). */
function fenceFor(text: string): string {
const longestRun = Math.max(0, ...[...text.matchAll(/`+/g)].map((match) => match[0].length))
return '`'.repeat(Math.max(3, longestRun + 1))
}
/**
* Code block whose markdown serializer sizes the fence to the interior backtick runs, so a code
* block that itself contains a ``` line round-trips instead of shattering. Shared by the test
* (plain) and live ({@link CodeBlockWithLanguage}) paths.
*/
export const MarkdownCodeBlock = CodeBlock.extend({
renderMarkdown: (node: JSONContent) => {
const language = typeof node.attrs?.language === 'string' ? node.attrs.language : ''
const text = codeBlockText(node)
const fence = fenceFor(text)
return `${fence}${language}\n${text}\n${fence}`
},
})
/**
* Code block with hover-revealed controls (language picker, line-wrap toggle, copy). The
* `language` attribute drives {@link CodeBlockHighlight}'s Prism highlighting and serializes to
@@ -0,0 +1,55 @@
/**
* @vitest-environment node
*/
import { describe, expect, it } from 'vitest'
import { Awareness } from 'y-protocols/awareness'
import * as Y from 'yjs'
import {
announceAgentApplying,
clearAgentApplying,
isAgentStreamLeader,
} from './agent-stream-leader'
/** An Awareness with explicit peer states injected (self, if present, carries no `agentApplying`). */
function awarenessWith(entries: Array<[number, Record<string, unknown>]>): Awareness {
const aw = new Awareness(new Y.Doc())
const states = aw.getStates() as Map<number, Record<string, unknown>>
for (const [clientId, state] of entries) states.set(clientId, state)
return aw
}
describe('agent-stream leader election', () => {
it('a sole announcer is the leader', () => {
expect(isAgentStreamLeader(awarenessWith([[5, { agentApplying: true }]]), 5)).toBe(true)
})
it('the lowest clientID among announcers leads; higher announcers do not', () => {
const aw = awarenessWith([
[7, { agentApplying: true }],
[3, { agentApplying: true }],
[9, { user: { name: 'someone else, not applying' } }],
])
expect(isAgentStreamLeader(aw, 3)).toBe(true)
expect(isAgentStreamLeader(aw, 7)).toBe(false)
})
it('a client that is not announcing is never the leader', () => {
expect(isAgentStreamLeader(awarenessWith([[3, { agentApplying: true }]]), 8)).toBe(false)
})
it('with no announcers, nobody leads', () => {
expect(isAgentStreamLeader(awarenessWith([[3, { user: {} }]]), 3)).toBe(false)
})
it('announce makes self the leader; clear relinquishes it', () => {
const doc = new Y.Doc()
const aw = new Awareness(doc)
announceAgentApplying(aw)
expect(aw.getLocalState()?.agentApplying).toBe(true)
expect(isAgentStreamLeader(aw, doc.clientID)).toBe(true)
clearAgentApplying(aw)
expect(aw.getLocalState()?.agentApplying ?? null).toBeNull()
expect(isAgentStreamLeader(aw, doc.clientID)).toBe(false)
})
})
@@ -0,0 +1,37 @@
import type { Awareness } from 'y-protocols/awareness'
/**
* Awareness field a collaborative client sets on its OWN state while it is applying an agent stream into
* the shared doc. Read by every peer to run the single-writer election below. Coexists with the caret
* `user` field (`setLocalStateField` writes one field without clobbering others).
*/
const AGENT_APPLYING_FIELD = 'agentApplying'
/** Announce that this client is applying an agent stream (candidate in the leader election). */
export function announceAgentApplying(awareness: Awareness): void {
awareness.setLocalStateField(AGENT_APPLYING_FIELD, true)
}
/** Stop announcing (this client is no longer applying an agent stream). */
export function clearAgentApplying(awareness: Awareness): void {
awareness.setLocalStateField(AGENT_APPLYING_FIELD, null)
}
/**
* Single-writer election: exactly one collaborative client applies a given agent stream into the shared
* doc, so N tabs/windows watching the same live copilot stream don't each insert it under a different
* Yjs clientID and duplicate the content. The leader is the MINIMUM clientID among all clients currently
* announcing (via {@link announceAgentApplying}) that they are applying a deterministic tie-break that
* needs no coordinator. A brief startup race (before an announcement propagates to peers) is bounded to a
* frame or two self-corrected the moment awareness converges, and reconciled anyway by the durable
* server write. In the common single-client case the caller is the only announcer, so it always leads.
*/
export function isAgentStreamLeader(awareness: Awareness, selfClientId: number): boolean {
let leader = Number.POSITIVE_INFINITY
awareness.getStates().forEach((state, clientId) => {
if ((state as Record<string, unknown> | undefined)?.[AGENT_APPLYING_FIELD] === true) {
leader = Math.min(leader, clientId)
}
})
return leader === selfClientId
}
@@ -0,0 +1,189 @@
/**
* @vitest-environment jsdom
*/
import { Editor } from '@tiptap/core'
import { initProseMirrorDoc, updateYFragment, ySyncPluginKey } from '@tiptap/y-tiptap'
import { afterEach, beforeAll, describe, expect, it } from 'vitest'
import { Awareness } from 'y-protocols/awareness'
import * as Y from 'yjs'
import { createMarkdownEditorExtensions } from '../editor-extensions'
import { applyAgentStreamFrame, beginAgentStream, endAgentStream } from './apply-streamed-markdown'
beforeAll(() => {
// jsdom does not implement elementFromPoint; the Placeholder extension's viewport tracking calls it
// on view mount. Returning null makes ProseMirror's posAtCoords fall back gracefully.
if (!document.elementFromPoint) {
document.elementFromPoint = () => null
}
})
/** A headless collaborative editor bound to a fresh Y.Doc — the same extension wiring the component uses. */
function makeCollabEditor() {
const doc = new Y.Doc()
const awareness = new Awareness(doc)
const editor = new Editor({
extensions: createMarkdownEditorExtensions({
placeholder: '',
collaboration: {
doc,
awareness,
user: { name: 'Tester', color: '#ffffff', clientId: doc.clientID },
},
}),
content: '',
})
return { editor, doc, awareness }
}
const teardown: Array<() => void> = []
afterEach(() => {
for (const fn of teardown.splice(0)) fn()
})
function track(t: { editor: Editor; doc: Y.Doc; awareness: Awareness }) {
teardown.push(() => {
t.editor.destroy()
t.awareness.destroy()
t.doc.destroy()
})
return t
}
describe('agent-stream applier', () => {
it('relies on y-tiptap internals that still exist (upgrade guardrail)', () => {
// beginAgentStream/applyAgentStreamFrame reach into y-tiptap internals (not public TipTap API):
// `ySyncPluginKey`, `updateYFragment`, `initProseMirrorDoc`. A y-tiptap bump that renames or drops
// any of them can pass typecheck yet break at runtime — assert their runtime shape here so an upgrade
// fails loudly at test time instead of in production. Pinned to an exact y-tiptap version in
// package.json; bump that pin and this guard together.
expect(typeof updateYFragment).toBe('function')
expect(typeof initProseMirrorDoc).toBe('function')
expect(ySyncPluginKey).toBeDefined()
expect(typeof ySyncPluginKey.getState).toBe('function')
})
it('streams agent content into the live collaborative doc and broadcasts it as Yjs ops', () => {
const { editor, doc } = track(makeCollabEditor())
const session = beginAgentStream(editor)
expect(session).not.toBeNull()
expect(applyAgentStreamFrame(editor, session!, '# Title\n\nHello world.')).toBe(true)
expect(editor.getText()).toContain('Hello world')
// The write lands as ops on the shared doc, so any peer receives it (this is what makes a
// collaborator on /files see the stream without ever holding `streamingContent`).
const peer = new Y.Doc()
Y.applyUpdate(peer, Y.encodeStateAsUpdate(doc))
expect(peer.getXmlFragment('default').toString()).toContain('Hello world')
peer.destroy()
endAgentStream(session!)
})
it('beginAgentStream returns null when the editor has no ySync binding', () => {
// A plain editor with no collaboration has no ySync binding, so a stream cannot start against it.
const editor = new Editor({
extensions: createMarkdownEditorExtensions({ placeholder: '' }),
content: '',
})
teardown.push(() => editor.destroy())
expect(beginAgentStream(editor)).toBeNull()
})
it('keeps agent-streamed ops out of the undo stack while user edits stay undoable', () => {
const { editor } = track(makeCollabEditor())
const session = beginAgentStream(editor)!
applyAgentStreamFrame(editor, session, '# Streamed\n\nAgent wrote this.')
endAgentStream(session)
// The streamed op relayed under a non-`ySyncPluginKey` origin, which the Collaboration UndoManager
// does not track — so there is nothing to undo, and an undo must not revert the agent's content.
expect(editor.can().undo()).toBe(false)
editor.commands.undo()
expect(editor.getText()).toContain('Agent wrote this')
// A genuine user edit IS captured (origin ySyncPluginKey) — proving the test isn't vacuous:
// undo works, and it reverts only the user edit, leaving the agent content intact.
editor.commands.focus('end')
editor.commands.insertContent(' USER-TYPED')
expect(editor.getText()).toContain('USER-TYPED')
expect(editor.can().undo()).toBe(true)
editor.commands.undo()
expect(editor.getText()).not.toContain('USER-TYPED')
expect(editor.getText()).toContain('Agent wrote this')
})
it('a shadow reused after the live doc advanced duplicates content; a fresh one does not', () => {
// The invariant behind the component's leadership-regain teardown: a shadow tracks only ITS OWN
// reconciles, so once the live doc advances under another writer, REUSING that stale shadow re-emits
// ops for content already present (duplication). Seeding a FRESH shadow from the current doc fixes it.
const stale = track(makeCollabEditor())
const staleSession = beginAgentStream(stale.editor)! // seeded from the empty base
applyAgentStreamFrame(stale.editor, staleSession, 'Alpha paragraph.')
// Another writer advances the live doc while this shadow is NOT looking (a handoff to an interim leader).
stale.editor.commands.focus('end')
stale.editor.commands.insertContent('\n\nBeta paragraph.')
// Reusing the stale shadow (only knows "Alpha") to reconcile toward the full body re-inserts "Beta".
applyAgentStreamFrame(
stale.editor,
staleSession,
'Alpha paragraph.\n\nBeta paragraph.\n\nGamma paragraph.'
)
endAgentStream(staleSession)
const staleText = stale.editor.getText()
expect(staleText.match(/Beta paragraph/g)?.length).toBe(2) // duplicated — what the teardown prevents
// Fresh shadow re-seeded from the CURRENT doc (what a regaining leader does after teardown) emits only
// the genuine delta, so no content duplicates.
const fresh = track(makeCollabEditor())
const first = beginAgentStream(fresh.editor)!
applyAgentStreamFrame(fresh.editor, first, 'Alpha paragraph.')
fresh.editor.commands.focus('end')
fresh.editor.commands.insertContent('\n\nBeta paragraph.')
endAgentStream(first)
const regained = beginAgentStream(fresh.editor)! // re-seeded from the advanced doc
applyAgentStreamFrame(
fresh.editor,
regained,
'Alpha paragraph.\n\nBeta paragraph.\n\nGamma paragraph.'
)
endAgentStream(regained)
const freshText = fresh.editor.getText()
expect(freshText.match(/Beta paragraph/g)?.length).toBe(1)
expect(freshText).toContain('Gamma paragraph')
})
it('preserves a concurrent peer edit to a region the agent snapshot does not include', () => {
// This is the core "AI as a CRDT peer" guarantee: the agent relays only its OWN delta (computed
// against a private shadow), never a whole-document reconcile that would revert a peer's edit.
const { editor, doc } = track(makeCollabEditor())
const session = beginAgentStream(editor)!
applyAgentStreamFrame(editor, session, 'Alpha paragraph.\n\nBeta paragraph.')
// A peer edits the FIRST paragraph directly on the shared doc — the agent's later snapshot still
// carries the ORIGINAL first paragraph (it was built from the base, before this edit).
const peer = new Y.Doc()
Y.applyUpdate(peer, Y.encodeStateAsUpdate(doc))
const peerFrag = peer.getXmlFragment('default')
peer.transact(() => {
const firstPara = peerFrag.get(0) as Y.XmlElement
const textNode = firstPara.get(0) as Y.XmlText
textNode.insert(textNode.toString().length, ' EDITED')
})
Y.applyUpdate(doc, Y.encodeStateAsUpdate(peer, Y.encodeStateVector(doc)))
peer.destroy()
// The agent appends a third paragraph. Its snapshot's first paragraph is the stale original, but the
// shadow-relayed delta only inserts the new paragraph — so the peer's " EDITED" must survive.
applyAgentStreamFrame(
editor,
session,
'Alpha paragraph.\n\nBeta paragraph.\n\nGamma paragraph.'
)
endAgentStream(session)
const live = doc.getXmlFragment('default').toString()
expect(live).toContain('EDITED')
expect(live).toContain('Gamma paragraph')
})
})
@@ -0,0 +1,78 @@
import type { Editor } from '@tiptap/core'
import { Node as PMNode } from '@tiptap/pm/model'
import { initProseMirrorDoc, updateYFragment, ySyncPluginKey } from '@tiptap/y-tiptap'
import * as Y from 'yjs'
import { parseMarkdownToDoc } from '../markdown-parse'
/** The Yjs fragment name TipTap's Collaboration extension binds to (its default `field`). */
const COLLAB_DOC_FIELD = 'default'
/**
* Transaction origin for agent-streamed writes into a live collaborative doc. It is deliberately NOT
* the `ySyncPluginKey` origin that local user edits use, so the Collaboration UndoManager which
* tracks only `ySyncPluginKey` excludes streamed ops from the user's undo stack.
*/
export const AGENT_STREAM_ORIGIN = Symbol('agent-stream')
/**
* A private Yjs replica the agent stream reconciles against, so a stream writes into the live doc as a
* TRUE peer: only the agent's own delta reaches the shared doc, never a whole-document reconcile that
* would revert a collaborator's concurrent edit. Seeded from the live doc at stream start; it receives
* ONLY agent reconciles (never peer updates), so `shadow → nextTarget` yields exactly the agent's change.
*/
export interface AgentStreamSession {
shadow: Y.Doc
fragment: Y.XmlFragment
}
/**
* Begin an agent stream by snapshotting the live doc into a private shadow replica. Returns `null` when
* the editor has no live ySync binding (e.g. a non-collaborative editor).
*/
export function beginAgentStream(editor: Editor): AgentStreamSession | null {
const binding = ySyncPluginKey.getState(editor.state)?.binding
if (!binding) return null
const shadow = new Y.Doc()
Y.applyUpdate(shadow, Y.encodeStateAsUpdate(binding.doc))
return { shadow, fragment: shadow.getXmlFragment(COLLAB_DOC_FIELD) }
}
/**
* Apply one streamed markdown body. Reconciles the shadow toward `body` with `updateYFragment` (the same
* minimal-diff primitive TipTap runs per keystroke), captures ONLY the resulting agent delta, and relays
* it into the live doc under {@link AGENT_STREAM_ORIGIN}. Because the shadow never sees peer updates, the
* delta touches only what the agent changed so concurrent peer edits elsewhere in the live doc survive,
* the change renders locally (via the binding's observer, the remote-edit path), broadcasts to every
* peer, and stays out of the user's undo stack. Returns `false` when the editor has no live ySync binding.
*/
export function applyAgentStreamFrame(
editor: Editor,
session: AgentStreamSession,
body: string
): boolean {
const binding = ySyncPluginKey.getState(editor.state)?.binding
if (!binding) return false
const target = PMNode.fromJSON(editor.schema, parseMarkdownToDoc(body))
let delta: Uint8Array | null = null
const capture = (update: Uint8Array, origin: unknown) => {
if (origin === AGENT_STREAM_ORIGIN) delta = update
}
session.shadow.on('update', capture)
try {
session.shadow.transact(() => {
// `updateYFragment` diffs against the fragment's CURRENT content, so it needs the fragment↔PM
// binding metadata; `initProseMirrorDoc` reconstructs it from the fragment's present state.
const { meta } = initProseMirrorDoc(session.fragment, editor.schema)
updateYFragment(session.shadow, session.fragment, target, meta)
}, AGENT_STREAM_ORIGIN)
} finally {
session.shadow.off('update', capture)
}
if (delta) Y.applyUpdate(binding.doc, delta, AGENT_STREAM_ORIGIN)
return true
}
/** End an agent stream and free its shadow replica. */
export function endAgentStream(session: AgentStreamSession): void {
session.shadow.destroy()
}
@@ -0,0 +1,55 @@
/**
* @vitest-environment jsdom
*/
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { activateCaretLabel, CARET_LABEL_HOLD_MS, renderCaret } from './caret-presence'
const ACTIVE = 'collaboration-carets__caret--active'
const FLIP = 'collaboration-carets__caret--flip'
describe('caret-presence', () => {
beforeEach(() => vi.useFakeTimers())
afterEach(() => vi.useRealTimers())
it('builds a tagged caret with a name label, shown on appearance', () => {
const caret = renderCaret({ name: 'Ada', color: '#f783ac', clientId: 4242 })
expect(caret.classList.contains('collaboration-carets__caret')).toBe(true)
expect(caret.dataset.caretClientId).toBe('4242')
expect(caret.style.getPropertyValue('--caret-color')).toBeTruthy()
const label = caret.querySelector('.collaboration-carets__label')
expect(label?.textContent).toBe('Ada')
expect(caret.classList.contains(ACTIVE)).toBe(true)
})
it('falls back to a default name for a bare user state', () => {
const caret = renderCaret({ clientId: 1 })
expect(caret.querySelector('.collaboration-carets__label')?.textContent).toBe('Collaborator')
})
it('hides the label after the inactivity hold, and re-activation restarts it', () => {
const caret = renderCaret({ name: 'Ada', color: '#f783ac', clientId: 4242 })
vi.advanceTimersByTime(CARET_LABEL_HOLD_MS - 1)
expect(caret.classList.contains(ACTIVE)).toBe(true)
vi.advanceTimersByTime(1)
expect(caret.classList.contains(ACTIVE)).toBe(false)
activateCaretLabel(caret)
expect(caret.classList.contains(ACTIVE)).toBe(true)
vi.advanceTimersByTime(CARET_LABEL_HOLD_MS)
expect(caret.classList.contains(ACTIVE)).toBe(false)
})
it('flips the label left only when it would overflow the editor right edge', () => {
const caret = renderCaret({ name: 'Ada', color: '#f783ac', clientId: 4242 })
const label = caret.querySelector<HTMLElement>('.collaboration-carets__label')
if (!label) throw new Error('label missing')
// double-cast-allowed: jsdom has no layout; stub the label's right edge for the measure
label.getBoundingClientRect = () => ({ right: 500 }) as unknown as DOMRect
activateCaretLabel(caret, 600) // editor edge past the label → no flip
expect(caret.classList.contains(FLIP)).toBe(false)
activateCaretLabel(caret, 400) // editor edge before the label's right → flip
expect(caret.classList.contains(FLIP)).toBe(true)
})
})
@@ -0,0 +1,163 @@
import { Extension } from '@tiptap/core'
import { Plugin, PluginKey } from '@tiptap/pm/state'
import type { Awareness } from 'y-protocols/awareness'
/**
* Remote-collaborator caret presence for the file editor: the name label's
* show-then-fade behavior and the edge-aware flip, plus the ProseMirror plugin
* that drives them off awareness activity.
*
* Matches Google Docs: the name flag shows only while the peer has typed in the last
* few seconds or on local hover, then hides after inactivity, leaving just the caret.
*/
/**
* How long a peer's name label stays visible after their last activity (cursor
* move / edit) before it fades, leaving just the colored caret bar.
*/
export const CARET_LABEL_HOLD_MS = 2000
/** Fallback caret color when a peer's awareness carries no `color`. */
export const DEFAULT_CARET_COLOR = '#000000'
/**
* The active-state class {@link activateCaretLabel} toggles on the caret node to reveal the
* name label; CSS transitions it back to hidden when removed. (yCursorPlugin reuses the DOM
* node and never re-runs `render`, which is why this file drives the class itself.)
*/
const CARET_ACTIVE_CLASS = 'collaboration-carets__caret--active'
/** The class that flips the label to the caret's left near the editor's right edge. */
const CARET_FLIP_CLASS = 'collaboration-carets__caret--flip'
/** Per-caret fade timers, keyed by the (reused) caret DOM node. */
const caretFadeTimers = new WeakMap<HTMLElement, ReturnType<typeof setTimeout>>()
/**
* Show a peer's name label, (re)start its fade timer, and flip it to the caret's
* left when it would run off the editor's right edge.
*
* yCursorPlugin renders each caret as a keyed widget decoration, so ProseMirror
* REUSES the same DOM node as the caret moves (verified: the render function is
* not re-invoked on a position change) a CSS animation therefore cannot restart
* on activity. Instead the activity signal is the awareness `change` event, which
* (unlike `update`) fires only on a real state change, not the 15s heartbeat, so
* an idle peer's label correctly stays hidden.
*/
export function activateCaretLabel(caret: HTMLElement, editorRight?: number) {
caret.classList.add(CARET_ACTIVE_CLASS)
const existing = caretFadeTimers.get(caret)
if (existing) clearTimeout(existing)
caretFadeTimers.set(
caret,
setTimeout(() => {
caret.classList.remove(CARET_ACTIVE_CLASS)
caretFadeTimers.delete(caret)
}, CARET_LABEL_HOLD_MS)
)
if (editorRight === undefined) return
const label = caret.querySelector<HTMLElement>('.collaboration-carets__label')
if (!label) return
// Measure the default (rightward) position, then flip left only if it overflows.
caret.classList.remove(CARET_FLIP_CLASS)
if (label.getBoundingClientRect().right > editorRight) {
caret.classList.add(CARET_FLIP_CLASS)
}
}
/**
* Builds a remote peer's caret DOM: a colored bar plus a name label tagged with
* the peer's Yjs client id (so {@link createCaretActivityExtension} can find and
* re-activate the reused node on later awareness changes). Shown immediately on
* (re)appearance; the fade timer hides it after inactivity. Passed to
* CollaborationCaret as its `render` option, which only supplies `user` so the
* client id rides along in the awareness `user` payload (each client stamps its own
* `doc.clientID`; see `use-file-doc-collaboration.ts`).
*/
export function renderCaret(user: Record<string, unknown>): HTMLElement {
const color = typeof user.color === 'string' ? user.color : DEFAULT_CARET_COLOR
const name = typeof user.name === 'string' && user.name ? user.name : 'Collaborator'
const clientId = typeof user.clientId === 'number' ? user.clientId : undefined
const caret = document.createElement('span')
caret.className = 'collaboration-carets__caret'
// One inline var drives the caret bar, the dormant cap, and the name tag (all in CSS).
caret.style.setProperty('--caret-color', color)
if (clientId !== undefined) caret.dataset.caretClientId = String(clientId)
// The visible caret bar is a SEPARATE, absolutely-positioned child — never an inline border on the
// caret span. The caret is a ProseMirror inline widget inserted between characters; an in-flow bar
// (border + width) reflows the surrounding text by ~1px each time a peer's caret appears or moves.
// An out-of-flow bar has zero layout footprint, so peer carets never nudge the document.
const bar = document.createElement('span')
bar.className = 'collaboration-carets__bar'
caret.appendChild(bar)
const label = document.createElement('div')
label.className = 'collaboration-carets__label'
label.textContent = name
caret.appendChild(label)
activateCaretLabel(caret)
return caret
}
/**
* Drives the caret name-label show-then-fade off awareness activity. Because the
* caret DOM node is reused across moves (see {@link activateCaretLabel}), the
* `render` function alone can't reveal the label when a peer moves this listens
* for awareness `change` events and re-activates the matching caret node. Deferred
* to the next frame so the node exists and is laid out (for the edge-flip measure).
*/
export function createCaretActivityExtension(awareness: Awareness): Extension {
return Extension.create({
name: 'collaborationCaretActivity',
addProseMirrorPlugins() {
return [
new Plugin({
key: new PluginKey('collaborationCaretActivity'),
view: (editorView) => {
// Coalesce bursts of awareness changes into a single rAF per frame, however many
// peers moved: accumulate the changed client ids, then re-activate each matching
// (reused) caret node once. The shared editorRight is read once up front; each moving
// caret then does one getBoundingClientRect for its edge-flip measure (bounded by the
// small number of concurrently-moving peers, not the awareness event rate).
let raf = 0
const pending = new Set<number>()
const flush = () => {
raf = 0
const editorRight = editorView.dom.getBoundingClientRect().right
for (const id of pending) {
const caret = editorView.dom.querySelector<HTMLElement>(
`.collaboration-carets__caret[data-caret-client-id="${id}"]`
)
if (caret) activateCaretLabel(caret, editorRight)
}
pending.clear()
}
const onChange = ({ added, updated }: { added: number[]; updated: number[] }) => {
for (const id of added) pending.add(id)
for (const id of updated) pending.add(id)
if (pending.size > 0 && !raf) raf = requestAnimationFrame(flush)
}
awareness.on('change', onChange)
return {
destroy: () => {
awareness.off('change', onChange)
if (raf) cancelAnimationFrame(raf)
// Clear any pending fade timers for this editor's carets so they don't fire on
// detached nodes after unmount (harmless no-op, but a genuine leaked timer).
for (const caret of editorView.dom.querySelectorAll<HTMLElement>(
'.collaboration-carets__caret'
)) {
const timer = caretFadeTimers.get(caret)
if (timer) {
clearTimeout(timer)
caretFadeTimers.delete(caret)
}
}
},
}
},
}),
]
},
})
}
@@ -0,0 +1,14 @@
'use client'
import { PresenceAvatars } from '@/app/workspace/[workspaceId]/components/presence/presence-avatars'
import { useFileDocOthers } from './file-doc-room-context'
/**
* Avatar stack of the collaborators currently in the open file the `useOthers` avatar
* stack, reading the room roster from {@link useFileDocOthers}. Renders nothing until
* someone else joins. Must sit inside a `FileDocRoomProvider`.
*/
export function FileDocAvatars() {
const others = useFileDocOthers()
return <PresenceAvatars users={others} className='mr-1' />
}
@@ -0,0 +1,431 @@
/**
* @vitest-environment node
*/
import {
FILE_DOC_EVENTS,
FILE_DOC_MESSAGE_TYPE,
FILE_DOC_SEED,
} from '@sim/realtime-protocol/file-doc'
import * as encoding from 'lib0/encoding'
import type { Socket } from 'socket.io-client'
import { describe, expect, it, vi } from 'vitest'
import * as awarenessProtocol from 'y-protocols/awareness'
import * as syncProtocol from 'y-protocols/sync'
import * as Y from 'yjs'
import { AGENT_STREAM_ORIGIN } from './apply-streamed-markdown'
import { FileDocProvider } from './file-doc-provider'
/** A minimal fake Socket.IO client whose server→client events can be fired in tests. */
function createSocket(connected = true) {
const listeners = new Map<string, Set<(...args: unknown[]) => void>>()
const emit = vi.fn()
const socket = {
connected,
emit,
on(event: string, cb: (...args: unknown[]) => void) {
let set = listeners.get(event)
if (!set) {
set = new Set()
listeners.set(event, set)
}
set.add(cb)
},
off(event: string, cb: (...args: unknown[]) => void) {
listeners.get(event)?.delete(cb)
},
}
const fire = (event: string, ...args: unknown[]) => {
for (const cb of listeners.get(event) ?? []) cb(...args)
}
return { socket: socket as unknown as Socket, emit, fire }
}
function createProvider(connected = true) {
const { socket, emit, fire } = createSocket(connected)
const doc = new Y.Doc()
const awareness = new awarenessProtocol.Awareness(doc)
const provider = new FileDocProvider(socket, 'file-1', doc, awareness)
return { provider, doc, awareness, emit, fire }
}
/** Messages emitted to the server, decoded to their `{ type, bytes }`. */
function emittedMessages(emit: ReturnType<typeof vi.fn>) {
return emit.mock.calls
.filter(
([event, payload]) => event === FILE_DOC_EVENTS.MESSAGE && payload instanceof Uint8Array
)
.map(([, payload]) => payload as Uint8Array)
}
describe('FileDocProvider', () => {
it('joins immediately with its client id when the socket is already connected', () => {
const { doc, emit } = createProvider(true)
expect(emit).toHaveBeenCalledWith(FILE_DOC_EVENTS.JOIN, {
fileId: 'file-1',
clientId: doc.clientID,
})
})
it('waits for connect before joining when the socket is offline', () => {
const { emit, fire } = createProvider(false)
expect(emit).not.toHaveBeenCalledWith(FILE_DOC_EVENTS.JOIN, expect.anything())
fire('connect')
expect(emit).toHaveBeenCalledWith(
FILE_DOC_EVENTS.JOIN,
expect.objectContaining({ fileId: 'file-1' })
)
})
it('exchanges sync only after JOIN_SUCCESS', () => {
const { emit, fire } = createProvider(true)
emit.mockClear()
fire(FILE_DOC_EVENTS.JOIN_SUCCESS, { fileId: 'file-1' })
// A sync step 1 (type tag 0) is sent to exchange state with the server.
const messages = emittedMessages(emit)
expect(messages.length).toBeGreaterThan(0)
expect(messages[0][0]).toBe(FILE_DOC_MESSAGE_TYPE.SYNC)
})
it('ignores a join ack for a different file', () => {
const { emit, fire } = createProvider(true)
emit.mockClear()
fire(FILE_DOC_EVENTS.JOIN_SUCCESS, { fileId: 'other-file' })
// No sync/awareness exchange starts for a file this provider does not own.
expect(emittedMessages(emit)).toHaveLength(0)
})
it('applies a server sync step 2 and becomes synced', () => {
const { provider, doc, fire } = createProvider(true)
const synced = vi.fn()
provider.on('synced', synced)
const serverDoc = new Y.Doc()
serverDoc.getText('default').insert(0, 'hello world')
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.SYNC)
syncProtocol.writeSyncStep2(encoder, serverDoc)
fire(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
expect(doc.getText('default').toString()).toBe('hello world')
expect(provider.synced).toBe(true)
expect(synced).toHaveBeenCalledWith(true)
})
it('sends local document edits to the server as sync updates', () => {
const { doc, emit } = createProvider(true)
emit.mockClear()
doc.getText('default').insert(0, 'x')
const messages = emittedMessages(emit)
expect(messages.length).toBe(1)
expect(messages[0][0]).toBe(FILE_DOC_MESSAGE_TYPE.SYNC)
})
it('tags agent-streamed edits as SYNC_NO_PERSIST so the relay skips the durable persist', () => {
const { doc, emit } = createProvider(true)
emit.mockClear()
// An agent-streamed frame is applied under AGENT_STREAM_ORIGIN; it must still reach the server (peers
// see it live) but as SYNC_NO_PERSIST, so the relay fans it out without treating it as a user edit.
doc.transact(() => doc.getText('default').insert(0, 'agent'), AGENT_STREAM_ORIGIN)
const messages = emittedMessages(emit)
expect(messages.length).toBe(1)
expect(messages[0][0]).toBe(FILE_DOC_MESSAGE_TYPE.SYNC_NO_PERSIST)
})
it('does not echo updates it applied from the server', () => {
const { provider, emit, fire } = createProvider(true)
fire(FILE_DOC_EVENTS.JOIN_SUCCESS, { fileId: 'file-1' })
emit.mockClear()
const serverDoc = new Y.Doc()
serverDoc.getText('default').insert(0, 'remote')
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.SYNC)
syncProtocol.writeUpdate(encoder, Y.encodeStateAsUpdate(serverDoc))
fire(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
// The applied remote update must not be re-emitted back to the server.
expect(emittedMessages(emit)).toHaveLength(0)
expect(provider.doc.getText('default').toString()).toBe('remote')
})
it('sends local awareness (cursor/selection) changes', () => {
const { awareness, emit } = createProvider(true)
emit.mockClear()
awareness.setLocalStateField('user', { name: 'Ada', color: '#f783ac' })
const messages = emittedMessages(emit)
expect(messages.some((m) => m[0] === FILE_DOC_MESSAGE_TYPE.AWARENESS)).toBe(true)
})
it('reseeds a cleared awareness so a reused instance can publish again', () => {
const { socket, emit } = createSocket(true)
const doc = new Y.Doc()
const awareness = new awarenessProtocol.Awareness(doc)
// Simulate a prior provider teardown having cleared the local state — after
// this, y-protocols' setLocalStateField is a permanent no-op, so the caret
// extension could never publish the local user/cursor on a reused instance.
awarenessProtocol.removeAwarenessStates(awareness, [doc.clientID], 'prior-destroy')
expect(awareness.getLocalState()).toBeNull()
// Constructing a provider on the reused, cleared awareness must restore it.
new FileDocProvider(socket, 'file-1', doc, awareness)
expect(awareness.getLocalState()).not.toBeNull()
emit.mockClear()
// The caret extension setting the user field must now actually publish.
awareness.setLocalStateField('user', { name: 'Ada', color: '#f783ac' })
expect(emittedMessages(emit).some((m) => m[0] === FILE_DOC_MESSAGE_TYPE.AWARENESS)).toBe(true)
})
it('does not forward awareness it applied from the server', () => {
const { emit, fire } = createProvider(true)
emit.mockClear()
const remoteDoc = new Y.Doc()
remoteDoc.clientID = 8888
const remoteAwareness = new awarenessProtocol.Awareness(remoteDoc)
remoteAwareness.setLocalStateField('user', { name: 'Remote' })
const update = awarenessProtocol.encodeAwarenessUpdate(remoteAwareness, [8888])
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.AWARENESS)
encoding.writeVarUint8Array(encoder, update)
fire(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
// The remote peer's awareness (client 8888, not ours) must not be re-published.
expect(emittedMessages(emit).some((m) => m[0] === FILE_DOC_MESSAGE_TYPE.AWARENESS)).toBe(false)
})
it('stops attempting to join and latches joinError after a non-retryable error', () => {
const { provider, emit, fire } = createProvider(true)
emit.mockClear()
const error = {
fileId: 'file-1',
error: 'Access denied',
code: 'ACCESS_DENIED',
retryable: false,
}
fire(FILE_DOC_EVENTS.JOIN_ERROR, error)
fire('connect')
expect(emit).not.toHaveBeenCalledWith(FILE_DOC_EVENTS.JOIN, expect.anything())
// Latched so a consumer subscribing after the event can still detect the failure.
expect(provider.joinError).toEqual(error)
})
it('still rejoins on reconnect after a retryable error', () => {
const { emit, fire } = createProvider(true)
fire(FILE_DOC_EVENTS.JOIN_ERROR, {
fileId: 'file-1',
error: 'Realtime unavailable',
code: 'ROOM_MANAGER_UNAVAILABLE',
retryable: true,
})
emit.mockClear()
fire('connect')
expect(emit).toHaveBeenCalledWith(
FILE_DOC_EVENTS.JOIN,
expect.objectContaining({ fileId: 'file-1' })
)
})
it('resets synced and rejoins on a reconnect', () => {
const { provider, emit, fire } = createProvider(true)
// Become synced.
const serverDoc = new Y.Doc()
serverDoc.getText('default').insert(0, 'hi')
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.SYNC)
syncProtocol.writeSyncStep2(encoder, serverDoc)
fire(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
expect(provider.synced).toBe(true)
emit.mockClear()
// A reconnect must drop synced and re-issue JOIN so the doc re-syncs.
fire('connect')
expect(provider.synced).toBe(false)
expect(emit).toHaveBeenCalledWith(
FILE_DOC_EVENTS.JOIN,
expect.objectContaining({ fileId: 'file-1' })
)
})
it('leaves the room and detaches on destroy', () => {
const { provider, doc, emit } = createProvider(true)
emit.mockClear()
provider.destroy()
expect(emit).toHaveBeenCalledWith(FILE_DOC_EVENTS.LEAVE, { fileId: 'file-1' })
// After destroy, local edits are no longer forwarded.
emit.mockClear()
doc.getText('default').insert(0, 'y')
expect(emittedMessages(emit)).toHaveLength(0)
})
it('leaves the room only when the LAST provider for a file on a shared socket is destroyed', () => {
// Two surfaces in one tab (Files editor + embedded chat panel) share one socket and both open the
// same file. Tearing the first down must NOT strand the second — the server drops the socket from
// the room on any LEAVE, so LEAVE may fire only when the last provider goes away.
const { socket, emit } = createSocket(true)
const docA = new Y.Doc()
const docB = new Y.Doc()
const first = new FileDocProvider(
socket,
'shared-file',
docA,
new awarenessProtocol.Awareness(docA)
)
const second = new FileDocProvider(
socket,
'shared-file',
docB,
new awarenessProtocol.Awareness(docB)
)
emit.mockClear()
first.destroy()
expect(emit).not.toHaveBeenCalledWith(FILE_DOC_EVENTS.LEAVE, expect.anything())
second.destroy()
expect(emit).toHaveBeenCalledWith(FILE_DOC_EVENTS.LEAVE, { fileId: 'shared-file' })
})
it('scopes the shared-membership refcount per file (a sibling file leaves independently)', () => {
const { socket, emit } = createSocket(true)
const docA = new Y.Doc()
const docB = new Y.Doc()
const fileA = new FileDocProvider(socket, 'file-a', docA, new awarenessProtocol.Awareness(docA))
const fileB = new FileDocProvider(socket, 'file-b', docB, new awarenessProtocol.Awareness(docB))
emit.mockClear()
fileA.destroy()
// A different file's sole provider still leaves immediately.
expect(emit).toHaveBeenCalledWith(FILE_DOC_EVENTS.LEAVE, { fileId: 'file-a' })
expect(emit).not.toHaveBeenCalledWith(FILE_DOC_EVENTS.LEAVE, { fileId: 'file-b' })
fileB.destroy()
expect(emit).toHaveBeenCalledWith(FILE_DOC_EVENTS.LEAVE, { fileId: 'file-b' })
})
it('gives up with a non-retryable join-error when the first sync never arrives (offline)', () => {
vi.useFakeTimers()
try {
const { provider, emit, fire } = createProvider(false) // socket never connects
const onError = vi.fn()
provider.on('join-error', onError)
vi.advanceTimersByTime(12_000)
// Surfaces the same non-retryable rejection the fatal path uses, so the editor falls back to
// showing the file read-only.
expect(onError).toHaveBeenCalledWith(
expect.objectContaining({ code: 'READINESS_TIMEOUT', retryable: false })
)
expect(provider.joinError).toEqual(
expect.objectContaining({ code: 'READINESS_TIMEOUT', retryable: false })
)
// Latched fatal: a later connect must not re-join (which could sync server state in and
// duplicate the locally-seeded content).
emit.mockClear()
fire('connect')
expect(emit).not.toHaveBeenCalledWith(FILE_DOC_EVENTS.JOIN, expect.anything())
} finally {
vi.useRealTimers()
}
})
it('does not fire the fallback once the doc is synced AND seeded', () => {
vi.useFakeTimers()
try {
const { provider, fire } = createProvider(true)
const onError = vi.fn()
provider.on('join-error', onError)
// The initial sync brings BOTH content and the server seed flag before the deadline.
fire(FILE_DOC_EVENTS.JOIN_SUCCESS, { fileId: 'file-1' })
const remote = new Y.Doc()
remote.getText('default').insert(0, 'hi')
remote.getMap(FILE_DOC_SEED.configMap).set(FILE_DOC_SEED.flag, true)
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.SYNC)
syncProtocol.writeSyncStep2(encoder, remote)
fire(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
expect(provider.synced).toBe(true)
vi.advanceTimersByTime(12_000)
expect(onError).not.toHaveBeenCalled()
} finally {
vi.useRealTimers()
}
})
it('fires the fallback when the doc synced but the server seed never landed', () => {
vi.useFakeTimers()
try {
const { provider, fire } = createProvider(true)
const onError = vi.fn()
provider.on('join-error', onError)
// The socket syncs an empty doc, but the server-side seed never arrives (its build persistently
// failed) — `synced` is true yet `initialContentLoaded` is never set.
fire(FILE_DOC_EVENTS.JOIN_SUCCESS, { fileId: 'file-1' })
const remote = new Y.Doc()
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.SYNC)
syncProtocol.writeSyncStep2(encoder, remote)
fire(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
expect(provider.synced).toBe(true)
vi.advanceTimersByTime(12_000)
// The readiness deadline still fires → the editor falls back to the stored content read-only,
// and `synced` is dropped so the `synced && seeded` gate stays closed (read-only, not editable).
expect(onError).toHaveBeenCalledWith(
expect.objectContaining({ code: 'READINESS_TIMEOUT', retryable: false })
)
expect(provider.synced).toBe(false)
} finally {
vi.useRealTimers()
}
})
it('ignores a late SyncStep2 that arrives after the readiness deadline (no merge, stays gated)', () => {
vi.useFakeTimers()
try {
const { provider, doc, fire } = createProvider(true)
fire(FILE_DOC_EVENTS.JOIN_SUCCESS, { fileId: 'file-1' })
// Deadline lapses with no first sync → fatal fallback (editor falls back to a read-only seed).
vi.advanceTimersByTime(12_000)
expect(provider.joinError).toEqual(expect.objectContaining({ code: 'READINESS_TIMEOUT' }))
// A delayed SyncStep2 finally arrives. Applying it would merge server content into the
// already-seeded doc (duplication) and flip synced→true (un-gating autosave), so it MUST be
// dropped once fatal.
const remote = new Y.Doc()
remote.getText('default').insert(0, 'server content')
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.SYNC)
syncProtocol.writeSyncStep2(encoder, remote)
fire(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
expect(provider.synced).toBe(false)
expect(doc.getText('default').toString()).toBe('')
} finally {
vi.useRealTimers()
}
})
})
@@ -0,0 +1,372 @@
import {
FILE_DOC_EVENTS,
FILE_DOC_MESSAGE_TYPE,
FILE_DOC_SEED,
FILE_DOC_TIMEOUTS,
type JoinFileDocError,
type JoinFileDocSuccess,
toFileDocBytes,
} from '@sim/realtime-protocol/file-doc'
import * as decoding from 'lib0/decoding'
import * as encoding from 'lib0/encoding'
import { ObservableV2 } from 'lib0/observable'
import type { Socket } from 'socket.io-client'
import * as awarenessProtocol from 'y-protocols/awareness'
import * as syncProtocol from 'y-protocols/sync'
import type * as Y from 'yjs'
import { AGENT_STREAM_ORIGIN } from './apply-streamed-markdown'
/**
* Events emitted by {@link FileDocProvider}.
* - `synced`: the first full document sync with the server completed.
* - `join-error`: the server rejected the join (e.g. lost write access).
*/
interface FileDocProviderEvents {
synced: (synced: boolean) => void
'join-error': (error: JoinFileDocError) => void
}
/**
* How long to wait to reach a USABLE editor connected, synced, AND seeded (`initialContentLoaded`
* set by the server seed) before giving up. It guards two failure modes with one timer:
* - the realtime server is unreachable, so the first sync never arrives; and
* - the socket syncs an empty doc but the server-side seed never lands (its build persistently fails
* / exhausts its retries), which `synced` alone would wrongly treat as "connected, all good".
*
* On the deadline the provider latches fatal and surfaces a non-retryable `join-error` the exact
* path a fatal rejection uses so the editor falls back to showing the file's stored content
* read-only instead of a permanently blank pane. Generous enough to clear a slow connect + seed
* round-trip; a healthy cold open reaches readiness well within it. Shared with (and must exceed) the
* relay's seed-fetch timeout see `FILE_DOC_TIMEOUTS` and its ordering test.
*/
const READINESS_DEADLINE_MS = FILE_DOC_TIMEOUTS.readinessDeadlineMs
/**
* Live-provider counts per file, per shared socket. Two surfaces in one tab (the Files editor and the
* embedded chat resource panel) share ONE Socket.IO connection, so both a first and a second provider
* for the same file JOIN the same room over that socket. The server's `leave(name)` drops the socket
* from the room outright no membership refcount so the FIRST provider's `destroy()` would strand
* the second (still-mounted) one: no more content or presence updates. Keyed by the {@link Socket}
* OBJECT (stable across reconnects, unlike `socket.id`), so the count survives a reconnect.
*
* The single-provider case is unchanged: the count goes `0 → 1 → 0` and `LEAVE` fires exactly as
* before. `LEAVE` is emitted only when the LAST provider for a file on a socket tears down.
*/
const roomJoinCounts = new WeakMap<Socket, Map<string, number>>()
/** Record another live provider for `fileId` on `socket` (called at construction). */
function retainRoomMembership(socket: Socket, fileId: string): void {
let counts = roomJoinCounts.get(socket)
if (!counts) {
counts = new Map()
roomJoinCounts.set(socket, counts)
}
counts.set(fileId, (counts.get(fileId) ?? 0) + 1)
}
/**
* Drop one live provider for `fileId` on `socket` (called at teardown). Returns `true` when this was
* the last one i.e. the caller should emit `LEAVE` so the socket leaves the room.
*/
function releaseRoomMembership(socket: Socket, fileId: string): boolean {
const counts = roomJoinCounts.get(socket)
const next = (counts?.get(fileId) ?? 1) - 1
if (next > 0) {
counts?.set(fileId, next)
return false
}
counts?.delete(fileId)
return true
}
/**
* The client half of the collaborative file-document protocol: a Yjs provider
* that carries document sync + awareness over the shared, already-authenticated
* Socket.IO connection (the server relay lives in
* `apps/realtime/src/handlers/file-doc.ts`). It is the Socket.IO analogue of
* `y-websocket`'s `WebsocketProvider` the same `y-protocols` message framing
* so TipTap's `Collaboration` (bound to {@link doc}) and `CollaborationCaret`
* (bound to this provider's {@link awareness}) work unmodified.
*
* The document and awareness are owned by the caller (the hook) and are NOT
* destroyed here, so the provider can be torn down and rebuilt (e.g. on a socket
* reconnect) without discarding local edits.
*/
export class FileDocProvider extends ObservableV2<FileDocProviderEvents> {
synced = false
/**
* The latched non-retryable join rejection, or `null`. The `join-error` event is
* transient and can fire before a consumer subscribes,
* so consumers read this on subscription to detect a fatal failure they missed.
*/
joinError: JoinFileDocError | null = null
private disposed = false
/** Set on a non-retryable join rejection (e.g. lost write access) so the
* provider stops attempting to (re)join until the owner tears it down. */
private fatal = false
/** Deadline for reaching readiness (synced + seeded); fires the fallback if it is never reached. */
private readinessTimer: ReturnType<typeof setTimeout> | null = null
constructor(
private readonly socket: Socket,
private readonly fileId: string,
readonly doc: Y.Doc,
readonly awareness: awarenessProtocol.Awareness
) {
super()
// Restore an empty local awareness state if it has been cleared. A fresh
// Awareness starts with `{}`, but a *reused* one whose local state was removed
// (a prior provider's `destroy()` clears it, and so does `Awareness.destroy()`)
// returns `null` here — and y-protocols' `setLocalStateField` is a no-op while
// the local state is `null`. The editor binds CollaborationCaret to this exact
// awareness for its whole life, so without this reseed a remount (e.g. React
// StrictMode's mount→unmount→mount, which re-runs the provider effect on the
// same instance) would leave the caret extension unable to ever publish the
// local user/cursor — remote peers would see no caret or selection, even though
// document sync (which does not depend on local awareness) keeps working.
if (awareness.getLocalState() === null) awareness.setLocalState({})
socket.on(FILE_DOC_EVENTS.MESSAGE, this.handleMessage)
socket.on(FILE_DOC_EVENTS.JOIN_SUCCESS, this.handleJoinSuccess)
socket.on(FILE_DOC_EVENTS.JOIN_ERROR, this.handleJoinError)
socket.on('connect', this.handleConnect)
doc.on('update', this.handleDocUpdate)
awareness.on('update', this.handleAwarenessUpdate)
// Watch the seed flag so reaching "seeded" (server seed applied) can clear the readiness deadline.
doc.getMap(FILE_DOC_SEED.configMap).observe(this.handleConfigChange)
// Count this provider against the shared socket's membership of the file's room, so the room is
// left only when the last provider for this file tears down (see {@link releaseRoomMembership}).
retainRoomMembership(socket, fileId)
if (socket.connected) this.join()
// Arm the fallback: if we don't reach readiness (synced + seeded) before the deadline, give up.
this.readinessTimer = setTimeout(this.handleReadinessDeadline, READINESS_DEADLINE_MS)
}
/** Whether the server seed has recorded the initial content on the doc. */
private isSeeded(): boolean {
return this.doc.getMap(FILE_DOC_SEED.configMap).get(FILE_DOC_SEED.flag) === true
}
/** Clear the readiness deadline once the editor is usable (synced AND seeded). */
private handleConfigChange = () => {
if (this.synced && this.isSeeded()) this.clearReadinessTimer()
}
/**
* Readiness was never reached within {@link READINESS_DEADLINE_MS} either the realtime server is
* unreachable (never synced) or it synced but the server-side seed never landed (synced yet
* unseeded). Reset `synced` (so the editor gates read-only), latch fatal (so a late reconnect or
* seed can't sync server state in and merge-duplicate the content the editor is about to render
* locally), and surface a synthetic non-retryable join-error the exact path a fatal rejection
* uses so the owner falls back to the read-only view of the file's stored content instead of a
* blank pane. No-op if we already reached readiness, already failed fatally, or were torn down.
*/
private handleReadinessDeadline = () => {
this.readinessTimer = null
if ((this.synced && this.isSeeded()) || this.fatal || this.disposed) return
const error: JoinFileDocError = {
fileId: this.fileId,
error: 'Realtime document was not ready in time',
code: 'READINESS_TIMEOUT',
retryable: false,
}
this.fatal = true
this.joinError = error
// Drop `synced` so the editor's `synced && seeded` gate stays closed → the fallback renders the
// stored content read-only rather than becoming editable on a doc the server never seeded.
this.setSynced(false)
this.emit('join-error', [error])
}
private clearReadinessTimer() {
if (this.readinessTimer !== null) {
clearTimeout(this.readinessTimer)
this.readinessTimer = null
}
}
/** Join the room, binding our client id so the server only accepts awareness we own. */
private join = () => {
if (this.fatal) return
this.socket.emit(FILE_DOC_EVENTS.JOIN, { fileId: this.fileId, clientId: this.doc.clientID })
}
/**
* Re-join after a (re)connect. The server re-registers the room before acking,
* so the sync/awareness exchange is deferred to {@link handleJoinSuccess}.
*/
private handleConnect = () => {
if (this.fatal) return
this.setSynced(false)
this.join()
}
/**
* Handle the join ack. The server registers the room before acking, so an earlier
* send could be dropped the initial sync + local awareness exchange begins here.
*/
private handleJoinSuccess = (data: JoinFileDocSuccess) => {
if (data.fileId !== this.fileId) return
this.sendSyncStep1()
this.sendLocalAwareness()
}
/**
* Handle a join rejection. A non-retryable rejection (access denied, invalid)
* won't succeed on retry, so latch {@link fatal} to stop (re)joining and let the
* owner fall back to the non-collaborative view.
*/
private handleJoinError = (data: JoinFileDocError) => {
if (data.fileId !== this.fileId) return
if (data.retryable === false) {
this.fatal = true
this.joinError = data
this.clearReadinessTimer()
}
this.emit('join-error', [data])
}
private handleMessage = (data: unknown) => {
// Once we've given up (a non-retryable rejection, or the connect deadline lapsed and the editor
// fell back to a read-only local seed), ignore ALL inbound frames. A late SyncStep2 arriving
// after the deadline would otherwise merge the server's state into the already-seeded doc —
// duplicating content — and flip `synced` true, which un-gates autosave and would persist the
// duplicate back to the real file. `fatal` guarding (re)join alone is not enough; it must also
// stop applying sync here.
if (this.fatal) return
const bytes = toFileDocBytes(data)
if (!bytes) return
const decoder = decoding.createDecoder(bytes)
const messageType = decoding.readVarUint(decoder)
switch (messageType) {
case FILE_DOC_MESSAGE_TYPE.SYNC: {
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.SYNC)
// `this` is the transaction origin, so our own `doc.on('update')` skips
// re-sending updates we just applied from the server.
const syncType = syncProtocol.readSyncMessage(decoder, encoder, this.doc, this)
if (encoding.length(encoder) > 1) {
this.socket.emit(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
}
if (syncType === syncProtocol.messageYjsSyncStep2 && !this.synced) this.setSynced(true)
break
}
case FILE_DOC_MESSAGE_TYPE.AWARENESS: {
awarenessProtocol.applyAwarenessUpdate(
this.awareness,
decoding.readVarUint8Array(decoder),
this
)
break
}
}
}
private handleDocUpdate = (update: Uint8Array, origin: unknown) => {
// Once fatal (a non-retryable rejection, or the readiness deadline lapsed), the editor may render
// the stored content into the doc locally as its read-only fallback. Never relay those local
// writes — the server never seeded this doc, so echoing them would push unseeded content to peers
// (and each fallen-back client would do so, union-duplicating). A fatal client is fully local.
if (this.fatal) return
// Updates we applied from the server carry `this` as origin — don't echo them.
if (origin === this) return
// Agent-streamed frames must reach peers (so a collaborator sees the stream live) but must NOT be
// treated by the server as a durable user edit — the copilot's final `edit_content` write is the
// authoritative persist. Tag them so the relay applies + fans out but skips persist bookkeeping.
const messageType =
origin === AGENT_STREAM_ORIGIN
? FILE_DOC_MESSAGE_TYPE.SYNC_NO_PERSIST
: FILE_DOC_MESSAGE_TYPE.SYNC
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, messageType)
syncProtocol.writeUpdate(encoder, update)
this.socket.emit(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
}
private handleAwarenessUpdate = (
{ added, updated, removed }: { added: number[]; updated: number[]; removed: number[] },
origin: unknown
) => {
// Only ever publish OUR OWN awareness. Remote changes (origin === this) were
// applied from the server; and a local `Awareness` also emits 30s `timeout`
// removals for remote peers — forwarding either would be a frame for a client
// id we don't own, which the server (correctly) rejects. Filter to our own id
// so honest traffic never trips the ownership guard.
if (origin === this) return
const localId = this.doc.clientID
const changed = [...added, ...updated, ...removed].filter((id) => id === localId)
if (changed.length === 0) return
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.AWARENESS)
encoding.writeVarUint8Array(
encoder,
awarenessProtocol.encodeAwarenessUpdate(this.awareness, changed)
)
this.socket.emit(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
}
private sendSyncStep1() {
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.SYNC)
syncProtocol.writeSyncStep1(encoder, this.doc)
this.socket.emit(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
}
private sendLocalAwareness() {
if (this.awareness.getLocalState() === null) return
const encoder = encoding.createEncoder()
encoding.writeVarUint(encoder, FILE_DOC_MESSAGE_TYPE.AWARENESS)
encoding.writeVarUint8Array(
encoder,
awarenessProtocol.encodeAwarenessUpdate(this.awareness, [this.doc.clientID])
)
this.socket.emit(FILE_DOC_EVENTS.MESSAGE, encoding.toUint8Array(encoder))
}
private setSynced(synced: boolean) {
if (this.synced === synced) return
this.synced = synced
// Readiness needs synced AND seeded; only clear the deadline when both hold (the seed may have
// arrived first, or may still be pending — `handleConfigChange` clears it if seeded arrives later).
if (synced && this.isSeeded()) this.clearReadinessTimer()
this.emit('synced', [synced])
}
/**
* Tear down the provider: leave the room, clear our awareness (so peers drop our
* caret immediately rather than after the server's 30s timeout), and detach all
* listeners. The document and awareness objects are the caller's and are left intact.
*/
destroy() {
if (this.disposed) {
super.destroy()
return
}
this.disposed = true
this.clearReadinessTimer()
awarenessProtocol.removeAwarenessStates(this.awareness, [this.doc.clientID], 'provider-destroy')
// Only actually leave the room when this was the last provider for the file on the shared socket —
// otherwise a sibling surface (e.g. the Files editor vs. the embedded chat panel) would be stranded.
if (releaseRoomMembership(this.socket, this.fileId)) {
this.socket.emit(FILE_DOC_EVENTS.LEAVE, { fileId: this.fileId })
}
this.socket.off(FILE_DOC_EVENTS.MESSAGE, this.handleMessage)
this.socket.off(FILE_DOC_EVENTS.JOIN_SUCCESS, this.handleJoinSuccess)
this.socket.off(FILE_DOC_EVENTS.JOIN_ERROR, this.handleJoinError)
this.socket.off('connect', this.handleConnect)
this.doc.off('update', this.handleDocUpdate)
this.doc.getMap(FILE_DOC_SEED.configMap).unobserve(this.handleConfigChange)
this.awareness.off('update', this.handleAwarenessUpdate)
super.destroy()
}
}
@@ -0,0 +1,43 @@
'use client'
import { createContext, type ReactNode, useContext, useState } from 'react'
import type { PresenceAvatarUser } from '@/app/workspace/[workspaceId]/components/presence/presence-avatars'
const EMPTY_OTHERS: PresenceAvatarUser[] = []
const noop = () => {}
// Split into two contexts on purpose: the roster (`others`) changes on every join/leave,
// but the setter is stable. The editor (which owns the awareness) only ever *reports* the
// roster, so it subscribes to the setter context — which never changes identity — and never
// re-renders when the roster does; only the header avatar stack subscribes to `others`.
const FileDocOthersContext = createContext<PresenceAvatarUser[]>(EMPTY_OTHERS)
const FileDocSetOthersContext = createContext<(users: PresenceAvatarUser[]) => void>(noop)
/**
* Scopes "who's in this file" presence to the open document the `RoomProvider` +
* `useOthers` pattern (Liveblocks / y-presence) adapted to our component tree. The editor
* owns the Yjs awareness but sits *below* the file-detail header that renders the avatar
* stack, so it publishes the SERVER-AUTHENTICATED roster into this context
* ({@link useReportFileDocOthers}) and the header reads it ({@link useFileDocOthers}).
* Presence is ephemeral and room-scoped, so it lives in this provider, not a global store.
*/
export function FileDocRoomProvider({ children }: { children: ReactNode }) {
const [others, setOthers] = useState<PresenceAvatarUser[]>(EMPTY_OTHERS)
return (
<FileDocSetOthersContext.Provider value={setOthers}>
<FileDocOthersContext.Provider value={others}>{children}</FileDocOthersContext.Provider>
</FileDocSetOthersContext.Provider>
)
}
/** The roster of collaborators currently in the open file, for an avatar stack. Empty
* outside a {@link FileDocRoomProvider}. */
export function useFileDocOthers(): PresenceAvatarUser[] {
return useContext(FileDocOthersContext)
}
/** Publishes the server roster into the room context (editor side). Returns a stable no-op
* outside a {@link FileDocRoomProvider}. */
export function useReportFileDocOthers(): (users: PresenceAvatarUser[]) => void {
return useContext(FileDocSetOthersContext)
}
@@ -0,0 +1,77 @@
/**
* @vitest-environment node
*/
import { describe, expect, it } from 'vitest'
import { type CollabReadinessInputs, nextCollabReadiness } from './readiness'
/** Drive a sequence of observations through the latch, returning the readiness at each step. */
function run(steps: CollabReadinessInputs[]): boolean[] {
let syncedOnce = false
return steps.map((input) => {
const next = nextCollabReadiness(syncedOnce, input)
syncedOnce = next.syncedOnce
return next.ready
})
}
describe('nextCollabReadiness', () => {
it('is not ready before syncing or seeding', () => {
const { syncedOnce, ready } = nextCollabReadiness(false, {
synced: false,
seeded: false,
offlineSeed: false,
})
expect(syncedOnce).toBe(false)
expect(ready).toBe(false)
})
it('is not ready when synced but not yet seeded', () => {
const { syncedOnce, ready } = nextCollabReadiness(false, {
synced: true,
seeded: false,
offlineSeed: false,
})
expect(syncedOnce).toBe(true) // latched
expect(ready).toBe(false) // waits for the seed
})
it('opens on the new-file flap sequence: synced true, then seed lands while synced flapped false', () => {
// The exact bug: `synced` and `seeded` are never true in the same observation. The latch must still
// open once BOTH have been seen across observations.
const readiness = run([
{ synced: false, seeded: false, offlineSeed: false }, // joining
{ synced: true, seeded: false, offlineSeed: false }, // initial (empty) sync
{ synced: false, seeded: false, offlineSeed: false }, // synced flaps false on re-sync
{ synced: false, seeded: true, offlineSeed: false }, // server seed lands (synced still false)
])
expect(readiness).toEqual([false, false, false, true])
})
it('opens even if the seed lands before we ever observed synced (server seed proves a sync)', () => {
// If the flap beat our first observation, the seed flag alone (not the offline fallback) proves a
// completed sync happened.
const { syncedOnce, ready } = nextCollabReadiness(false, {
synced: false,
seeded: true,
offlineSeed: false,
})
expect(syncedOnce).toBe(true)
expect(ready).toBe(true)
})
it('stays read-only for an offline (local) seed that never reached the server', () => {
const readiness = run([
{ synced: false, seeded: false, offlineSeed: false },
{ synced: false, seeded: true, offlineSeed: true }, // offline fallback seeded locally
])
expect(readiness).toEqual([false, false])
})
it('never reverts once ready, even if synced later flaps false', () => {
const readiness = run([
{ synced: true, seeded: true, offlineSeed: false }, // ready
{ synced: false, seeded: true, offlineSeed: false }, // synced flaps — must stay ready
])
expect(readiness).toEqual([true, true])
})
})
@@ -0,0 +1,37 @@
/**
* Collaborative-readiness latch for the file editor.
*
* A file becomes "ready" (editable + agent streaming enabled) once its shared doc has both SYNCED and
* SEEDED. The subtlety this latch solves: a brand-new file's provider reports `synced: true` on the
* initial (empty) sync, then the SERVER pushes the seed and receiving that update flips `synced` back to
* `false` so `synced` and `seeded` are never `true` in the same observation. An un-latched
* `synced && seeded` gate would therefore never open, and agent streaming would be dropped for the whole
* run (the file only fills in on reload).
*
* The latch: `syncedOnce` is sticky set the first time a completed sync is observed, and it never
* reverts. A completed sync is proven by EITHER a live `synced`, OR the seed flag being present without
* the offline fallback having set it (`offlineSeed`) because a SERVER seed can only arrive after a
* sync, whereas the offline fallback seeds locally without ever reaching the server and must stay
* read-only. Once `syncedOnce` is set, a later `synced` flap can no longer re-gate the doc.
*/
export interface CollabReadinessInputs {
/** The provider's current `synced` flag (may flap false after the seed update). */
synced: boolean
/** Whether the shared doc carries the seed flag. */
seeded: boolean
/** Whether the seed flag was set by the offline fallback (no server sync) rather than the server. */
offlineSeed: boolean
}
/**
* Pure transition for the readiness latch. `syncedOnce` is the sticky prior state pass the returned
* `syncedOnce` back in on the next call. `ready` is whether the doc is synced-and-seeded.
*/
export function nextCollabReadiness(
syncedOnce: boolean,
input: CollabReadinessInputs
): { syncedOnce: boolean; ready: boolean } {
const next = syncedOnce || input.synced || (input.seeded && !input.offlineSeed)
return { syncedOnce: next, ready: next && input.seeded }
}
@@ -0,0 +1,172 @@
'use client'
import { useEffect, useMemo, useRef, useState } from 'react'
import { FILE_DOC_EVENTS, type FileDocPresence } from '@sim/realtime-protocol/file-doc'
import { Awareness } from 'y-protocols/awareness'
import * as Y from 'yjs'
import { getUserColor } from '@/lib/workspaces/colors'
import { useSocket } from '@/app/workspace/providers/socket-provider'
import { FileDocProvider } from './file-doc-provider'
import { useReportFileDocOthers } from './file-doc-room-context'
/** The live collaboration binding the editor wires into TipTap's Collaboration
* (the {@link Y.Doc}) and CollaborationCaret (the awareness). */
export interface FileDocCollaboration {
/** Bound to TipTap's Collaboration extension (created synchronously at mount). */
doc: Y.Doc
/** Bound to CollaborationCaret (via `{ awareness }`); relayed by the provider. */
awareness: Awareness
/**
* The realtime provider, or `null` until the socket is available. `doc` and
* `awareness` exist before it connects, so the editor can bind immediately; the
* provider is consumed for its readiness signal (`synced`) and fatal `join-error`.
*/
provider: FileDocProvider | null
/**
* The local caret identity published to awareness: `name`/`color` for CollaborationCaret,
* and `clientId` so the caret activity extension can tag each caret node (see
* caret-presence.ts). The avatar roster does NOT come from here it's server-authenticated
* (see the PRESENCE subscription below) so a peer can't spoof identity via awareness.
*/
user: { name: string; color: string; clientId: number | undefined }
}
interface UseFileDocCollaborationParams {
fileId: string
userId: string
userName: string
/**
* Whether to establish collaboration. Decided once at editor mount only for a
* live, editable, non-streaming workspace document. When `false` the hook
* returns `null` and the editor stays fully local.
*/
enabled: boolean
}
/**
* Owns the per-file Yjs document, awareness, and {@link FileDocProvider} for
* collaborative editing. The document + awareness are created once (this hook
* lives inside an editor that is keyed by file id, so one instance == one file)
* and are the stable objects TipTap binds to; the provider connects them to the
* realtime relay over the shared socket. Returns `null` while disabled.
*/
export function useFileDocCollaboration({
fileId,
userId,
userName,
enabled,
}: UseFileDocCollaborationParams): FileDocCollaboration | null {
const { socket } = useSocket()
// The Y.Doc + Awareness are the editor's authoritative binding — created once
// and stable for the hook's life (see sim-react-performance: lazy-init ref).
// Only allocated when collaboration is enabled, so read-only / streaming /
// round-trip-unsafe views never build a Yjs document they won't use.
const docRef = useRef<Y.Doc | null>(null)
const awarenessRef = useRef<Awareness | null>(null)
// Created ONCE and kept stable for the whole editor lifetime. The editor freezes these into
// its extension set at mount (`useEditor` fixes extensions at creation), so the instances the
// provider binds MUST stay byte-identical to what the editor holds — never destroyed-and-
// recreated mid-life. If a StrictMode dev remount destroyed them, the still-mounted editor
// would keep the dead doc while the provider synced a fresh one, and a joining peer would see a
// blank document. Teardown is therefore deferred to a REAL unmount only (see below).
if (enabled && docRef.current === null) {
docRef.current = new Y.Doc()
awarenessRef.current = new Awareness(docRef.current)
}
// Destroy the doc + awareness on a REAL unmount only. `Awareness` runs a setInterval to expire
// stale peers, so it MUST be destroyed or that timer leaks for every file ever opened. But a
// StrictMode dev remount fires this cleanup and then re-runs the setup synchronously after — so
// we SCHEDULE the teardown and the remount cancels it, keeping the frozen instances alive. A
// genuine unmount has no remount, so the scheduled teardown runs on the next tick.
const pendingTeardownRef = useRef<ReturnType<typeof setTimeout> | null>(null)
useEffect(() => {
if (pendingTeardownRef.current !== null) {
clearTimeout(pendingTeardownRef.current)
pendingTeardownRef.current = null
}
return () => {
pendingTeardownRef.current = setTimeout(() => {
awarenessRef.current?.destroy()
docRef.current?.destroy()
}, 0)
}
}, [])
const [provider, setProvider] = useState<FileDocProvider | null>(null)
useEffect(() => {
if (!enabled || !socket) return
// Non-null: both refs are set during render before any effect runs, and are never destroyed
// (see above), so this always binds the same doc/awareness the editor froze at mount.
const doc = docRef.current as Y.Doc
const awareness = awarenessRef.current as Awareness
const fileProvider = new FileDocProvider(socket, fileId, doc, awareness)
setProvider(fileProvider)
return () => {
fileProvider.destroy()
setProvider(null)
}
}, [enabled, socket, fileId])
const reportOthers = useReportFileDocOthers()
const reportOthersRef = useRef(reportOthers)
reportOthersRef.current = reportOthers
// "Who's in this file" roster (the useOthers side of the pattern). The server broadcasts a
// roster of SERVER-AUTHENTICATED identities (see FILE_DOC_EVENTS.PRESENCE) — trusted,
// unlike the client-set awareness `user` field a peer could spoof. Publish it (minus self)
// to the room context for the file-detail avatar stack; cleared on unmount so a file switch
// never shows the previous file's occupants.
useEffect(() => {
if (!enabled || !socket) return
const handlePresence = (data: FileDocPresence) => {
if (data.fileId !== fileId) return
// Exclude only our OWN socket (this session), NOT every session that shares our userId —
// so a second tab of the same account still counts as present, matching the canvas avatars
// (avatars.tsx filters by socketId). Then dedupe per user for the display stack, so
// multiple tabs of one person collapse to a single avatar.
const byUser = new Map<
string,
{ userId: string; userName: string; avatarUrl: string | null }
>()
for (const peer of data.users) {
if (peer.socketId === socket.id || byUser.has(peer.userId)) continue
byUser.set(peer.userId, {
userId: peer.userId,
userName: peer.userName,
avatarUrl: peer.avatarUrl,
})
}
reportOthersRef.current([...byUser.values()])
}
socket.on(FILE_DOC_EVENTS.PRESENCE, handlePresence)
return () => {
socket.off(FILE_DOC_EVENTS.PRESENCE, handlePresence)
reportOthersRef.current([])
}
}, [enabled, socket, fileId])
// The client id rides in the awareness `user` payload so the caret `render` (which only
// receives `user`) can tag each caret node for the activity-driven name label (see
// caret-presence.ts). `doc.clientID` is stable for the doc's life, so reading it from the
// ref needs no memo dep.
const user = useMemo(
() => ({ name: userName, color: getUserColor(userId), clientId: docRef.current?.clientID }),
[userName, userId]
)
return useMemo(
() =>
enabled
? {
doc: docRef.current as Y.Doc,
awareness: awarenessRef.current as Awareness,
provider,
user,
}
: null,
[enabled, provider, user]
)
}
@@ -1,8 +1,18 @@
import type { Extensions } from '@tiptap/core'
import Collaboration from '@tiptap/extension-collaboration'
import CollaborationCaret from '@tiptap/extension-collaboration-caret'
import Placeholder from '@tiptap/extension-placeholder'
import type { Awareness } from 'y-protocols/awareness'
import type * as Y from 'yjs'
import { withAlpha } from '@/lib/workspaces/colors'
import { BlockMover } from './block-mover'
import { CodeBlockWithLanguage } from './code-block'
import { CodeBlockHighlight } from './code-highlight'
import {
createCaretActivityExtension,
DEFAULT_CARET_COLOR,
renderCaret,
} from './collaboration/caret-presence'
import { LinkEmbed } from './embed/link-embed'
import { createMarkdownContentExtensions } from './extensions'
import { ResizableImage } from './image'
@@ -13,10 +23,20 @@ import { MentionChip } from './mention/mention-chip'
import { FootnoteDefWithView, RawHtmlBlockWithView } from './raw-markdown-snippet'
import { SlashCommand } from './slash-command/slash-command'
/** Live collaboration binding for the editor. When present, the editor's history
* is Yjs-backed and remote carets/selection render via CollaborationCaret. */
export interface EditorCollaboration {
doc: Y.Doc
awareness: Awareness
user: { name: string; color: string }
}
interface MarkdownEditorExtensionOptions {
placeholder: string
/** Renders supported media links as live players beneath a standalone link. Off by default. */
embeds?: boolean
/** When set, wires TipTap Collaboration + CollaborationCaret onto the shared document. */
collaboration?: EditorCollaboration
}
/**
@@ -32,15 +52,41 @@ interface MarkdownEditorExtensionOptions {
export function createMarkdownEditorExtensions({
placeholder,
embeds = false,
collaboration,
}: MarkdownEditorExtensionOptions): Extensions {
return [
...createMarkdownContentExtensions({
codeBlock: CodeBlockWithLanguage,
image: ResizableImage,
mention: MentionChip,
rawHtmlBlock: RawHtmlBlockWithView,
footnoteDef: FootnoteDefWithView,
}),
...createMarkdownContentExtensions(
{
codeBlock: CodeBlockWithLanguage,
image: ResizableImage,
mention: MentionChip,
rawHtmlBlock: RawHtmlBlockWithView,
footnoteDef: FootnoteDefWithView,
},
{ disableHistory: Boolean(collaboration) }
),
...(collaboration
? [
Collaboration.configure({ document: collaboration.doc }),
// CollaborationCaret reads only `provider.awareness` (created synchronously,
// relayed by the socket provider once connected). `render` tags each caret
// with the peer's client id and shows its name label; the selection tint is
// a translucent fill of the peer's identity color.
CollaborationCaret.configure({
provider: { awareness: collaboration.awareness },
user: collaboration.user,
render: renderCaret,
selectionRender: (user) => {
const hex = typeof user.color === 'string' ? user.color : DEFAULT_CARET_COLOR
return {
class: 'collaboration-carets__selection',
style: `background-color: ${withAlpha(hex, 0.2)};`,
}
},
}),
createCaretActivityExtension(collaboration.awareness),
]
: []),
CodeBlockHighlight,
SlashCommand,
Mention,
@@ -11,13 +11,18 @@ import {
} from '@tiptap/extension-table'
import { Markdown } from '@tiptap/markdown'
import StarterKit from '@tiptap/starter-kit'
import { MarkdownCodeBlock } from './code-block'
import { MarkdownCodeBlock } from './code-block-schema'
import { Highlight } from './highlight'
import { MarkdownImage } from './image'
import { MarkdownImage } from './image-schema'
import { MarkdownLinkInputRule } from './link-input-rule'
import { MarkdownMention } from './mention/mention-node'
import { SIM_LINK_SCHEME } from './mention/sim-link'
import { FootnoteDef, FootnoteRef, RawHtmlBlock, RawInlineHtml } from './raw-markdown-snippet'
import {
FootnoteDef,
FootnoteRef,
RawHtmlBlock,
RawInlineHtml,
} from './raw-markdown-snippet-schema'
/**
* The `@`-mention link scheme, registered on the Link mark without it the schema strips the
@@ -117,7 +122,10 @@ export interface ContentNodeViews {
* registry. The live editor passes the node-view nodes via {@link createMarkdownEditorExtensions}; the
* schema and markdown output are identical either way.
*/
export function createMarkdownContentExtensions(nodeViews: ContentNodeViews = {}): Extensions {
export function createMarkdownContentExtensions(
nodeViews: ContentNodeViews = {},
options: { disableHistory?: boolean } = {}
): Extensions {
const codeBlock = (nodeViews.codeBlock ?? MarkdownCodeBlock).configure({
HTMLAttributes: { class: 'code-editor-theme' },
})
@@ -128,6 +136,9 @@ export function createMarkdownContentExtensions(nodeViews: ContentNodeViews = {}
codeBlock: false,
code: false,
paragraph: false,
// Collaboration provides its own (Yjs-backed) undo/redo — disabling the
// built-in history avoids the two fighting over the shared document.
...(options.disableHistory ? { undoRedo: false as const } : {}),
}),
BlockSafeParagraph,
InlineCode,
@@ -0,0 +1,161 @@
import type { JSONContent } from '@tiptap/core'
import { Image } from '@tiptap/extension-image'
/**
* React-free schema half of the image node. Lives apart from {@link ./image} (its React resize node
* view) so the shared editor schema `createMarkdownContentExtensions` in `./extensions` can be
* imported by server code (the collab-doc seed converter) without pulling a client component
* (`useEffect`) into a Server Component module. The client editor injects the node-view variant
* ({@link ResizableImage}) via `nodeViews`.
*/
/**
* A markdown linked image `[![alt](src "t")](href "t2")` an image wrapped in a link, the canonical
* form of a README badge. `@tiptap/markdown` parses this as a link mark over an image node, but an
* image node can't carry inline marks, so the wrapping link is silently dropped. We instead tokenize
* the whole construct ourselves and hang the link target on the image node's `href` attribute, so it
* round-trips losslessly (and the file stays editable rather than opening read-only).
*/
const LINKED_IMAGE_RE =
/^\[!\[([^\]]*)\]\(([^)\s]+)(?:\s+"([^"]*)")?\)\]\(([^)\s]+)(?:\s+"([^"]*)")?\)/
/** Escape a value for safe interpolation into a double-quoted HTML attribute. */
function escapeAttr(value: string): string {
return value
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
}
/**
* Serialize an image to markdown when it has no explicit size, and to an HTML `<img>` tag when
* it does standard markdown has no width syntax, so a resized image must round-trip as HTML to
* preserve its dimensions. Unsized images stay clean `![alt](src)`. An image with an `href` is
* wrapped in a markdown link so a linked badge round-trips as `[![alt](src)](href)`.
*
* A *sized **and** linked* image is the one case markdown can't represent: the linked-image tokenizer
* only recognizes `[![alt](src)](href)`, so emitting `[<img …>](href)` would silently drop the link on
* reparse (and the round-trip-safety probe wouldn't catch it). We keep the link and fall back to the
* unsized `[![alt](src)](href)` form the link matters more than the exact dimensions for a badge.
*/
function imageMarkdown(node: JSONContent): string {
const attrs = node.attrs ?? {}
const src = typeof attrs.src === 'string' ? attrs.src : ''
const alt = typeof attrs.alt === 'string' ? attrs.alt : ''
const title = typeof attrs.title === 'string' ? attrs.title : ''
const href = typeof attrs.href === 'string' ? attrs.href : ''
const hrefTitle = typeof attrs.hrefTitle === 'string' ? attrs.hrefTitle : ''
const width = attrs.width
const height = attrs.height
let image: string
if ((width || height) && !href) {
const parts = [`src="${escapeAttr(src)}"`]
if (alt) parts.push(`alt="${escapeAttr(alt)}"`)
if (title) parts.push(`title="${escapeAttr(title)}"`)
if (width) parts.push(`width="${escapeAttr(String(width))}"`)
if (height) parts.push(`height="${escapeAttr(String(height))}"`)
image = `<img ${parts.join(' ')}>`
} else {
// Escape so an alt with `]`/`[` or a title with `"` can't break out of the `![…](… "…")` syntax
// and corrupt the round-trip; a src with spaces/parens goes in angle brackets (CommonMark).
const titlePart = title ? ` "${title.replace(/["\\]/g, '\\$&')}"` : ''
const safeSrc = /[\s()]/.test(src) ? `<${src}>` : src
image = `![${alt.replace(/[\\[\]]/g, '\\$&')}](${safeSrc}${titlePart})`
}
if (!href) return image
// Escape `"`/`\` so an href title can't break out of the `[…](href "title")` syntax (mirrors the
// image title escaping above).
const hrefTitlePart = hrefTitle ? ` "${hrefTitle.replace(/["\\]/g, '\\$&')}"` : ''
return `[${image}](${href}${hrefTitlePart})`
}
interface MarkdownImageToken {
/** Set only by our linked-image tokenizer; absent on the built-in `![](src)` token. */
src?: string
alt?: string
title?: string | null
/** Built-in image token holds the source URL here; our linked token holds the link target. */
href?: string
hrefTitle?: string | null
/** Built-in image token holds the alt text here. */
text?: string
}
/** Map both the built-in image token and our linked-image token onto the image node's attributes. */
function parseImageToken(token: MarkdownImageToken): JSONContent {
const isLinked = typeof token.src === 'string'
return {
type: 'image',
attrs: isLinked
? {
src: token.src,
alt: token.alt ?? '',
title: token.title ?? null,
href: token.href ?? null,
hrefTitle: token.hrefTitle ?? null,
}
: {
src: token.href ?? '',
alt: token.text ?? '',
title: token.title ?? null,
href: null,
hrefTitle: null,
},
}
}
const widthAttr = {
default: null,
parseHTML: (element: HTMLElement) => element.getAttribute('width'),
renderHTML: (attributes: Record<string, unknown>) =>
attributes.width ? { width: String(attributes.width) } : {},
}
const heightAttr = {
default: null,
parseHTML: (element: HTMLElement) => element.getAttribute('height'),
renderHTML: (attributes: Record<string, unknown>) =>
attributes.height ? { height: String(attributes.height) } : {},
}
/** Link target of a linked image — markdown-only state, never emitted as an HTML `<img>` attribute. */
const hrefAttr = { default: null, rendered: false }
const hrefTitleAttr = { default: null, rendered: false }
/**
* Image node that carries optional `width`/`height` (serialized as an HTML `<img>` tag) and an
* optional `href`/`hrefTitle` (a wrapping markdown link, for badges). Shared by the headless
* round-trip path (no node view) and the live {@link ResizableImage}.
*/
export const MarkdownImage = Image.extend({
addAttributes() {
return {
...this.parent?.(),
width: widthAttr,
height: heightAttr,
href: hrefAttr,
hrefTitle: hrefTitleAttr,
}
},
markdownTokenizer: {
name: 'image',
level: 'inline',
start: (src: string) => src.indexOf('[!['),
tokenize: (src: string): (MarkdownImageToken & { type: string; raw: string }) | undefined => {
const match = LINKED_IMAGE_RE.exec(src)
if (!match) return undefined
return {
type: 'image',
raw: match[0],
alt: match[1] ?? '',
src: match[2],
title: match[3] ?? null,
href: match[4],
hrefTitle: match[5] ?? null,
}
},
},
parseMarkdown: parseImageToken,
renderMarkdown: imageMarkdown,
})
@@ -1,167 +1,15 @@
import { useEffect, useRef, useState } from 'react'
import { cn } from '@sim/emcn'
import type { JSONContent } from '@tiptap/core'
import { Image } from '@tiptap/extension-image'
import { NodeSelection, Plugin } from '@tiptap/pm/state'
import type { ReactNodeViewProps } from '@tiptap/react'
import { NodeViewWrapper, ReactNodeViewRenderer } from '@tiptap/react'
import { useFileContentSource } from '@/hooks/use-file-content-source'
import { MarkdownImage } from './image-schema'
import { normalizeLinkHref } from './markdown-fidelity'
import { useEditorEditable } from './use-editor-editable'
const MIN_WIDTH = 64
/**
* A markdown linked image `[![alt](src "t")](href "t2")` an image wrapped in a link, the canonical
* form of a README badge. `@tiptap/markdown` parses this as a link mark over an image node, but an
* image node can't carry inline marks, so the wrapping link is silently dropped. We instead tokenize
* the whole construct ourselves and hang the link target on the image node's `href` attribute, so it
* round-trips losslessly (and the file stays editable rather than opening read-only).
*/
const LINKED_IMAGE_RE =
/^\[!\[([^\]]*)\]\(([^)\s]+)(?:\s+"([^"]*)")?\)\]\(([^)\s]+)(?:\s+"([^"]*)")?\)/
/** Escape a value for safe interpolation into a double-quoted HTML attribute. */
function escapeAttr(value: string): string {
return value
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
}
/**
* Serialize an image to markdown when it has no explicit size, and to an HTML `<img>` tag when
* it does standard markdown has no width syntax, so a resized image must round-trip as HTML to
* preserve its dimensions. Unsized images stay clean `![alt](src)`. An image with an `href` is
* wrapped in a markdown link so a linked badge round-trips as `[![alt](src)](href)`.
*
* A *sized **and** linked* image is the one case markdown can't represent: the linked-image tokenizer
* only recognizes `[![alt](src)](href)`, so emitting `[<img …>](href)` would silently drop the link on
* reparse (and the round-trip-safety probe wouldn't catch it). We keep the link and fall back to the
* unsized `[![alt](src)](href)` form the link matters more than the exact dimensions for a badge.
*/
function imageMarkdown(node: JSONContent): string {
const attrs = node.attrs ?? {}
const src = typeof attrs.src === 'string' ? attrs.src : ''
const alt = typeof attrs.alt === 'string' ? attrs.alt : ''
const title = typeof attrs.title === 'string' ? attrs.title : ''
const href = typeof attrs.href === 'string' ? attrs.href : ''
const hrefTitle = typeof attrs.hrefTitle === 'string' ? attrs.hrefTitle : ''
const width = attrs.width
const height = attrs.height
let image: string
if ((width || height) && !href) {
const parts = [`src="${escapeAttr(src)}"`]
if (alt) parts.push(`alt="${escapeAttr(alt)}"`)
if (title) parts.push(`title="${escapeAttr(title)}"`)
if (width) parts.push(`width="${escapeAttr(String(width))}"`)
if (height) parts.push(`height="${escapeAttr(String(height))}"`)
image = `<img ${parts.join(' ')}>`
} else {
// Escape so an alt with `]`/`[` or a title with `"` can't break out of the `![…](… "…")` syntax
// and corrupt the round-trip; a src with spaces/parens goes in angle brackets (CommonMark).
const titlePart = title ? ` "${title.replace(/["\\]/g, '\\$&')}"` : ''
const safeSrc = /[\s()]/.test(src) ? `<${src}>` : src
image = `![${alt.replace(/[\\[\]]/g, '\\$&')}](${safeSrc}${titlePart})`
}
if (!href) return image
// Escape `"`/`\` so an href title can't break out of the `[…](href "title")` syntax (mirrors the
// image title escaping above).
const hrefTitlePart = hrefTitle ? ` "${hrefTitle.replace(/["\\]/g, '\\$&')}"` : ''
return `[${image}](${href}${hrefTitlePart})`
}
interface MarkdownImageToken {
/** Set only by our linked-image tokenizer; absent on the built-in `![](src)` token. */
src?: string
alt?: string
title?: string | null
/** Built-in image token holds the source URL here; our linked token holds the link target. */
href?: string
hrefTitle?: string | null
/** Built-in image token holds the alt text here. */
text?: string
}
/** Map both the built-in image token and our linked-image token onto the image node's attributes. */
function parseImageToken(token: MarkdownImageToken): JSONContent {
const isLinked = typeof token.src === 'string'
return {
type: 'image',
attrs: isLinked
? {
src: token.src,
alt: token.alt ?? '',
title: token.title ?? null,
href: token.href ?? null,
hrefTitle: token.hrefTitle ?? null,
}
: {
src: token.href ?? '',
alt: token.text ?? '',
title: token.title ?? null,
href: null,
hrefTitle: null,
},
}
}
const widthAttr = {
default: null,
parseHTML: (element: HTMLElement) => element.getAttribute('width'),
renderHTML: (attributes: Record<string, unknown>) =>
attributes.width ? { width: String(attributes.width) } : {},
}
const heightAttr = {
default: null,
parseHTML: (element: HTMLElement) => element.getAttribute('height'),
renderHTML: (attributes: Record<string, unknown>) =>
attributes.height ? { height: String(attributes.height) } : {},
}
/** Link target of a linked image — markdown-only state, never emitted as an HTML `<img>` attribute. */
const hrefAttr = { default: null, rendered: false }
const hrefTitleAttr = { default: null, rendered: false }
/**
* Image node that carries optional `width`/`height` (serialized as an HTML `<img>` tag) and an
* optional `href`/`hrefTitle` (a wrapping markdown link, for badges). Shared by the headless
* round-trip path (no node view) and the live {@link ResizableImage}.
*/
export const MarkdownImage = Image.extend({
addAttributes() {
return {
...this.parent?.(),
width: widthAttr,
height: heightAttr,
href: hrefAttr,
hrefTitle: hrefTitleAttr,
}
},
markdownTokenizer: {
name: 'image',
level: 'inline',
start: (src: string) => src.indexOf('[!['),
tokenize: (src: string): (MarkdownImageToken & { type: string; raw: string }) | undefined => {
const match = LINKED_IMAGE_RE.exec(src)
if (!match) return undefined
return {
type: 'image',
raw: match[0],
alt: match[1] ?? '',
src: match[2],
title: match[3] ?? null,
href: match[4],
hrefTitle: match[5] ?? null,
}
},
},
parseMarkdown: parseImageToken,
renderMarkdown: imageMarkdown,
})
/**
* Drag-to-resize image node view (handle at the bottom-right, revealed on selection). Dragging
* commits the new pixel width to the `width` attribute, which serializes to `<img width>`.
@@ -11,6 +11,7 @@ import { GapCursor } from '@tiptap/pm/gapcursor'
import { AllSelection, NodeSelection } from '@tiptap/pm/state'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { createMarkdownEditorExtensions } from './editor-extensions'
import { postProcessSerializedMarkdown } from './markdown-fidelity'
import { MENTION_PLUGIN_KEY } from './mention'
import { SLASH_COMMAND_PLUGIN_KEY } from './slash-command/slash-command'
@@ -192,13 +193,11 @@ describe('empty wrapped-block Backspace', () => {
it.each([
['bullet middle', '- one\n- two\n- three', 'two', '- one\n- three'],
['bullet first', '- one\n- two\n- three', 'one', '- two\n- three'],
['bullet last', '- one\n- two', 'two', '- one'],
['ordered middle', '1. one\n2. two\n3. three', 'two', '1. one\n2. three'],
['task middle', '- [ ] one\n- [ ] two\n- [ ] three', 'two', '- [ ] one\n- [ ] three'],
['blockquote middle', '> one\n>\n> two\n>\n> three', 'two', '> one\n>\n> three'],
['nested item', '- one\n - two\n- three', 'two', '- one\n- three'],
])(
'removes the emptied %s cleanly — one container, no stray paragraph, round-trips',
'removes an emptied non-trailing %s cleanly — one container, no stray paragraph, round-trips',
(_label, markdown, word, expected) => {
const editor = editorWith('')
editor.commands.setContent(markdown, { contentType: 'markdown' })
@@ -213,10 +212,10 @@ describe('empty wrapped-block Backspace', () => {
}
)
it('leaves a gap cursor — never a NodeSelection — when the removed bullet was followed by an image at doc start', () => {
// Regression: `Selection.near` after the delete silently NodeSelected the following image, so a
// second Backspace while "clearing the bullet" deleted the image (and typing would have replaced
// it). The selection left behind must never make the next keystroke destructive.
it('clears a lone empty bullet before an image to a paragraph and never destroys the image', () => {
// Regression: an earlier delete-and-jump path silently NodeSelected the following image, so a
// second Backspace while "clearing the bullet" deleted it. The lone empty bullet now lifts into an
// empty paragraph in place, the image is untouched, and no repeated Backspace destroys it.
const editor = editorWith({
type: 'doc',
content: [
@@ -227,11 +226,11 @@ describe('empty wrapped-block Backspace', () => {
editor.commands.setTextSelection(3)
pressBackspace(editor)
expect(blockShape(editor)).toEqual(['image', 'paragraph'])
expect(blockShape(editor)).toEqual(['paragraph', 'image', 'paragraph'])
expect(editor.state.selection).not.toBeInstanceOf(NodeSelection)
pressBackspace(editor)
expect(blockShape(editor)).toEqual(['image', 'paragraph'])
expect(blockShape(editor)).toContain('image')
editor.destroy()
})
@@ -252,11 +251,10 @@ describe('empty wrapped-block Backspace', () => {
editor.destroy()
})
it('never NodeSelects a leaf BEFORE the removed bullet either (findFrom textOnly skips atoms)', () => {
// `Selection.findFrom($gap, -1, true)` cannot return a NodeSelection: with textOnly,
// prosemirror-state's findSelectionIn skips atoms entirely (`!text && isSelectable`). With an
// image directly before the emptied bullet and no textblock behind it, the backward search
// returns null and the gap-cursor branch takes over — the image is never silently selected.
it('clears a lone empty bullet after an image to a paragraph without selecting the image', () => {
// With an image directly before the emptied bullet, clearing the bullet must not silently select
// (and so endanger) the image. The bullet lifts into an empty paragraph in place; the image is
// untouched and no repeated Backspace destroys it.
const editor = editorWith({
type: 'doc',
content: [
@@ -268,11 +266,11 @@ describe('empty wrapped-block Backspace', () => {
expect(editor.state.selection.$from.parent.type.name).toBe('paragraph')
pressBackspace(editor)
expect(blockShape(editor)).toEqual(['image', 'paragraph'])
expect(blockShape(editor)).toEqual(['image', 'paragraph', 'paragraph'])
expect(editor.state.selection).not.toBeInstanceOf(NodeSelection)
pressBackspace(editor)
expect(blockShape(editor)).toEqual(['image', 'paragraph'])
expect(blockShape(editor)).toContain('image')
editor.destroy()
})
@@ -291,7 +289,7 @@ describe('empty wrapped-block Backspace', () => {
editor.destroy()
})
it('still prefers the previous textblock caret when one exists (image after the bullet untouched)', () => {
it('clears a lone empty bullet between a paragraph and an image to a paragraph, leaving the image', () => {
const editor = editorWith({
type: 'doc',
content: [
@@ -303,10 +301,204 @@ describe('empty wrapped-block Backspace', () => {
editor.commands.setTextSelection(10)
pressBackspace(editor)
expect(blockShape(editor)).toEqual(['paragraph', 'image', 'paragraph'])
expect(blockShape(editor)).toEqual(['paragraph', 'paragraph', 'image', 'paragraph'])
expect(editor.state.selection.empty).toBe(true)
expect(editor.state.selection).not.toBeInstanceOf(NodeSelection)
expect(editor.state.selection.$from.parent.textContent).toBe('hello')
// The cleared bullet is now an empty paragraph, and 'hello' is untouched above it.
expect(editor.state.doc.firstChild?.textContent).toBe('hello')
editor.destroy()
})
})
describe('list Backspace (clear / outdent)', () => {
beforeEach(() => {
Element.prototype.scrollIntoView = vi.fn()
})
/** Puts the caret at the very start of the item text `word`. */
function caretAtStartOf(editor: Editor, word: string): void {
editor.state.doc.descendants((node, pos) => {
if (node.isText && node.text === word) editor.commands.setTextSelection(pos)
})
}
it('lifts a top-level bullet WITH TEXT into a paragraph, keeping the text (round-trips)', () => {
const editor = editorWith('')
editor.commands.setContent('- one\n- two', { contentType: 'markdown' })
editor.commands.focus()
caretAtStartOf(editor, 'two')
pressBackspace(editor)
// A bulletList (just 'one') followed by the lifted 'two' paragraph (+ TipTap's trailing filler).
expect(blockShape(editor).slice(0, 2)).toEqual(['bulletList', 'paragraph'])
const { md, reparsed } = markdownRoundTrip(editor)
expect(md.trim()).toBe('- one\n\ntwo')
expect(reparsed).toBe(md)
editor.destroy()
})
it('clears an empty TRAILING bullet to a paragraph in place — no delete, caret stays on the line', () => {
const editor = editorWith('')
editor.commands.setContent('- one\n- two', { contentType: 'markdown' })
editor.commands.focus()
emptyItem(editor, 'two')
pressBackspace(editor)
// The bullet becomes a paragraph; the list keeps only 'one'. The caret sits in the new empty
// paragraph rather than jumping back into the 'one' bullet.
const list = editor.getJSON().content?.find((node) => node.type === 'bulletList')
expect(list?.content).toHaveLength(1)
expect(editor.state.selection.empty).toBe(true)
expect(editor.state.selection.$from.parent.type.name).toBe('paragraph')
expect(editor.state.selection.$from.parent.textContent).toBe('')
expect(editor.getMarkdown().trim()).toBe('- one')
editor.destroy()
})
it('clears a lone empty bullet to an empty paragraph (whole doc)', () => {
const editor = editorWith('')
editor.commands.setContent('- one', { contentType: 'markdown' })
editor.commands.focus()
emptyItem(editor, 'one')
pressBackspace(editor)
expect(editor.getJSON().content?.some((n) => n.type === 'bulletList')).toBe(false)
expect(editor.state.selection.$from.parent.type.name).toBe('paragraph')
editor.destroy()
})
it('outdents a nested bullet WITH TEXT one level instead of merging it (round-trips)', () => {
const editor = editorWith('')
editor.commands.setContent('- one\n - two', { contentType: 'markdown' })
editor.commands.focus()
caretAtStartOf(editor, 'two')
pressBackspace(editor)
const { md, reparsed } = markdownRoundTrip(editor)
expect(md.trim()).toBe('- one\n- two')
expect(reparsed).toBe(md)
editor.destroy()
})
it('outdents an empty nested bullet one level (round-trips)', () => {
const editor = editorWith('')
editor.commands.setContent('- one\n - two\n- three', { contentType: 'markdown' })
editor.commands.focus()
emptyItem(editor, 'two')
pressBackspace(editor)
const { md, reparsed } = markdownRoundTrip(editor)
expect(md.trim()).toBe('- one\n- \n- three')
expect(reparsed).toBe(md)
editor.destroy()
})
it('clears a checklist item the same way (task item → paragraph)', () => {
const editor = editorWith('')
editor.commands.setContent('- [ ] one\n- [ ] two', { contentType: 'markdown' })
editor.commands.focus()
caretAtStartOf(editor, 'two')
pressBackspace(editor)
expect(blockShape(editor).slice(0, 2)).toEqual(['taskList', 'paragraph'])
expect(editor.getMarkdown().trim()).toBe('- [ ] one\n\ntwo')
editor.destroy()
})
it('does not delete a non-trailing item whose block holds only a non-text atom', () => {
// Emptiness is the caret block's content.size, not its text: a bullet holding only an inline atom
// (image/mention — here a hardBreak stand-in) is NOT block-empty, so Backspace clears it to a
// paragraph (content preserved) instead of removeEmptyWrappedBlock deleting the whole row.
const editor = editorWith('')
editor.commands.setContent({
type: 'doc',
content: [
{
type: 'bulletList',
content: [
{
type: 'listItem',
content: [{ type: 'paragraph', content: [{ type: 'hardBreak' }] }],
},
{
type: 'listItem',
content: [{ type: 'paragraph', content: [{ type: 'text', text: 'two' }] }],
},
],
},
],
})
const atomPos = firstPosOf(editor, 'hardBreak')
editor.commands.setTextSelection(atomPos)
pressBackspace(editor)
// The atom survives (not deleted) and 'two' is untouched.
expect(firstPosOf(editor, 'hardBreak')).toBeGreaterThanOrEqual(0)
expect(editor.state.doc.textContent).toContain('two')
editor.destroy()
})
it('removes only the empty first block of a multi-block item, not the whole item', () => {
// An empty first block whose item has sibling blocks must not lift the whole item out of the list;
// only that empty block is removed, the rest of the item (and the list) stays intact.
const editor = editorWith('')
editor.commands.setContent({
type: 'doc',
content: [
{
type: 'bulletList',
content: [
{
type: 'listItem',
content: [
{ type: 'paragraph' },
{ type: 'paragraph', content: [{ type: 'text', text: 'more' }] },
],
},
],
},
],
})
// Caret at the start of the empty first paragraph (position 3: doc>bulletList>listItem>paragraph).
editor.commands.setTextSelection(3)
pressBackspace(editor)
// Still a list (item was NOT lifted out to a top-level paragraph), and 'more' survives.
expect(blockShape(editor)[0]).toBe('bulletList')
expect(editor.state.doc.textContent).toBe('more')
const list = editor.getJSON().content?.find((n) => n.type === 'bulletList')
expect(list?.content).toHaveLength(1)
editor.destroy()
})
})
describe('empty nested bullet does not corrupt its parent (Enter → Tab)', () => {
beforeEach(() => {
Element.prototype.scrollIntoView = vi.fn()
})
it('serializes a stranded empty sub-bullet away instead of turning the parent into a heading', () => {
// Repro: type a bullet, Enter for a new bullet, Tab to indent it into an empty sub-bullet, then
// leave it. The serialized `- one\n - ` would re-parse as `- ## one` (Setext underline). The
// serialize step must strip the empty sub-bullet so the parent stays a bullet and round-trips.
const editor = editorWith('')
editor.commands.setContent('- one', { contentType: 'markdown' })
editor.commands.focus()
let end = -1
editor.state.doc.descendants((node, pos) => {
if (node.isText && node.text === 'one') end = pos + 3
})
editor.commands.setTextSelection(end)
pressKey(editor, 'Enter')
pressKey(editor, 'Tab')
const saved = postProcessSerializedMarkdown(editor.getMarkdown())
expect(saved).toBe('- one\n')
// Reloading the saved markdown keeps a bullet — never a heading.
editor.commands.setContent(saved, { contentType: 'markdown' })
expect(blockShape(editor)).not.toContain('heading')
expect(blockShape(editor)).toContain('bulletList')
editor.destroy()
})
})
@@ -341,6 +533,52 @@ describe('empty list-item Enter', () => {
expect(editor.getJSON().content?.some((node) => node.type === 'paragraph')).toBe(true)
editor.destroy()
})
it('outdents an empty NESTED item one level instead of removing it', () => {
const editor = editorWith('')
editor.commands.setContent('- one\n - two', { contentType: 'markdown' })
editor.commands.focus()
emptyItem(editor, 'two')
pressKey(editor, 'Enter')
// The emptied nested item outdents to a second top-level bullet rather than being deleted.
const list = editor.getJSON().content?.find((node) => node.type === 'bulletList')
expect(list?.content).toHaveLength(2)
expect(list?.content?.every((item) => item.type === 'listItem')).toBe(true)
editor.destroy()
})
it('removes only the empty first block of a multi-block item, matching Backspace (keeps the list)', () => {
// Symmetry with the Backspace multi-block case: an empty first block whose item has sibling blocks
// is removed in place — the continuation and the list stay intact — rather than exiting the list.
const editor = editorWith('')
editor.commands.setContent({
type: 'doc',
content: [
{
type: 'bulletList',
content: [
{
type: 'listItem',
content: [
{ type: 'paragraph' },
{ type: 'paragraph', content: [{ type: 'text', text: 'more' }] },
],
},
],
},
],
})
// Caret at the start of the empty first paragraph (doc>bulletList>listItem>paragraph).
editor.commands.setTextSelection(3)
pressKey(editor, 'Enter')
expect(blockShape(editor)[0]).toBe('bulletList')
expect(editor.state.doc.textContent).toBe('more')
const list = editor.getJSON().content?.find((n) => n.type === 'bulletList')
expect(list?.content).toHaveLength(1)
editor.destroy()
})
})
describe('verbatim block boundary (isolating)', () => {
@@ -20,6 +20,50 @@ const WRAPPER_TYPES = new Set(['listItem', 'taskItem', 'blockquote'])
/** Item node types a list is built from, used to detect an empty item's position within its list. */
const LIST_ITEM_TYPES = new Set(['listItem', 'taskItem'])
/** The enclosing list/task item at a caret position, with the facts the boundary keys branch on. */
interface ListItemContext {
/** `'listItem'` or `'taskItem'` — the type name `liftListItem` must be called with. */
itemType: string
/** The item's list is itself inside another list item, i.e. the item is indented. */
isNested: boolean
/**
* The caret's own block (the row the boundary key acts on) has no content. Uses `content.size`, so an
* inline image or mention atom counts as content a bullet holding only an image is NOT block-empty.
*/
blockEmpty: boolean
/** The item has more than one child block (continuation paragraph, nested list, block image, …). */
hasSiblingBlocks: boolean
/** The item is the last child of its immediate list. */
isTrailing: boolean
/** The caret sits in the item's first child block (the row a boundary key should act on). */
isFirstBlock: boolean
}
/**
* Resolves the nearest enclosing list/task item at `$from` and the facts the Backspace/Enter handlers
* branch on (nesting, emptiness, trailing position, whether the caret is in the item's first block), or
* null when the caret is not inside a list item. Walking up from `$from` finds the item regardless of
* how deeply the caret's block is nested inside it.
*/
function getListItemContext($from: ResolvedPos): ListItemContext | null {
for (let depth = $from.depth; depth >= 1; depth--) {
const item = $from.node(depth)
if (!LIST_ITEM_TYPES.has(item.type.name)) continue
const listDepth = depth - 1
const isNested = listDepth >= 1 && LIST_ITEM_TYPES.has($from.node(listDepth - 1).type.name)
const list = $from.node(listDepth)
return {
itemType: item.type.name,
isNested,
blockEmpty: $from.parent.content.size === 0,
hasSiblingBlocks: item.childCount > 1,
isTrailing: $from.index(listDepth) === list.childCount - 1,
isFirstBlock: $from.index(depth) === 0,
}
}
return null
}
const RICH_LEAF_SELECTION_FOCUS_KEY = new PluginKey<boolean>('richLeafSelectionFocus')
/** True when the resolved position sits anywhere inside a {@link WRAPPER_TYPES} ancestor. */
@@ -136,21 +180,26 @@ function selectAdjacentSelectedLeaf(editor: Editor, direction: 'up' | 'down'): b
* Editor-specific keyboard behavior layered on top of StarterKit's defaults:
*
* - **Backspace** at the start of a heading reverts it to a paragraph (ProseMirror's default joins or
* no-ops, stranding the heading style; a second Backspace then merges as usual). At the start of an
* *empty block inside a list item, task item, or blockquote* it removes that whole emptied wrapper via
* {@link removeEmptyWrappedBlock} instead of ProseMirror's default lift lifting an empty item out of
* the middle of a list/quote splits the container in two and strands an empty paragraph (a visible gap
* that also re-parses to a different markdown document), while the default `joinBackward` alternately
* no-ops on nested items (leaving them stuck) or merges an empty continuation paragraph into the
* previous item. At the start of a block whose previous sibling is a divider or image, where
* ProseMirror's `joinBackward` can't cross the leaf and no-ops: an *empty* block is deleted (clearing
* the blank line between/below dividers without touching the divider itself), while a *non-empty*
* block selects the leaf so a first Backspace highlights what a second deletes, the same
* highlight-before-delete affordance as clicking it and parity with the arrow-key leaf selection.
* - **Enter** on an *empty, non-trailing list/task item* removes the empty item ({@link
* removeEmptyWrappedBlock}) rather than letting the default split the list into two around a stranded
* empty paragraph (which does not round-trip). A *trailing* empty item still falls through to the
* default, which exits the list the standard "press Enter on a blank bullet to leave the list".
* no-ops, stranding the heading style; a second Backspace then merges as usual). At the start of a
* *list or task item* it outdents or clears in place via {@link getListItemContext}: a nested item outdents one
* level, a top-level item with text lifts out of the list into a paragraph (keeping the text), and a
* top-level *empty trailing* (or sole) item lifts into an empty paragraph in place so the blank
* bullet made by pressing Enter can be cleared back to normal text on the same line instead of being
* deleted with the caret jumping to the previous block. The one case lift can't take is a top-level
* *empty, non-trailing* item: lifting it strands an empty paragraph between the two list halves, which
* re-parses to a different markdown document (an empty line between list items is a loose list, not a
* break); that item is removed via {@link removeEmptyWrappedBlock} instead, keeping the list whole. An
* empty block inside a *blockquote* is likewise removed via {@link removeEmptyWrappedBlock}. At the
* start of a block whose previous sibling is a divider or image, where ProseMirror's `joinBackward`
* can't cross the leaf and no-ops: an *empty* block is deleted (clearing the blank line between/below
* dividers without touching the divider itself), while a *non-empty* block selects the leaf so a
* first Backspace highlights what a second deletes, the same highlight-before-delete affordance as
* clicking it and parity with the arrow-key leaf selection.
* - **Enter** on an empty *nested* list/task item outdents it one level, on an empty
* *non-trailing top-level* item removes it ({@link removeEmptyWrappedBlock}) rather than splitting the
* list around a stranded empty paragraph (which does not round-trip), and on an empty *trailing* item
* falls through to the default, which exits the list the standard "press Enter on a blank bullet to
* leave the list".
* - **Mod-A** inside a code block selects only that block's contents; pressing it again (when the
* block is already fully selected) falls through to the default whole-document select-all, the
* same scoped behavior as a code editor.
@@ -183,6 +232,28 @@ export const RichMarkdownKeymap = Extension.create({
if ($from.parent.type.name === 'heading') {
return editor.commands.setParagraph()
}
const listCtx = getListItemContext($from)
if (listCtx?.isFirstBlock) {
const { itemType, isNested, blockEmpty, hasSiblingBlocks, isTrailing } = listCtx
// Backspace at the start of a bullet outdents or clears it in place rather than
// deleting the row and jumping the caret to the previous block.
// - Nested item → outdent one level (empty or not).
// - Top-level item whose first line has content (text OR an inline image/mention) → lift out of
// the list into a paragraph, keeping that content.
// - Top-level item whose empty first block has *sibling* blocks (a continuation paragraph, a
// block image, a nested list) → remove only that empty first block via {@link
// removeEmptyWrappedBlock}, leaving the rest of the item intact (never lift the whole item).
// - Top-level empty single-block item that is trailing (or the sole item) → lift into an empty
// paragraph in place, so a fresh bullet made with Enter can be cleared to normal text in place.
// A top-level *empty, non-trailing* single-block item is the one case lift can't take: it strands
// an empty paragraph between the two list halves, which re-parses to a different markdown document
// (an empty line between list items is a loose list, not a break). That case removes the row via
// {@link removeEmptyWrappedBlock} instead, which keeps the list whole and round-trips.
if (isNested || !blockEmpty) return editor.commands.liftListItem(itemType)
if (hasSiblingBlocks) return removeEmptyWrappedBlock(editor, $from)
if (isTrailing) return editor.commands.liftListItem(itemType)
return removeEmptyWrappedBlock(editor, $from)
}
if ($from.parent.content.size === 0 && isInsideWrapper($from)) {
return removeEmptyWrappedBlock(editor, $from)
}
@@ -207,11 +278,17 @@ export const RichMarkdownKeymap = Extension.create({
if (!selection.empty || selection.$from.parentOffset !== 0) return false
const { $from } = selection
if ($from.parent.content.size !== 0) return false
const itemDepth = $from.depth - 1
if (itemDepth < 1 || !LIST_ITEM_TYPES.has($from.node(itemDepth).type.name)) return false
const listDepth = itemDepth - 1
const isTrailingItem = $from.index(listDepth) === $from.node(listDepth).childCount - 1
if (isTrailingItem) return false
const listCtx = getListItemContext($from)
if (!listCtx?.isFirstBlock) return false
// Enter on an empty item, mirroring the Backspace cases above: a nested item outdents one level;
// an empty first block that has *sibling* blocks (continuation paragraph, block image, nested
// list) removes only that empty block in place, keeping the rest of the item — never exiting the
// list or splitting it; a trailing single-block item falls through to the default (exits the
// list); and a non-trailing single-block item is removed rather than splitting the list around a
// stranded empty paragraph (which does not round-trip).
if (listCtx.isNested) return editor.commands.liftListItem(listCtx.itemType)
if (listCtx.hasSiblingBlocks) return removeEmptyWrappedBlock(editor, $from)
if (listCtx.isTrailing) return false
return removeEmptyWrappedBlock(editor, $from)
},
'Mod-a': ({ editor }) => {
@@ -0,0 +1,64 @@
import { describe, expect, it } from 'vitest'
import { postProcessSerializedMarkdown } from './markdown-fidelity'
describe('postProcessSerializedMarkdown — empty list-item stripping', () => {
it('drops a nested empty bullet that would re-parse as a Setext heading', () => {
// `- one\n - ` re-parses as `- ## one` (the ` - ` acts as a Setext underline). Stripping the
// empty bullet on serialize keeps the parent a bullet and makes the round-trip stable.
expect(postProcessSerializedMarkdown('- one\n - \n\n')).toBe('- one\n')
})
it('drops a nested empty ordered item', () => {
expect(postProcessSerializedMarkdown('1. one\n 2. \n')).toBe('1. one\n')
})
it('preserves a top-level empty bullet (placeholder / imported blank — round-trips faithfully)', () => {
// A top-level empty item is not Setext-hazardous and round-trips as an empty item, so it must be
// kept: it may be a placeholder row the user is about to fill, or an intentionally-blank imported item.
expect(postProcessSerializedMarkdown('- one\n- \n')).toBe('- one\n- \n')
expect(postProcessSerializedMarkdown('- one\n- \n- three\n')).toBe('- one\n- \n- three\n')
expect(postProcessSerializedMarkdown('1. one\n2. \n')).toBe('1. one\n2. \n')
})
it('keeps bullets that have content', () => {
expect(postProcessSerializedMarkdown('- a\n- b\n')).toBe('- a\n- b\n')
expect(postProcessSerializedMarkdown('- a\n - b\n')).toBe('- a\n - b\n')
})
it('keeps a nested empty parent whose next line is an indented child (no orphaning)', () => {
expect(postProcessSerializedMarkdown('- top\n - \n - child\n')).toBe(
'- top\n - \n - child\n'
)
})
it('keeps a nested empty item that follows a same-indent sibling (real placeholder, no Setext hazard)', () => {
// ` - ` after ` - two` (same indent) is a real empty item the parser keeps — it does NOT underline
// a shallower parent's text, so it must not be stripped. (Only ` - ` directly under `- one` does.)
expect(postProcessSerializedMarkdown('- one\n - two\n - \n - three\n')).toBe(
'- one\n - two\n - \n - three\n'
)
// The hazard case — empty item directly under the shallower parent — is still stripped.
expect(postProcessSerializedMarkdown('- one\n - \n - three\n')).toBe('- one\n - three\n')
})
it('keeps a thematic break and empty checklist items (not Setext-hazardous)', () => {
expect(postProcessSerializedMarkdown('text\n\n---\n\nmore\n')).toBe('text\n\n---\n\nmore\n')
expect(postProcessSerializedMarkdown('- [ ] a\n- [ ] \n')).toBe('- [ ] a\n- [ ] \n')
})
it('leaves marker-only lines inside a fenced code block untouched', () => {
const code = '```\n- \n-\n1. \n```\n'
expect(postProcessSerializedMarkdown(code)).toBe(code)
})
it('leaves marker-only lines inside a tilde (~~~) fence untouched', () => {
const code = '~~~\n- \n1. \n~~~\n'
expect(postProcessSerializedMarkdown(code)).toBe(code)
})
it('does not strip inside an unterminated fence (fence stays open to EOF)', () => {
// A fence with no closing delimiter must keep every interior line, including marker-only ones.
const code = '```\n- \n-\n'
expect(postProcessSerializedMarkdown(code)).toBe(code)
})
})
@@ -106,16 +106,78 @@ export function normalizeLinkHref(href: string): string {
return `https://${trimmed}`
}
/** A line that is a bullet/ordered list marker with no content (`-`, ` - `, `1. `). Task items (`- [ ]`) don't match. */
const EMPTY_LIST_ITEM_LINE = /^([ \t]*)(?:[-*+]|\d+[.)])[ \t]*$/
/** A fenced code-block delimiter (``` or ~~~), used to leave code interiors untouched. */
const FENCE_DELIMITER = /^[ \t]*(`{3,}|~{3,})/
/** Leading indentation of a line, used to detect whether an empty list item has indented children. */
const LEADING_INDENT = /^[ \t]*/
/**
* Cleans up serializer output: restores callout markers the serializer backslash-escapes
* (`> \[!NOTE\]` `> [!NOTE]`) and collapses trailing blank lines to a single newline. The
* table serializer's spurious surrounding blank lines are trimmed at the source (PipeSafeTable),
* so no global leading-newline strip is needed here avoiding clobbering content that legitimately
* begins with whitespace.
* Removes only the *nested* empty list-item marker lines that re-parse as a Setext heading underline:
* a nested empty bullet (` - `) sitting DIRECTLY under a shallower parent line silently turns that
* parent's text into an `## heading` and drops the bullet on the next load (a data-corrupting
* round-trip). The strip is therefore scoped by three conditions, all required:
* - *indented* (`indent > 0`): a top-level empty bullet (`- ` / `1. `) round-trips faithfully as an
* empty item, never a heading, so a placeholder/blank imported row is preserved.
* - the immediately-preceding line is *shallower* (the parent whose text the underline would consume):
* an empty item after a *same-indent sibling* (` - two` then ` - `) does NOT corrupt the parser
* keeps it as a real empty item so it is preserved. A blank line above also breaks the hazard.
* - no more-indented children on the next non-blank line, so its children are never orphaned.
*
* Operates only on the editor's own serialized output, which uses fenced (never 4-space-indented) code
* blocks and `\n` newlines so tracking fences is sufficient and a bare `-` inside an indented code
* block or a `-\r` line is not a case that can occur here.
*/
function stripEmptyListItemLines(markdown: string): string {
const lines = markdown.split('\n')
const kept: string[] = []
let fence: string | null = null
for (let i = 0; i < lines.length; i++) {
const line = lines[i]
const delimiter = line.match(FENCE_DELIMITER)?.[1]
if (fence) {
kept.push(line)
if (delimiter && delimiter[0] === fence[0] && delimiter.length >= fence.length) fence = null
continue
}
if (delimiter) {
fence = delimiter
kept.push(line)
continue
}
const empty = line.match(EMPTY_LIST_ITEM_LINE)
if (empty) {
const indent = empty[1].length
let next = i + 1
while (next < lines.length && lines[next].trim() === '') next++
const hasChildren =
next < lines.length && (lines[next].match(LEADING_INDENT)?.[0].length ?? 0) > indent
// The Setext-underline hazard exists only when the empty item follows a SHALLOWER parent line
// (whose text the underline would consume). An empty item after a same/deeper-indent sibling
// (` - two` then ` - `) is a real empty item the parser keeps — a nested placeholder between
// siblings must not be lost. Uses the preceding non-blank line's indent; a lone empty item with
// nothing above it (`prevIndent = -1`) has no parent text to corrupt but stays stripped as before.
let prevIdx = i - 1
while (prevIdx >= 0 && lines[prevIdx].trim() === '') prevIdx--
const prevIndent = prevIdx >= 0 ? (lines[prevIdx].match(LEADING_INDENT)?.[0].length ?? 0) : -1
if (indent > 0 && !hasChildren && prevIndent < indent) continue
}
kept.push(line)
}
return kept.join('\n')
}
/**
* Cleans up serializer output: drops empty list-item marker lines that would otherwise corrupt on
* round-trip ({@link stripEmptyListItemLines}), restores callout markers the serializer
* backslash-escapes (`> \[!NOTE\]` `> [!NOTE]`), and collapses trailing blank lines to a single
* newline. The table serializer's spurious surrounding blank lines are trimmed at the source
* (PipeSafeTable), so no global leading-newline strip is needed here avoiding clobbering content
* that legitimately begins with whitespace.
*/
export function postProcessSerializedMarkdown(markdown: string): string {
return collapseAutolinkedUrls(markdown.replace(ESCAPED_CALLOUT_REGEX, '$1[!$2]')).replace(
/\n+$/,
'\n'
)
return collapseAutolinkedUrls(
stripEmptyListItemLines(markdown).replace(ESCAPED_CALLOUT_REGEX, '$1[!$2]')
).replace(/\n+$/, '\n')
}
@@ -59,6 +59,12 @@ const CASES: Array<[string, string]> = [
'1. First\n - sub bullet\n - another\n 1. deep ordered\n 2. item\n2. Second',
],
['heading-separated sections', '# A\n\nalpha\n\n## B\n\nbeta\n\n## C\n\ngamma'],
// Blank-line spacing: `@tiptap/markdown` reconstructs empty paragraphs from runs of blank lines, so
// the chunker must reinsert them or a saved blank line vanishes on reload. See the dedicated
// "empty paragraphs" suite below for the exact whole-document-parser parity.
['one empty paragraph between paragraphs', 'first\n\n\n\nsecond'],
['two empty paragraphs between paragraphs', 'first\n\n\n\n\n\nsecond'],
['empty paragraphs between headings and text', '# A\n\n\n\nalpha\n\n\n\n## B'],
]
describe('parseMarkdownToDoc (chunked)', () => {
@@ -87,6 +93,66 @@ describe('parseMarkdownToDoc (chunked)', () => {
expect(splitMarkdownBlocks('\n\n \n')).toEqual([])
})
// The chunker used to drop empty paragraphs (visual blank lines between blocks) that the whole-document
// parser preserves, so a saved blank line silently vanished on the next load. These assert the chunked
// parse reconstructs the SAME empty-paragraph structure the whole-document parser does — at document
// edges and between blocks, for one or many blank lines, and around lists.
describe('empty paragraphs (blank-line spacing) match the whole-document parser', () => {
/** Block-type shape of a doc, `∅` for an empty paragraph, normalized through the editor. */
function shapeOf(md: string, parse: 'chunked' | 'whole'): string {
editor = new Editor({ extensions: createMarkdownContentExtensions() })
if (parse === 'whole') editor.commands.setContent(md, { contentType: 'markdown' })
else editor.commands.setContent(parseMarkdownToDoc(md), { contentType: 'json' })
const shape = (editor.getJSON().content ?? [])
.map((n) => (n.type === 'paragraph' && !n.content?.length ? '∅' : n.type))
.join(',')
editor.destroy()
editor = null
return shape
}
it.each([
['one empty between paragraphs', 'a\n\n\n\nb'],
['two empties between paragraphs', 'a\n\n\n\n\n\nb'],
['three empties between paragraphs', 'a\n\n\n\n\n\n\n\nb'],
['even blank-line gap (rounds down)', 'a\n\n\n\n\nb'],
['leading empties', '\n\n\n\na'],
['leading + between', '\n\n\na\n\n\n\nb'],
['empties between a heading and text', '# H\n\n\n\ntext'],
['empties after a tight list', '- a\n- b\n\n\n\ntext'],
['empties before a tight list', 'text\n\n\n\n- a\n- b'],
// Line-ending variants: the whole-vs-chunked routing must normalize first, or a `\r`-only body
// skips the empty-paragraph guard and is chunked (dropping the empties this fix restores).
['CRLF between empties', 'a\r\n\r\n\r\n\r\nb'],
['CR-only (classic Mac) between empties', 'a\r\r\r\rb'],
])('chunked matches whole-doc: %s', (_label, md) => {
expect(shapeOf(md, 'chunked')).toBe(shapeOf(md, 'whole'))
})
})
// Regression: a file ending in a blank line (a trailing empty paragraph) must stay EDITABLE. Such an
// empty paragraph can't be serialized stably (postProcess collapses trailing newlines), so the parser
// strips it — keeping the doc round-trip-safe/idempotent instead of flipping the file read-only.
describe('trailing blank lines stay editable (regression)', () => {
it.each([
['plain paragraph', 'abc\n\n'],
['heading + text', '# Title\n\nSome text\n\n'],
['three trailing newlines', 'hello\n\n\n'],
['two paragraphs', 'para one\n\npara two\n\n'],
['interior empties + trailing', 'a\n\n\n\nb\n\n'],
])('a file ending in a blank line is round-trip-safe: %s', (_label, md) => {
expect(isRoundTripSafe(md)).toBe(true)
})
it('strips the trailing empty paragraph but keeps interior ones', () => {
const trailing = parseMarkdownToDoc('abc\n\n').content ?? []
expect(trailing.at(-1)?.type).toBe('paragraph')
expect(trailing.at(-1)?.content?.length ?? 0).toBeGreaterThan(0)
const interior = parseMarkdownToDoc('a\n\n\n\nb').content ?? []
expect(interior.some((n) => n.type === 'paragraph' && !n.content?.length)).toBe(true)
})
})
it('parses reference-style links whole (non-chunkable) without dropping the definition', () => {
const body = 'See [the docs][ref] for details.\n\n[ref]: https://example.com/docs'
expect(serializeMarkdownBody(body)).toBe(oneShot(body))
@@ -195,13 +261,18 @@ function buildFuzzDoc(seed: number): string {
describe('chunked parse — property test over randomized documents', () => {
it('chunked === one-shot for every document, and idempotent for every editable one', () => {
const failures: Array<{ seed: number; kind: string }> = []
// Compare modulo trailing whitespace: `parseMarkdownToDoc` strips trailing empty paragraphs (they
// can't be serialized stably — postProcess collapses trailing newlines — so keeping them would flip
// the file read-only), whereas the raw one-shot parse keeps them. That trailing-only divergence is
// intended and invisible after save; interior/leading fidelity is still compared exactly.
const trimEnd = (md: string) => md.replace(/\n+$/, '')
for (let seed = 1; seed <= 400; seed++) {
const body = buildFuzzDoc(seed)
const chunked = serializeMarkdownBody(body)
// Fidelity is the load-bearing invariant — chunked must never diverge from the whole-document
// parse, for ANY input; idempotency only needs to hold where the doc is editable (raw HTML is
// non-idempotent in the underlying editor regardless of chunking, which is why it opens read-only).
if (chunked !== oneShot(body)) failures.push({ seed, kind: 'fidelity' })
if (trimEnd(chunked) !== trimEnd(oneShot(body))) failures.push({ seed, kind: 'fidelity' })
else if (isRoundTripSafe(body) && serializeMarkdownBody(chunked) !== chunked) {
failures.push({ seed, kind: 'idempotency' })
}
@@ -47,6 +47,20 @@ const FENCE_CLOSE = /^ {0,3}(`{3,}|~{3,})[ \t]*$/
const LIST_MARKER = /^[ ]{0,3}(?:[-*+]|\d+[.)])\s/
const BLOCKQUOTE = /^[ ]{0,3}>/
/**
* Blank-line spacing that `@tiptap/markdown` reconstructs as *interior* or *leading* empty paragraphs
* a run of two or more blank lines somewhere, or blank line(s) at the document's leading edge. `[^\S\n]`
* matches horizontal whitespace, so a "blank" line may carry spaces/tabs. This is only ever tested
* against the `\r`-normalized body ({@link parseMarkdownToDoc}), so no CRLF handling is needed here.
*
* A *single* trailing blank line is deliberately not matched purely to avoid routing an otherwise-plain
* file to the slower whole-document parser. Correctness does not depend on it: {@link parseMarkdownToDoc}
* strips trailing empty paragraphs on *both* parse paths ({@link stripTrailingEmptyParagraphs}), so
* serializeparse stays idempotent regardless of which parser ran. (A trailing run of two or more blanks
* still matches the interior alternative harmless, since the strip cleans it either way.)
*/
const EMPTY_PARAGRAPH_SPACING = /\n[^\S\n]*\n[^\S\n]*\n|^[^\S\n]*\n[^\S\n]*\n/
/**
* Split a markdown body into top-level blocks that can each be parsed independently and reassembled
* without changing meaning. Blank lines separate candidate groups (fenced code blocks stay atomic),
@@ -120,20 +134,57 @@ export function splitMarkdownBlocks(body: string): string[] {
* vs ~1270ms at 61KB and byte-identical, because each block is parsed with the same tokenizers.
* Documents whose constructs span blocks ({@link NON_CHUNKABLE}) parse whole, and any failure falls
* back to a single whole-document parse, so correctness never depends on the splitter.
*
* Blank-line spacing ({@link EMPTY_PARAGRAPH_SPACING}) also parses whole: the chunker parses each block
* stripped of the blank lines between them, so it drops the empty paragraphs `@tiptap/markdown` builds
* from runs of blank lines a saved visual blank line would silently vanish on reload. Whether a gap
* yields an empty paragraph is a global, block-type-dependent decision (kept between two paragraphs,
* dropped after a heading), so it can't be reconstructed block-locally; these documents parse whole for
* exact fidelity. Ordinary single-blank-line separation still takes the fast chunked path.
*/
export function parseMarkdownToDoc(body: string): JSONContent {
const manager = markdownManager()
if (NON_CHUNKABLE.test(body)) return manager.parse(body)
try {
const content: JSONContent[] = []
for (const block of splitMarkdownBlocks(body)) {
// `MarkdownManager.parse` always returns a doc node with a `content` array; spread its blocks.
content.push(...(manager.parse(block).content ?? []))
// Normalize line endings up front so the routing guards see the same `\n` the chunker and parser
// do — the guards' `\n`-anchored tests would otherwise miss a classic `\r`-only body (its blank
// lines are `\r`), routing it to the chunker that then drops its empty paragraphs.
const normalized = body.replace(/\r\n?/g, '\n')
let doc: JSONContent
if (NON_CHUNKABLE.test(normalized) || EMPTY_PARAGRAPH_SPACING.test(normalized)) {
doc = manager.parse(normalized)
} else {
try {
const content: JSONContent[] = []
for (const block of splitMarkdownBlocks(normalized)) {
// `MarkdownManager.parse` always returns a doc node with a `content` array; spread its blocks.
content.push(...(manager.parse(block).content ?? []))
}
doc = { type: 'doc', content }
} catch {
doc = manager.parse(normalized)
}
return { type: 'doc', content }
} catch {
return manager.parse(body)
}
return stripTrailingEmptyParagraphs(doc)
}
/** An empty paragraph node — the shape a blank line reconstructs to (no content, or `content: []`). */
function isEmptyParagraph(node: JSONContent): boolean {
return node.type === 'paragraph' && !node.content?.length
}
/**
* Drop trailing empty paragraphs from a parsed doc. {@link postProcessSerializedMarkdown} collapses
* trailing blank lines to a single newline, so a trailing empty paragraph can never round-trip the
* whole-document parser reconstructs one from a file ending in a blank line, but keeping it makes
* serializeparse non-idempotent, which flips the file read-only via the round-trip-safety probe.
* Leading/interior empty paragraphs are untouched (postProcess never strips those). TipTap re-adds its
* own trailing filler paragraph on `setContent`, so the editor still has a place to type.
*/
function stripTrailingEmptyParagraphs(doc: JSONContent): JSONContent {
const content = doc.content
if (!content || content.length === 0) return doc
let end = content.length
while (end > 0 && isEmptyParagraph(content[end - 1])) end--
return end === content.length ? doc : { ...doc, content: content.slice(0, end) }
}
/**
@@ -142,8 +193,19 @@ export function parseMarkdownToDoc(body: string): JSONContent {
* normalization the live editor applies, keeping the output identical to `editor.getMarkdown()`.
*/
export function serializeMarkdownBody(body: string): string {
return serializeDocToMarkdown(parseMarkdownToDoc(body))
}
/**
* Serialize a ProseMirror document (as TipTap {@link JSONContent}) to the editor's canonical
* markdown. Loaded via `setContent` so it passes through the same schema normalization the live
* editor applies output identical to `editor.getMarkdown()`. The server-side collab-doc converter
* uses this to project a Yjs doc back to markdown through the exact client engine (parity by
* construction), so it must stay the single serialize path (do not inline `getMarkdown` elsewhere).
*/
export function serializeDocToMarkdown(doc: JSONContent): string {
const editor = parserEditor()
editor.commands.setContent(parseMarkdownToDoc(body), { contentType: 'json' })
editor.commands.setContent(doc, { contentType: 'json' })
return editor.getMarkdown()
}
@@ -0,0 +1,481 @@
/**
* React-free schema half of the raw-HTML/footnote nodes. Lives apart from {@link ./raw-markdown-snippet}
* (their React node views) so the shared editor schema `createMarkdownContentExtensions` in
* `./extensions` can be imported by server code (the collab-doc seed converter) without pulling a
* client component (`useEffect`) into a Server Component module. The client editor injects the
* node-view variants ({@link RawHtmlBlockWithView}, {@link FootnoteDefWithView}) via `nodeViews`.
*/
import type { JSONContent, MarkdownToken } from '@tiptap/core'
import { mergeAttributes, Node } from '@tiptap/core'
/**
* Constructs the schema has no node/mark for: raw HTML blocks (`<div>`, `<details>`, ), HTML
* comments, and footnotes. Before this file, all four made the *entire* document open read-only
* (see {@link isRoundTripSafe in ./round-trip-safety}) because the stock pipeline silently drops
* or mangles them. Each node below instead holds the exact source text as its content and
* re-emits it byte-for-byte on serialize the same "hold raw source, re-render specially" shape
* `MarkdownCodeBlock` uses for Mermaid (see `./code-block.tsx`), just without the diagram render.
*
* Inline tags already covered by a real mark/node `em`/`i`, `strong`/`b`, `s`/`del`/`strike`,
* `code`, `a`, `br`, `img` are deliberately excluded from {@link RawInlineHtml} so they keep
* parsing into their proper mark (e.g. `<em>x</em>` italic) instead of freezing as raw source.
*/
const HANDLED_INLINE_TAGS = new Set([
'br',
'img',
'em',
'i',
'strong',
'b',
's',
'del',
'strike',
'code',
'a',
])
const VOID_TAGS = new Set([
'area',
'base',
'br',
'col',
'embed',
'hr',
'img',
'input',
'link',
'meta',
'param',
'source',
'track',
'wbr',
])
function verbatimText(node: JSONContent): string {
return (node.content ?? []).map((child) => child.text ?? '').join('')
}
const RAW_HTML_COMMENT_RE = /^<!--[\s\S]*?-->/
/**
* One HTML attribute: `name` or `name="value"`/`name='value'`/`name=bare`. The quoted-value
* alternatives are what matter `[^"]*`/`[^']*` consume a literal `>` inside the quotes as part of
* the value, so an attribute like `data-example="a > b"` is treated as one unit instead of ending
* the tag match at the internal `>`.
*/
const ATTRS_RE_SOURCE = String.raw`(?:\s+[^\s"'=<>\`]+(?:\s*=\s*(?:"[^"]*"|'[^']*'|[^\s"'=<>\`]+))?)*`
/** Matches one opening HTML tag, attributes included (see {@link ATTRS_RE_SOURCE}). Group 1 is the
* tag name, group 2 is the self-closing `/` if present shared by inline and block tokenizing. */
const OPEN_TAG_RE = new RegExp(`^<([a-z][\\w-]*)\\b${ATTRS_RE_SOURCE}\\s*(/)?>`, 'i')
/** A fenced block's opening/closing marker may sit inside a blockquote (each line prefixed with
* up to 3 spaces then one or more `>` markers, each optionally followed by a space) and/or be
* independently indented up to 3 spaces with no blockquote at all (CommonMark's own fence-indent
* tolerance matches `FENCE_OPEN`/`FENCE_CLOSE` in `./markdown-parse.ts`) matched on both the
* open and close fence line so `> \`\`\`` and ` \`\`\`` both mask correctly. */
const FENCE_PREFIX_SOURCE = '(?:[ ]{0,3}>[ ]?)*[ ]{0,3}'
/** Same as {@link RAW_HTML_COMMENT_RE} but not anchored to the start of the string used by
* {@link maskCodeRegions} to find a comment anywhere in the scanned text, not just at position 0. */
const HTML_COMMENT_ANYWHERE_RE = /<!--[\s\S]*?-->/g
/**
* Mask fenced code blocks, inline code spans, and HTML comments with same-length filler (newlines
* kept, everything else replaced with a space) so a tag-like mention *inside one of these*
* `` `</details>` ``, a fenced example showing HTML syntax, or a comment documenting the tag
* (`<!-- see </div> below -->`) is never mistaken for a real balancing tag while scanning. Mirrors
* the fenced/inline patterns `stripCode` in `./round-trip-safety.ts` matches (extended to also
* tolerate an indented and/or blockquoted fence marker via {@link FENCE_PREFIX_SOURCE}, since a raw
* HTML block can itself be indented or quoted), but preserves length/position (masks in place)
* instead of deleting, so match indices still map onto the original, unmodified `src` the caller
* slices from.
*/
function maskCodeRegions(src: string): string {
const fenceRe = new RegExp(
`^${FENCE_PREFIX_SOURCE}([\`~]{3,})[^\\n]*\\n[\\s\\S]*?^${FENCE_PREFIX_SOURCE}\\1[\`~]*[ \\t]*$`,
'gm'
)
return src
.replace(fenceRe, (m) => m.replace(/[^\n]/g, ' '))
.replace(/`+[^`\n]*`+/g, (m) => ' '.repeat(m.length))
.replace(HTML_COMMENT_ANYWHERE_RE, (m) => m.replace(/[^\n]/g, ' '))
}
/**
* Find the end of the close tag that balances the open tag of `tag` ending at `src[0, fromIndex)`,
* tracking nesting depth from `fromIndex` onward so `<span>outer <span>inner</span></span>` consumes
* both levels instead of stopping at the first (inner) `</span>`. Returns -1 if unterminated. A
* nested self-closing same-name tag (`<span/>`) is skipped it neither opens nor closes a level.
* Shared by the inline tokenizer (single line) and the block tokenizer (spans blank lines).
*
* Scans a {@link maskCodeRegions}-masked copy of `src` so a tag name mentioned inside code doesn't
* count as real markup this narrows, but can't eliminate, the inherent ambiguity of regex-based
* (non-DOM) tag matching: a *bare, unescaped* mention of the same tag name in plain prose (not in
* code) is indistinguishable from a real closing tag here, exactly as it would be to a real HTML
* parser given the same ambiguous input (there is no valid way to "escape" a literal `</tag>` inside
* real HTML content other than an entity or code region). Verified this can't lose data even in that
* case the result still reaches a stable fixpoint on save, just restructured matching this file's
* "reject on doubt, but never require doubt-free input" gate (`isRoundTripSafe`).
*/
function findBalancedCloseEnd(src: string, tag: string, fromIndex: number): number {
const masked = maskCodeRegions(src)
const tagRe = new RegExp(`<(/?)${tag}\\b${ATTRS_RE_SOURCE}\\s*(/)?>`, 'gi')
tagRe.lastIndex = fromIndex
let depth = 1
for (let match = tagRe.exec(masked); match; match = tagRe.exec(masked)) {
const isClose = match[1] === '/'
const isSelfClosing = Boolean(match[2])
if (isSelfClosing) continue
if (isClose) {
depth -= 1
if (depth === 0) return match.index + match[0].length
} else {
depth += 1
}
}
return -1
}
/**
* Marked's own block tokenizer greedily consumes the blank-line run *after* an HTML block/comment
* or a def line as part of that token's own `raw` (the same behavior `PipeSafeTable` in
* `./extensions.ts` works around for tables) storing it verbatim would double it up with the
* block joiner's own separator, growing by two newlines on every save. Block-level callers trim it;
* inline callers never carry one (inline tokens can't span a blank line), so trimming is a no-op there.
*/
function verbatimParse(raw: string): JSONContent[] {
const trimmed = raw.replace(/\n+$/, '')
return trimmed ? [{ type: 'text', text: trimmed }] : []
}
interface VerbatimNodeOptions {
name: string
/** Whether this node sits among block content (own line) or inline content (mid-paragraph). */
inline: boolean
badgeLabel: string
}
/**
* Shared shape for a node that holds a markdown construct's exact source text and re-emits it
* unchanged parsing and rendering never inspect or transform the text, so there is nothing for
* these constructs to lose. `markdownTokenName`/`parseMarkdown`/`renderMarkdown` are read directly
* off the returned config by `@tiptap/markdown`'s `MarkdownManager` (see
* `node_modules/@tiptap/markdown/src/MarkdownManager.ts`), independent of the node's `name`.
*/
function verbatimNodeConfig({ name, inline, badgeLabel }: VerbatimNodeOptions) {
return {
name,
inline,
group: inline ? 'inline' : 'block',
content: 'text*',
marks: '',
code: true,
defining: !inline,
// Block verbatim nodes hold exact source text; `isolating` stops a boundary Backspace/Delete from
// joining across their edge, which would otherwise merge their raw markdown into an adjacent
// paragraph as HTML-escaped prose and destroy the node (silent data loss on save).
isolating: !inline,
selectable: true,
atom: false,
parseHTML() {
return [
{
tag: `${inline ? 'span' : 'div'}[data-raw-markdown="${name}"]`,
preserveWhitespace: 'full' as const,
},
]
},
renderHTML({ HTMLAttributes }: { HTMLAttributes: Record<string, unknown> }) {
return [
inline ? 'span' : 'div',
mergeAttributes(HTMLAttributes, {
'data-raw-markdown': name,
'data-raw-markdown-label': badgeLabel,
class: inline ? 'raw-markdown-inline' : 'raw-markdown-block',
}),
0,
] as const
},
renderMarkdown(node: JSONContent) {
return verbatimText(node)
},
}
}
/**
* Tag names CommonMark/GFM treat as "block-starting" HTML (marked's own type-6 list see
* `_tag` in `node_modules/marked/src/rules.ts`, verified against the CommonMark spec): a block
* opening with one of these ends at its *matching closing tag*, not at the first blank line. Tags
* NOT in this list (`em`, `a`, `span`, `code`, `kbd`, ) can legitimately start an ordinary
* paragraph (`<em>hi</em> there`), so they're deliberately left to marked's own stricter, single-line
* block-HTML detection below claiming them here would risk swallowing a paragraph that merely
* starts with inline HTML.
*/
const BLOCK_HTML_TAG_NAMES = new Set([
'address',
'article',
'aside',
'base',
'basefont',
'blockquote',
'body',
'caption',
'center',
'col',
'colgroup',
'dd',
'details',
'dialog',
'dir',
'div',
'dl',
'dt',
'fieldset',
'figcaption',
'figure',
'footer',
'form',
'frame',
'frameset',
'h1',
'h2',
'h3',
'h4',
'h5',
'h6',
'head',
'header',
'hr',
'html',
'iframe',
'legend',
'li',
'link',
'main',
'menu',
'menuitem',
'meta',
'nav',
'noframes',
'ol',
'optgroup',
'option',
'p',
'param',
'search',
'section',
'summary',
'table',
'tbody',
'td',
'tfoot',
'th',
'thead',
'title',
'tr',
'track',
'ul',
])
/**
* Marked's built-in block-HTML rule ends a `<details>`/`<div>`/ block at the *first blank line*
* (CommonMark's HTML-block-type-6 rule) correct for normal rendering, but wrong for verbatim
* preservation: any real-world `<details>` with a paragraph inside would fragment into a raw chip,
* an ordinary (rendered) paragraph, and a second raw chip, stranding genuine content in between.
* This tokenizer instead scans to the tag's *matching* close via {@link findBalancedCloseEnd}, blank
* lines included, for tags in {@link BLOCK_HTML_TAG_NAMES}; anything else returns `undefined` and
* falls through to the existing `markdownTokenName: 'html'` handling below (marked's own block
* tokenizer, unchanged). Comments are matched the same way as the inline case marked's own comment
* rule already spans blank lines correctly, but routing through one path keeps the two tokenizers
* symmetric and independently testable. CommonMark allows up to 3 leading spaces before a block-HTML
* opening line, so the leading indent is split off, matched against separately, and stitched back
* onto `raw` everything after that first line (including the tag's own body) can be indented
* however the author wrote it, since the balanced scan doesn't care about column position there.
*/
function tokenizeRawHtmlBlockTag(src: string): MarkdownToken | undefined {
const indent = /^ {0,3}/.exec(src)?.[0] ?? ''
const rest = src.slice(indent.length)
const comment = RAW_HTML_COMMENT_RE.exec(rest)
if (comment) {
const raw = indent + comment[0]
return { type: 'html', raw, text: raw, block: true }
}
const open = OPEN_TAG_RE.exec(rest)
if (!open) return undefined
const tag = open[1].toLowerCase()
if (!BLOCK_HTML_TAG_NAMES.has(tag)) return undefined
// A handful of BLOCK_HTML_TAG_NAMES entries (link, meta, base, col, …) are void elements with no
// closing tag at all — treat them as complete right after the open tag (like an explicit `/>`),
// same as `tokenizeRawInlineHtml` already does via VOID_TAGS. Without this, scanning for a
// `</meta>`/`</link>` that will never legitimately appear risks grabbing unrelated later content
// (or a stray same-name mention) as if it belonged to this block.
if (open[2] || VOID_TAGS.has(tag)) {
const raw = indent + open[0]
return { type: 'html', raw, text: raw, block: true }
}
const end = findBalancedCloseEnd(rest, tag, open[0].length)
if (end < 0) return undefined
const raw = indent + rest.slice(0, end)
return { type: 'html', raw, text: raw, block: true }
}
const SKIP_BLOCK_HTML_TAGS = /^<(img|br)\b[^>]*\/?>\s*$/i
export const RawHtmlBlock = Node.create({
...verbatimNodeConfig({ name: 'rawHtmlBlock', inline: false, badgeLabel: 'Raw HTML' }),
markdownTokenName: 'html',
markdownTokenizer: {
name: 'rawHtmlBlockTag',
level: 'block' as const,
// Always -1 (never claims an early interrupt point): when `start` is omitted, `@tiptap/markdown`
// auto-generates one that calls `this.createLexer()` on every paragraph-continuation check, which
// corrupts the in-progress lexer's shared state (verified directly — every other construct on the
// page silently loses its content once a tokenizer without an explicit `start` is registered).
// The other custom tokenizers below all reference this comment rather than repeating it.
//
// The tokenizer above emits `type: 'html'` explicitly, so its tokens flow into the same
// `markdownTokenName: 'html'` parse registration as marked's own block-HTML tokens below — the
// distinct `name` here only avoids colliding with marked's own built-in `html` extension.
start: () => -1,
tokenize: tokenizeRawHtmlBlockTag,
},
parseMarkdown(token: MarkdownToken) {
if (!token.block) return []
const raw = token.raw ?? token.text ?? ''
if (!raw.trim()) return []
// A lone `<img>`/`<br>` tag block — leave it to the stock path (Image node / hard break),
// matching the same exclusion `round-trip-safety.ts` used to carve out for these two tags.
if (SKIP_BLOCK_HTML_TAGS.test(raw.trim())) return []
return { type: 'rawHtmlBlock', content: verbatimParse(raw) }
},
})
const FOOTNOTE_DEF_HEAD_RE = /^ {0,3}\[\^[^\]]+\]:/
const FOOTNOTE_CONTINUATION_RE = /^ {4,}\S/
/**
* Consume a footnote definition's opening line plus any continuation lines GFM allows indented by
* 4 spaces, optionally with blank lines between them (a multi-paragraph definition). Stops at the
* first line that is neither indented nor blank, and never consumes a blank line that isn't followed
* by further continuation (that blank line belongs to whatever block comes next).
*/
function tokenizeFootnoteDef(src: string): MarkdownToken | undefined {
const lines = src.split('\n')
if (!FOOTNOTE_DEF_HEAD_RE.test(lines[0])) return undefined
let lineCount = 1
while (lineCount < lines.length) {
const line = lines[lineCount]
if (FOOTNOTE_CONTINUATION_RE.test(line)) {
lineCount += 1
continue
}
if (line === '' && FOOTNOTE_CONTINUATION_RE.test(lines[lineCount + 1] ?? '')) {
lineCount += 2
continue
}
break
}
const raw = lines.slice(0, lineCount).join('\n')
return { type: 'footnoteDef', raw, text: raw }
}
/** Footnote definition (`[^id]: the note`, with optional 4-space-indented continuation lines)
* marked has no footnote syntax at all, so without this tokenizer the definition is swallowed as a
* plain paragraph and the reference/definition link is lost. */
export const FootnoteDef = Node.create({
...verbatimNodeConfig({ name: 'footnoteDef', inline: false, badgeLabel: 'Footnote' }),
markdownTokenName: 'footnoteDef',
markdownTokenizer: {
name: 'footnoteDef',
level: 'block' as const,
// See the comment on `RawHtmlBlock`'s `start` — omitting it corrupts the shared lexer. The cost
// here is narrow and safe: a footnote def sharing a line-run with the preceding paragraph (no
// blank line between them) is picked up on the next block boundary instead of interrupting early.
start: () => -1,
tokenize: tokenizeFootnoteDef,
},
parseMarkdown(token: MarkdownToken) {
const raw = token.raw ?? token.text ?? ''
if (!raw) return []
return { type: 'footnoteDef', content: verbatimParse(raw) }
},
})
const FOOTNOTE_REF_RE = /^\[\^[^\]]+\]/
/** Footnote reference (`text[^id]`) — verbatim passthrough, same rationale as {@link FootnoteDef}. */
export const FootnoteRef = Node.create({
...verbatimNodeConfig({ name: 'footnoteRef', inline: true, badgeLabel: 'Footnote ref' }),
markdownTokenName: 'footnoteRef',
markdownTokenizer: {
name: 'footnoteRef',
level: 'inline' as const,
// See the comment on `RawHtmlBlock`'s `start` — omitting it corrupts the shared lexer.
start: () => -1,
tokenize(src: string) {
const match = FOOTNOTE_REF_RE.exec(src)
if (!match) return undefined
return { type: 'footnoteRef', raw: match[0], text: match[0] }
},
},
parseMarkdown(token: MarkdownToken) {
const raw = token.raw ?? token.text ?? ''
if (!raw) return []
return { type: 'footnoteRef', content: verbatimParse(raw) }
},
})
/**
* Attempt to consume an inline HTML comment or a tag (with its matching close tag, or as a single
* void/self-closing element) starting at `src[0]`. Returns `undefined` for a tag this schema
* already has a real mark/node for ({@link HANDLED_INLINE_TAGS}) so it keeps parsing normally, and
* for an unterminated open tag (rare/malformed input falls back to the stock, lossy behavior
* rather than risk mis-consuming the rest of the document).
*/
function tokenizeRawInlineHtml(src: string): MarkdownToken | undefined {
const comment = RAW_HTML_COMMENT_RE.exec(src)
if (comment) return { type: 'rawInlineHtml', raw: comment[0], text: comment[0] }
const open = OPEN_TAG_RE.exec(src)
if (!open) return undefined
const tag = open[1].toLowerCase()
if (HANDLED_INLINE_TAGS.has(tag)) return undefined
if (open[2] || VOID_TAGS.has(tag)) {
return { type: 'rawInlineHtml', raw: open[0], text: open[0] }
}
const end = findBalancedCloseEnd(src, tag, open[0].length)
if (end < 0) return undefined
const raw = src.slice(0, end)
return { type: 'rawInlineHtml', raw, text: raw }
}
/** Inline raw HTML `<kbd>`, `<sub>`, `<mark>`, `<span>`, `<u>` (no Underline mark is registered),
* and any other tag this schema has no mark/node for, plus an inline-position HTML comment. Marked
* classifies inline HTML as its own `'html'` token type, and `@tiptap/markdown`'s inline parser
* hardcodes handling for that type *before* checking its extension registry (unlike block tokens)
* so claiming it here needs a custom tokenizer, registered under a different token name
* (`rawInlineHtml`) so it's never confused with the stock `'html'` inline path. marked.js runs
* custom extension tokenizers before its own built-ins at both block and inline level (see
* `blockTokens`/`inlineTokens` in `node_modules/marked/lib/marked.esm.js`), so this reliably wins
* the race against marked's default inline HTML/tag tokenizer. */
export const RawInlineHtml = Node.create({
...verbatimNodeConfig({ name: 'rawInlineHtml', inline: true, badgeLabel: 'Raw HTML' }),
markdownTokenName: 'rawInlineHtml',
markdownTokenizer: {
name: 'rawInlineHtml',
level: 'inline' as const,
// See the comment on `RawHtmlBlock`'s `start` — omitting it corrupts the shared lexer.
start: () => -1,
tokenize: tokenizeRawInlineHtml,
},
parseMarkdown(token: MarkdownToken) {
const raw = token.raw ?? token.text ?? ''
if (!raw) return []
return { type: 'rawInlineHtml', content: verbatimParse(raw) }
},
})
@@ -1,479 +1,6 @@
import type { JSONContent, MarkdownToken } from '@tiptap/core'
import { mergeAttributes, Node } from '@tiptap/core'
import type { ReactNodeViewProps } from '@tiptap/react'
import { NodeViewContent, NodeViewWrapper, ReactNodeViewRenderer } from '@tiptap/react'
/**
* Constructs the schema has no node/mark for: raw HTML blocks (`<div>`, `<details>`, ), HTML
* comments, and footnotes. Before this file, all four made the *entire* document open read-only
* (see {@link isRoundTripSafe in ./round-trip-safety}) because the stock pipeline silently drops
* or mangles them. Each node below instead holds the exact source text as its content and
* re-emits it byte-for-byte on serialize the same "hold raw source, re-render specially" shape
* `MarkdownCodeBlock` uses for Mermaid (see `./code-block.tsx`), just without the diagram render.
*
* Inline tags already covered by a real mark/node `em`/`i`, `strong`/`b`, `s`/`del`/`strike`,
* `code`, `a`, `br`, `img` are deliberately excluded from {@link RawInlineHtml} so they keep
* parsing into their proper mark (e.g. `<em>x</em>` italic) instead of freezing as raw source.
*/
const HANDLED_INLINE_TAGS = new Set([
'br',
'img',
'em',
'i',
'strong',
'b',
's',
'del',
'strike',
'code',
'a',
])
const VOID_TAGS = new Set([
'area',
'base',
'br',
'col',
'embed',
'hr',
'img',
'input',
'link',
'meta',
'param',
'source',
'track',
'wbr',
])
function verbatimText(node: JSONContent): string {
return (node.content ?? []).map((child) => child.text ?? '').join('')
}
const RAW_HTML_COMMENT_RE = /^<!--[\s\S]*?-->/
/**
* One HTML attribute: `name` or `name="value"`/`name='value'`/`name=bare`. The quoted-value
* alternatives are what matter `[^"]*`/`[^']*` consume a literal `>` inside the quotes as part of
* the value, so an attribute like `data-example="a > b"` is treated as one unit instead of ending
* the tag match at the internal `>`.
*/
const ATTRS_RE_SOURCE = String.raw`(?:\s+[^\s"'=<>\`]+(?:\s*=\s*(?:"[^"]*"|'[^']*'|[^\s"'=<>\`]+))?)*`
/** Matches one opening HTML tag, attributes included (see {@link ATTRS_RE_SOURCE}). Group 1 is the
* tag name, group 2 is the self-closing `/` if present shared by inline and block tokenizing. */
const OPEN_TAG_RE = new RegExp(`^<([a-z][\\w-]*)\\b${ATTRS_RE_SOURCE}\\s*(/)?>`, 'i')
/** A fenced block's opening/closing marker may sit inside a blockquote (each line prefixed with
* up to 3 spaces then one or more `>` markers, each optionally followed by a space) and/or be
* independently indented up to 3 spaces with no blockquote at all (CommonMark's own fence-indent
* tolerance matches `FENCE_OPEN`/`FENCE_CLOSE` in `./markdown-parse.ts`) matched on both the
* open and close fence line so `> \`\`\`` and ` \`\`\`` both mask correctly. */
const FENCE_PREFIX_SOURCE = '(?:[ ]{0,3}>[ ]?)*[ ]{0,3}'
/** Same as {@link RAW_HTML_COMMENT_RE} but not anchored to the start of the string used by
* {@link maskCodeRegions} to find a comment anywhere in the scanned text, not just at position 0. */
const HTML_COMMENT_ANYWHERE_RE = /<!--[\s\S]*?-->/g
/**
* Mask fenced code blocks, inline code spans, and HTML comments with same-length filler (newlines
* kept, everything else replaced with a space) so a tag-like mention *inside one of these*
* `` `</details>` ``, a fenced example showing HTML syntax, or a comment documenting the tag
* (`<!-- see </div> below -->`) is never mistaken for a real balancing tag while scanning. Mirrors
* the fenced/inline patterns `stripCode` in `./round-trip-safety.ts` matches (extended to also
* tolerate an indented and/or blockquoted fence marker via {@link FENCE_PREFIX_SOURCE}, since a raw
* HTML block can itself be indented or quoted), but preserves length/position (masks in place)
* instead of deleting, so match indices still map onto the original, unmodified `src` the caller
* slices from.
*/
function maskCodeRegions(src: string): string {
const fenceRe = new RegExp(
`^${FENCE_PREFIX_SOURCE}([\`~]{3,})[^\\n]*\\n[\\s\\S]*?^${FENCE_PREFIX_SOURCE}\\1[\`~]*[ \\t]*$`,
'gm'
)
return src
.replace(fenceRe, (m) => m.replace(/[^\n]/g, ' '))
.replace(/`+[^`\n]*`+/g, (m) => ' '.repeat(m.length))
.replace(HTML_COMMENT_ANYWHERE_RE, (m) => m.replace(/[^\n]/g, ' '))
}
/**
* Find the end of the close tag that balances the open tag of `tag` ending at `src[0, fromIndex)`,
* tracking nesting depth from `fromIndex` onward so `<span>outer <span>inner</span></span>` consumes
* both levels instead of stopping at the first (inner) `</span>`. Returns -1 if unterminated. A
* nested self-closing same-name tag (`<span/>`) is skipped it neither opens nor closes a level.
* Shared by the inline tokenizer (single line) and the block tokenizer (spans blank lines).
*
* Scans a {@link maskCodeRegions}-masked copy of `src` so a tag name mentioned inside code doesn't
* count as real markup this narrows, but can't eliminate, the inherent ambiguity of regex-based
* (non-DOM) tag matching: a *bare, unescaped* mention of the same tag name in plain prose (not in
* code) is indistinguishable from a real closing tag here, exactly as it would be to a real HTML
* parser given the same ambiguous input (there is no valid way to "escape" a literal `</tag>` inside
* real HTML content other than an entity or code region). Verified this can't lose data even in that
* case the result still reaches a stable fixpoint on save, just restructured matching this file's
* "reject on doubt, but never require doubt-free input" gate (`isRoundTripSafe`).
*/
function findBalancedCloseEnd(src: string, tag: string, fromIndex: number): number {
const masked = maskCodeRegions(src)
const tagRe = new RegExp(`<(/?)${tag}\\b${ATTRS_RE_SOURCE}\\s*(/)?>`, 'gi')
tagRe.lastIndex = fromIndex
let depth = 1
for (let match = tagRe.exec(masked); match; match = tagRe.exec(masked)) {
const isClose = match[1] === '/'
const isSelfClosing = Boolean(match[2])
if (isSelfClosing) continue
if (isClose) {
depth -= 1
if (depth === 0) return match.index + match[0].length
} else {
depth += 1
}
}
return -1
}
/**
* Marked's own block tokenizer greedily consumes the blank-line run *after* an HTML block/comment
* or a def line as part of that token's own `raw` (the same behavior `PipeSafeTable` in
* `./extensions.ts` works around for tables) storing it verbatim would double it up with the
* block joiner's own separator, growing by two newlines on every save. Block-level callers trim it;
* inline callers never carry one (inline tokens can't span a blank line), so trimming is a no-op there.
*/
function verbatimParse(raw: string): JSONContent[] {
const trimmed = raw.replace(/\n+$/, '')
return trimmed ? [{ type: 'text', text: trimmed }] : []
}
interface VerbatimNodeOptions {
name: string
/** Whether this node sits among block content (own line) or inline content (mid-paragraph). */
inline: boolean
badgeLabel: string
}
/**
* Shared shape for a node that holds a markdown construct's exact source text and re-emits it
* unchanged parsing and rendering never inspect or transform the text, so there is nothing for
* these constructs to lose. `markdownTokenName`/`parseMarkdown`/`renderMarkdown` are read directly
* off the returned config by `@tiptap/markdown`'s `MarkdownManager` (see
* `node_modules/@tiptap/markdown/src/MarkdownManager.ts`), independent of the node's `name`.
*/
function verbatimNodeConfig({ name, inline, badgeLabel }: VerbatimNodeOptions) {
return {
name,
inline,
group: inline ? 'inline' : 'block',
content: 'text*',
marks: '',
code: true,
defining: !inline,
// Block verbatim nodes hold exact source text; `isolating` stops a boundary Backspace/Delete from
// joining across their edge, which would otherwise merge their raw markdown into an adjacent
// paragraph as HTML-escaped prose and destroy the node (silent data loss on save).
isolating: !inline,
selectable: true,
atom: false,
parseHTML() {
return [
{
tag: `${inline ? 'span' : 'div'}[data-raw-markdown="${name}"]`,
preserveWhitespace: 'full' as const,
},
]
},
renderHTML({ HTMLAttributes }: { HTMLAttributes: Record<string, unknown> }) {
return [
inline ? 'span' : 'div',
mergeAttributes(HTMLAttributes, {
'data-raw-markdown': name,
'data-raw-markdown-label': badgeLabel,
class: inline ? 'raw-markdown-inline' : 'raw-markdown-block',
}),
0,
] as const
},
renderMarkdown(node: JSONContent) {
return verbatimText(node)
},
}
}
/**
* Tag names CommonMark/GFM treat as "block-starting" HTML (marked's own type-6 list see
* `_tag` in `node_modules/marked/src/rules.ts`, verified against the CommonMark spec): a block
* opening with one of these ends at its *matching closing tag*, not at the first blank line. Tags
* NOT in this list (`em`, `a`, `span`, `code`, `kbd`, ) can legitimately start an ordinary
* paragraph (`<em>hi</em> there`), so they're deliberately left to marked's own stricter, single-line
* block-HTML detection below claiming them here would risk swallowing a paragraph that merely
* starts with inline HTML.
*/
const BLOCK_HTML_TAG_NAMES = new Set([
'address',
'article',
'aside',
'base',
'basefont',
'blockquote',
'body',
'caption',
'center',
'col',
'colgroup',
'dd',
'details',
'dialog',
'dir',
'div',
'dl',
'dt',
'fieldset',
'figcaption',
'figure',
'footer',
'form',
'frame',
'frameset',
'h1',
'h2',
'h3',
'h4',
'h5',
'h6',
'head',
'header',
'hr',
'html',
'iframe',
'legend',
'li',
'link',
'main',
'menu',
'menuitem',
'meta',
'nav',
'noframes',
'ol',
'optgroup',
'option',
'p',
'param',
'search',
'section',
'summary',
'table',
'tbody',
'td',
'tfoot',
'th',
'thead',
'title',
'tr',
'track',
'ul',
])
/**
* Marked's built-in block-HTML rule ends a `<details>`/`<div>`/ block at the *first blank line*
* (CommonMark's HTML-block-type-6 rule) correct for normal rendering, but wrong for verbatim
* preservation: any real-world `<details>` with a paragraph inside would fragment into a raw chip,
* an ordinary (rendered) paragraph, and a second raw chip, stranding genuine content in between.
* This tokenizer instead scans to the tag's *matching* close via {@link findBalancedCloseEnd}, blank
* lines included, for tags in {@link BLOCK_HTML_TAG_NAMES}; anything else returns `undefined` and
* falls through to the existing `markdownTokenName: 'html'` handling below (marked's own block
* tokenizer, unchanged). Comments are matched the same way as the inline case marked's own comment
* rule already spans blank lines correctly, but routing through one path keeps the two tokenizers
* symmetric and independently testable. CommonMark allows up to 3 leading spaces before a block-HTML
* opening line, so the leading indent is split off, matched against separately, and stitched back
* onto `raw` everything after that first line (including the tag's own body) can be indented
* however the author wrote it, since the balanced scan doesn't care about column position there.
*/
function tokenizeRawHtmlBlockTag(src: string): MarkdownToken | undefined {
const indent = /^ {0,3}/.exec(src)?.[0] ?? ''
const rest = src.slice(indent.length)
const comment = RAW_HTML_COMMENT_RE.exec(rest)
if (comment) {
const raw = indent + comment[0]
return { type: 'html', raw, text: raw, block: true }
}
const open = OPEN_TAG_RE.exec(rest)
if (!open) return undefined
const tag = open[1].toLowerCase()
if (!BLOCK_HTML_TAG_NAMES.has(tag)) return undefined
// A handful of BLOCK_HTML_TAG_NAMES entries (link, meta, base, col, …) are void elements with no
// closing tag at all — treat them as complete right after the open tag (like an explicit `/>`),
// same as `tokenizeRawInlineHtml` already does via VOID_TAGS. Without this, scanning for a
// `</meta>`/`</link>` that will never legitimately appear risks grabbing unrelated later content
// (or a stray same-name mention) as if it belonged to this block.
if (open[2] || VOID_TAGS.has(tag)) {
const raw = indent + open[0]
return { type: 'html', raw, text: raw, block: true }
}
const end = findBalancedCloseEnd(rest, tag, open[0].length)
if (end < 0) return undefined
const raw = indent + rest.slice(0, end)
return { type: 'html', raw, text: raw, block: true }
}
const SKIP_BLOCK_HTML_TAGS = /^<(img|br)\b[^>]*\/?>\s*$/i
export const RawHtmlBlock = Node.create({
...verbatimNodeConfig({ name: 'rawHtmlBlock', inline: false, badgeLabel: 'Raw HTML' }),
markdownTokenName: 'html',
markdownTokenizer: {
name: 'rawHtmlBlockTag',
level: 'block' as const,
// Always -1 (never claims an early interrupt point): when `start` is omitted, `@tiptap/markdown`
// auto-generates one that calls `this.createLexer()` on every paragraph-continuation check, which
// corrupts the in-progress lexer's shared state (verified directly — every other construct on the
// page silently loses its content once a tokenizer without an explicit `start` is registered).
// The other custom tokenizers below all reference this comment rather than repeating it.
//
// The tokenizer above emits `type: 'html'` explicitly, so its tokens flow into the same
// `markdownTokenName: 'html'` parse registration as marked's own block-HTML tokens below — the
// distinct `name` here only avoids colliding with marked's own built-in `html` extension.
start: () => -1,
tokenize: tokenizeRawHtmlBlockTag,
},
parseMarkdown(token: MarkdownToken) {
if (!token.block) return []
const raw = token.raw ?? token.text ?? ''
if (!raw.trim()) return []
// A lone `<img>`/`<br>` tag block — leave it to the stock path (Image node / hard break),
// matching the same exclusion `round-trip-safety.ts` used to carve out for these two tags.
if (SKIP_BLOCK_HTML_TAGS.test(raw.trim())) return []
return { type: 'rawHtmlBlock', content: verbatimParse(raw) }
},
})
const FOOTNOTE_DEF_HEAD_RE = /^ {0,3}\[\^[^\]]+\]:/
const FOOTNOTE_CONTINUATION_RE = /^ {4,}\S/
/**
* Consume a footnote definition's opening line plus any continuation lines GFM allows indented by
* 4 spaces, optionally with blank lines between them (a multi-paragraph definition). Stops at the
* first line that is neither indented nor blank, and never consumes a blank line that isn't followed
* by further continuation (that blank line belongs to whatever block comes next).
*/
function tokenizeFootnoteDef(src: string): MarkdownToken | undefined {
const lines = src.split('\n')
if (!FOOTNOTE_DEF_HEAD_RE.test(lines[0])) return undefined
let lineCount = 1
while (lineCount < lines.length) {
const line = lines[lineCount]
if (FOOTNOTE_CONTINUATION_RE.test(line)) {
lineCount += 1
continue
}
if (line === '' && FOOTNOTE_CONTINUATION_RE.test(lines[lineCount + 1] ?? '')) {
lineCount += 2
continue
}
break
}
const raw = lines.slice(0, lineCount).join('\n')
return { type: 'footnoteDef', raw, text: raw }
}
/** Footnote definition (`[^id]: the note`, with optional 4-space-indented continuation lines)
* marked has no footnote syntax at all, so without this tokenizer the definition is swallowed as a
* plain paragraph and the reference/definition link is lost. */
export const FootnoteDef = Node.create({
...verbatimNodeConfig({ name: 'footnoteDef', inline: false, badgeLabel: 'Footnote' }),
markdownTokenName: 'footnoteDef',
markdownTokenizer: {
name: 'footnoteDef',
level: 'block' as const,
// See the comment on `RawHtmlBlock`'s `start` — omitting it corrupts the shared lexer. The cost
// here is narrow and safe: a footnote def sharing a line-run with the preceding paragraph (no
// blank line between them) is picked up on the next block boundary instead of interrupting early.
start: () => -1,
tokenize: tokenizeFootnoteDef,
},
parseMarkdown(token: MarkdownToken) {
const raw = token.raw ?? token.text ?? ''
if (!raw) return []
return { type: 'footnoteDef', content: verbatimParse(raw) }
},
})
const FOOTNOTE_REF_RE = /^\[\^[^\]]+\]/
/** Footnote reference (`text[^id]`) — verbatim passthrough, same rationale as {@link FootnoteDef}. */
export const FootnoteRef = Node.create({
...verbatimNodeConfig({ name: 'footnoteRef', inline: true, badgeLabel: 'Footnote ref' }),
markdownTokenName: 'footnoteRef',
markdownTokenizer: {
name: 'footnoteRef',
level: 'inline' as const,
// See the comment on `RawHtmlBlock`'s `start` — omitting it corrupts the shared lexer.
start: () => -1,
tokenize(src: string) {
const match = FOOTNOTE_REF_RE.exec(src)
if (!match) return undefined
return { type: 'footnoteRef', raw: match[0], text: match[0] }
},
},
parseMarkdown(token: MarkdownToken) {
const raw = token.raw ?? token.text ?? ''
if (!raw) return []
return { type: 'footnoteRef', content: verbatimParse(raw) }
},
})
/**
* Attempt to consume an inline HTML comment or a tag (with its matching close tag, or as a single
* void/self-closing element) starting at `src[0]`. Returns `undefined` for a tag this schema
* already has a real mark/node for ({@link HANDLED_INLINE_TAGS}) so it keeps parsing normally, and
* for an unterminated open tag (rare/malformed input falls back to the stock, lossy behavior
* rather than risk mis-consuming the rest of the document).
*/
function tokenizeRawInlineHtml(src: string): MarkdownToken | undefined {
const comment = RAW_HTML_COMMENT_RE.exec(src)
if (comment) return { type: 'rawInlineHtml', raw: comment[0], text: comment[0] }
const open = OPEN_TAG_RE.exec(src)
if (!open) return undefined
const tag = open[1].toLowerCase()
if (HANDLED_INLINE_TAGS.has(tag)) return undefined
if (open[2] || VOID_TAGS.has(tag)) {
return { type: 'rawInlineHtml', raw: open[0], text: open[0] }
}
const end = findBalancedCloseEnd(src, tag, open[0].length)
if (end < 0) return undefined
const raw = src.slice(0, end)
return { type: 'rawInlineHtml', raw, text: raw }
}
/** Inline raw HTML `<kbd>`, `<sub>`, `<mark>`, `<span>`, `<u>` (no Underline mark is registered),
* and any other tag this schema has no mark/node for, plus an inline-position HTML comment. Marked
* classifies inline HTML as its own `'html'` token type, and `@tiptap/markdown`'s inline parser
* hardcodes handling for that type *before* checking its extension registry (unlike block tokens)
* so claiming it here needs a custom tokenizer, registered under a different token name
* (`rawInlineHtml`) so it's never confused with the stock `'html'` inline path. marked.js runs
* custom extension tokenizers before its own built-ins at both block and inline level (see
* `blockTokens`/`inlineTokens` in `node_modules/marked/lib/marked.esm.js`), so this reliably wins
* the race against marked's default inline HTML/tag tokenizer. */
export const RawInlineHtml = Node.create({
...verbatimNodeConfig({ name: 'rawInlineHtml', inline: true, badgeLabel: 'Raw HTML' }),
markdownTokenName: 'rawInlineHtml',
markdownTokenizer: {
name: 'rawInlineHtml',
level: 'inline' as const,
// See the comment on `RawHtmlBlock`'s `start` — omitting it corrupts the shared lexer.
start: () => -1,
tokenize: tokenizeRawInlineHtml,
},
parseMarkdown(token: MarkdownToken) {
const raw = token.raw ?? token.text ?? ''
if (!raw) return []
return { type: 'rawInlineHtml', content: verbatimParse(raw) }
},
})
import { FootnoteDef, RawHtmlBlock } from './raw-markdown-snippet-schema'
const BLOCK_CONTROL_CLASS =
'pointer-events-none absolute top-1.5 right-2 select-none rounded-md bg-[var(--surface-4)] px-1.5 py-0.5 text-[10px] text-[var(--text-muted)] uppercase tracking-wide opacity-0 transition-opacity group-focus-within:opacity-100 group-hover:opacity-100'
@@ -357,10 +357,13 @@
/* Borders, padding, typography, and header fill come from document-table.css the chrome shared
with the CSV/XLSX previews. Only the editor-specific bits live here: `table-layout: fixed` is
required by prosemirror-tables' column-resizing plugin, and the block margin is prose rhythm. */
required by prosemirror-tables' column-resizing plugin, the block margin is prose rhythm, and
`overflow: visible` lets a collaborator's caret name label escape the table box instead of being
clipped (fixed layout means columns can't exceed the table width, so nothing else needs clipping). */
.rich-markdown-prose table {
table-layout: fixed;
margin: 1rem 0;
overflow: visible;
}
.rich-markdown-prose th > p,
@@ -438,3 +441,122 @@
.rich-markdown-field-prose p.is-editor-empty:first-child::before {
color: var(--text-muted);
}
/*
* Collaborative carets (TipTap CollaborationCaret). The caret bar and the name
* label's background are colored inline from each collaborator's identity color
* (the same `getUserColor` mechanism the canvas cursors use); the selection is a
* translucent tint of that color (set via selectionRender). The name label shows
* while the peer is active (JS toggles `--active` on each awareness change) or on
* hover, then fades after inactivity matching Google Docs. `z-index` lifts the
* caret (and its label) above table cell backgrounds so a caret inside a table
* cell is not hidden behind adjacent cells.
*/
.rich-markdown-prose .collaboration-carets__caret {
/* Zero inline footprint: a positioned anchor with NO width/border/margin, so inserting or moving this
* inline widget never reflows the surrounding text. The visible bar, dormant cap (::before), name label,
* and hover slop (::after) are all positioned relative to this anchor and are out of the inline flow. */
position: relative;
word-break: normal;
z-index: 20;
}
/* The visible caret bar absolutely positioned so it draws over the text without occupying inline
* width. `left: -1px` centers the 2px bar on the cursor position; top/bottom span the line box height. */
.rich-markdown-prose .collaboration-carets__bar {
position: absolute;
top: -0.1em;
bottom: -0.1em;
left: -1px;
width: 2px;
background-color: var(--caret-color);
pointer-events: none;
}
/* Dormant affordance: a small cap at the top of the caret in the collaborator's color,
* shaped like a collapsed presence name tag (same `rounded-xs` + notch corner as the
* tables/canvas tags) so the whole presence system reads as one language. It signals
* "someone's here — hover for who", and fades out when the full name tag takes over
* (peer active or on hover). */
.rich-markdown-prose .collaboration-carets__caret::before {
content: "";
position: absolute;
/* Seat the cap's square bottom-left corner on the pole's top-left, flush like a flag on its pole.
* `left: -1px` matches the bar's `left: -1px` so the cap's left edge lines up with the bar's left edge;
* `top` overlaps the bar's top a hair so the notch reads as continuous, no gap. */
top: -2px;
left: -1px;
width: 8px;
height: 5px;
border-radius: 2px 2px 2px 0;
background-color: var(--caret-color);
transition: opacity 0.2s ease;
}
.rich-markdown-prose .collaboration-carets__caret--active::before,
.rich-markdown-prose .collaboration-carets__caret:hover::before {
opacity: 0;
}
/* Transparent hover hit-slop. The visible caret bar is only ~2px wide and the dormant cap is
* 8×5px, so "hover for who" (revealing the name label on a dormant caret) would be near
* impossible to trigger. This invisible strip widens the hover target across the caret's full
* height and over the cap nothing visible changes, only the pointer area. `pointer-events:
* auto` is required so it catches the hover; kept narrow so it barely intrudes on selecting
* text next to a remote caret. */
.rich-markdown-prose .collaboration-carets__caret::after {
content: "";
position: absolute;
top: -4px;
bottom: 0;
left: -4px;
right: -4px;
pointer-events: auto;
}
/* Matches the canvas cursor name tag (cursors.tsx): identity-color background with
* `--surface-1` text at `text-xs`/`font-medium`, so both presence surfaces read as
* one system. `--surface-1` is the base surface token (readable on every assigned
* identity color in both themes), not a hardcoded value. Hidden by default; the
* show/fade is driven by the `--active` class (see caret-presence.ts) and hover. */
.rich-markdown-prose .collaboration-carets__label {
position: absolute;
top: -1.4em;
left: -1px;
max-width: 10rem;
overflow: hidden;
padding: 0.1rem 0.35rem;
border-radius: 2px 2px 2px 0;
/* 11px = the `text-xs` the canvas/tables presence tags use, for pixel parity. */
font-size: 11px;
font-weight: 500;
line-height: 1.2;
white-space: nowrap;
text-overflow: ellipsis;
background-color: var(--caret-color);
color: var(--surface-1);
user-select: none;
pointer-events: none;
opacity: 0;
transition: opacity 0.2s ease;
}
.rich-markdown-prose .collaboration-carets__caret--active .collaboration-carets__label,
.rich-markdown-prose .collaboration-carets__caret:hover .collaboration-carets__label {
opacity: 1;
}
/* Near the editor's right edge the label is flipped to the caret's left so it never
* runs off (JS toggles `--flip` after measuring); mirror the tag's notch corner. */
.rich-markdown-prose .collaboration-carets__caret--flip .collaboration-carets__label {
left: auto;
right: -1px;
border-radius: 2px 2px 0 2px;
}
/* Remote text selection: a rounded translucent tint of the collaborator's identity
* color (the alpha fill is set inline by selectionRender). */
.rich-markdown-prose .collaboration-carets__selection {
border-radius: 2px;
pointer-events: none;
}
@@ -1,21 +1,38 @@
'use client'
import { memo, useEffect, useRef, useState } from 'react'
import { memo, useCallback, useEffect, useRef, useState } from 'react'
import { cn, toast } from '@sim/emcn'
import type { JSONContent } from '@tiptap/core'
import { FILE_DOC_SEED, type JoinFileDocError } from '@sim/realtime-protocol/file-doc'
import { type Extensions, generateHTML, type JSONContent } from '@tiptap/core'
import { isChangeOrigin } from '@tiptap/extension-collaboration'
import { Fragment, Slice } from '@tiptap/pm/model'
import { NodeSelection } from '@tiptap/pm/state'
import { dropPoint } from '@tiptap/pm/transform'
import type { Editor } from '@tiptap/react'
import { EditorContent, useEditor } from '@tiptap/react'
import { useRouter } from 'next/navigation'
import { useSession } from '@/lib/auth/auth-client'
import type { WorkspaceFileRecord } from '@/lib/uploads/contexts/workspace'
import { extractEmbeddedFileRef } from '@/lib/uploads/utils/embedded-image-ref'
import { isUntitledName } from '@/app/workspace/[workspaceId]/files/untitled-title'
import { useUploadWorkspaceFile } from '@/hooks/queries/workspace-files'
import type { SaveStatus } from '@/hooks/use-autosave'
import { useFileContentSource } from '@/hooks/use-file-content-source'
import { PreviewLoadingFrame } from '../preview-shared'
import { useEditableFileContent } from '../use-editable-file-content'
import {
announceAgentApplying,
clearAgentApplying,
isAgentStreamLeader,
} from './collaboration/agent-stream-leader'
import {
type AgentStreamSession,
applyAgentStreamFrame,
beginAgentStream,
endAgentStream,
} from './collaboration/apply-streamed-markdown'
import { nextCollabReadiness } from './collaboration/readiness'
import { useFileDocCollaboration } from './collaboration/use-file-doc-collaboration'
import { createMarkdownEditorExtensions } from './editor-extensions'
import { findHeadingPos } from './heading-anchors'
import {
@@ -38,12 +55,15 @@ import { LinkHoverCard } from './menus/link-hover-card'
import { TableBubbleMenu } from './menus/table-menu'
import { normalizeMarkdownContent } from './normalize-content'
import { isRoundTripSafe } from './round-trip-safety'
import { firstHeadingTitle } from './title-heading'
import '@sim/emcn/components/code/code.css'
import '../document-table.css'
import './rich-markdown-editor.css'
const PLACEHOLDER = "Write something, or press '/' for commands…"
const EXTENSIONS = createMarkdownEditorExtensions({
placeholder: "Write something, or press '/' for commands…",
placeholder: PLACEHOLDER,
embeds: true,
})
@@ -51,6 +71,9 @@ const EXTENSIONS = createMarkdownEditorExtensions({
const STREAM_REPARSE_THROTTLE_THRESHOLD = 40_000
const STREAM_REPARSE_THROTTLE_MS = 120
/** Debounce before naming a still-untitled file after its leading heading, so it fires once typing settles. */
const DERIVE_TITLE_DEBOUNCE_MS = 600
interface RichMarkdownEditorProps {
file: WorkspaceFileRecord
workspaceId: string
@@ -68,10 +91,30 @@ interface RichMarkdownEditorProps {
* applied live; a rebuild is only revealed while it extends what's shown (see the streaming tick).
*/
streamIsIncremental?: boolean
/**
* The agent edit operation driving the stream, when known (`create`/`append`/`update`/`patch`). In the
* collaborative path it decides only whether to stream mid-flight: an `update` (from-scratch rewrite) is
* HELD until settle so the open doc doesn't collapse to a partial result, while `append`/`patch`/`create`
* apply each frame.
*/
streamOperation?: string
disableStreamingAutoScroll?: boolean
previewContextKey?: string
/** Disable the `@` tag-insertion menu (existing tags still render). Defaults off — the file editor keeps tagging. */
disableTagging?: boolean
/**
* Opt this surface into live collaborative editing (Files page + the embedded chat file preview).
* Collaboration can coexist with agent streaming: while streaming, the growing content is applied to
* the shared Y.Doc as minimal CRDT diffs (see {@link applyAgentStreamFrame}) rather than a
* full-document `setContent`, so the stream stays smooth and every peer sees it live.
*/
collaborative?: boolean
/**
* Called (debounced) with the document's leading-heading text while the file is still untitled, so the
* caller can name the file after it. Omitted on read-only/non-editable surfaces. See
* {@link isUntitledName}.
*/
onDeriveTitleFromHeading?: (headingText: string) => void
}
/** Inline WYSIWYG markdown editor: agent output streams in read-only, then the same instance becomes editable on settle. */
@@ -87,10 +130,30 @@ export const RichMarkdownEditor = memo(function RichMarkdownEditor({
streamingContent,
isAgentEditing,
streamIsIncremental,
streamOperation,
disableStreamingAutoScroll = false,
previewContextKey,
disableTagging,
collaborative = false,
onDeriveTitleFromHeading,
}: RichMarkdownEditorProps) {
const { data: session, isPending: isSessionPending } = useSession()
const userId = session?.user?.id ?? ''
const userName = session?.user?.name?.trim() || 'Collaborator'
/**
* Client-autosave gate. For a NON-collaborative file this is `true` (the client owns durability and
* autosaves the markdown). For a collaborative file it stays `false`: the realtime relay persists the
* shared document to markdown server-side, so the client must never also autosave a stale keystroke
* saving over a server/copilot edit is exactly the clobber the server path closes. The child reports
* the right value up via `onCollabReadyChange`.
*
* Initialize from the `collaborative` prop (NOT unconditionally `true`): a collaborative file must
* start with autosave OFF, or a save could fire in the window before the child mounts and reports
* re-clobbering exactly what this closes. The child turns it on for the non-collaborative fallback.
*/
const [collabReady, setCollabReady] = useState(!collaborative)
const {
content,
setDraftContent,
@@ -109,9 +172,14 @@ export const RichMarkdownEditor = memo(function RichMarkdownEditor({
saveRef,
discardRef,
normalizeBaseline: normalizeMarkdownContent,
canAutosave: collabReady,
})
if (isContentLoading) return <PreviewLoadingFrame className='flex flex-1 flex-col' />
// Wait for the session too: the child decides collaboration ONCE at mount from
// `userId`, so mounting before the session resolves would latch collaboration off
// for a cold-loaded file (both users would then solo-save, last-write-wins).
if (isContentLoading || isSessionPending)
return <PreviewLoadingFrame className='flex flex-1 flex-col' />
if (hasContentError) {
return (
@@ -129,12 +197,18 @@ export const RichMarkdownEditor = memo(function RichMarkdownEditor({
content={content}
isStreaming={isStreamInteractionLocked}
canEdit={canEdit}
userId={userId}
userName={userName}
autoFocus={autoFocus}
streamIsIncremental={streamIsIncremental}
streamOperation={streamOperation}
disableStreamingAutoScroll={disableStreamingAutoScroll}
disableTagging={disableTagging}
collaborative={collaborative}
onChange={setDraftContent}
onSaveShortcut={saveImmediately}
onCollabReadyChange={setCollabReady}
onDeriveTitleFromHeading={onDeriveTitleFromHeading}
/>
)
})
@@ -147,13 +221,24 @@ interface LoadedRichMarkdownEditorProps {
/** True while agent output is streaming in: the editor renders it read-only and syncs each chunk. */
isStreaming: boolean
canEdit: boolean
/** Current user id + display name, for the collaborative caret identity. */
userId: string
userName: string
autoFocus?: boolean
/** See {@link RichMarkdownEditorProps.streamIsIncremental}. */
streamIsIncremental?: boolean
/** See {@link RichMarkdownEditorProps.streamOperation}. */
streamOperation?: string
disableStreamingAutoScroll?: boolean
disableTagging?: boolean
/** See {@link RichMarkdownEditorProps.collaborative}. */
collaborative?: boolean
onChange: (markdown: string) => void
onSaveShortcut: () => Promise<void>
/** Reports whether the collaborative document is synced+seeded (autosave gate). */
onCollabReadyChange: (ready: boolean) => void
/** See {@link RichMarkdownEditorProps.onDeriveTitleFromHeading}. */
onDeriveTitleFromHeading?: (headingText: string) => void
}
interface SettledContent {
@@ -173,12 +258,18 @@ export function LoadedRichMarkdownEditor({
content,
isStreaming,
canEdit,
userId,
userName,
autoFocus,
streamIsIncremental,
streamOperation,
disableStreamingAutoScroll,
disableTagging,
collaborative = false,
onChange,
onSaveShortcut,
onCollabReadyChange,
onDeriveTitleFromHeading,
}: LoadedRichMarkdownEditorProps) {
/** Whether this editor mounted mid-stream — if so it starts empty and syncs streamed chunks until settle. */
const streamingAtMountRef = useRef(isStreaming)
@@ -188,11 +279,62 @@ export function LoadedRichMarkdownEditor({
if (!streamingAtMountRef.current && settledRef.current === null) {
settledRef.current = lockSettled(content)
}
const isEditable = canEdit && !isStreaming && (settledRef.current?.verdict ?? false)
/**
* Collaboration is decided once at mount from synchronously-available inputs
* (`settledRef` is set just above) via `useState`-init, and never changes TipTap
* fixes the extension set at editor creation, so it cannot turn on later. Enabled on a
* `collaborative` surface (the Files page or the embedded chat file preview) for an editable,
* round-trip-safe workspace document with a known user, as long as it is not ALREADY streaming at
* mount (`!streamingAtMountRef.current`). An agent stream that begins AFTER mount is applied as CRDT
* diffs into the live doc, so collaboration and streaming coexist (see the streaming effect below).
*/
const [collaborationEnabled] = useState(
() =>
collaborative &&
canEdit &&
!streamingAtMountRef.current &&
(settledRef.current?.verdict ?? false) &&
Boolean(userId) &&
(file.storageContext ?? 'workspace') === 'workspace'
)
/**
* Whether the collaborative document is safe to edit + persist: synced and seeded.
* Starts `false` for a collaborative document so the editor is read-only and
* autosave gated until the shared content has arrived (a user must not type into an
* empty, unsynced doc, which the seed would then discard) and `true` for a local one.
*/
const [collabReady, setCollabReady] = useState(!collaborationEnabled)
const isEditable =
canEdit && !isStreaming && (settledRef.current?.verdict ?? false) && collabReady
/** Seed the doc once via lazy init — chunked parse is linear vs the editor's ~O(n²) whole-body markdown parse. */
const collaboration = useFileDocCollaboration({
fileId: file.id,
userId,
userName,
enabled: collaborationEnabled,
})
/**
* Initial editor content. When collaborating, the Y.Doc is the source of truth
* start empty and let the server-seeded Yjs sync fill it (below); otherwise seed from the
* parsed markdown (chunked parse is linear vs the editor's ~O(n²) whole-body parse).
*/
const [initialContent] = useState<JSONContent | string>(() =>
streamingAtMountRef.current ? '' : parseMarkdownToDoc(splitFrontmatter(content).body)
streamingAtMountRef.current || collaborationEnabled
? ''
: parseMarkdownToDoc(splitFrontmatter(content).body)
)
/**
* A read-only placeholder rendered from the already-fetched markdown while a collaborative doc waits
* for its server seed, so the pane shows content instantly instead of blocking blank on the socket
* round-trip (the seed IS the same markdown, so the swap on {@link collabReady} is seamless). Static
* HTML it holds no editor, doc, or awareness, so it structurally cannot write to the Y.Doc, which
* is the invariant that keeps seeding out of the client (a client seed duplicates the doc).
*/
const [placeholderHtml] = useState<string | null>(() =>
collaborationEnabled
? generateHTML(parseMarkdownToDoc(splitFrontmatter(content).body), EXTENSIONS)
: null
)
/**
* The body currently shown in the editor: seeded from a settled mount, updated on local edits (via
@@ -203,16 +345,55 @@ export function LoadedRichMarkdownEditor({
const lastSyncedBodyRef = useRef<string | null>(
streamingAtMountRef.current ? null : splitFrontmatter(content).body
)
/**
* The body the AGENT last applied into the collaborative doc a dedup guard for the collab streaming
* tick, so an unchanged frame skips a redundant shadow reconcile/reparse. Written ONLY by the streaming
* tick (never by `onUpdate`), and reset to `null` on settle for the next stream. It is NOT a string-prefix
* baseline: the mid-stream hold is decided by operation (`update` waits for settle), not by comparing the
* raw preview against the editor's canonical markdown.
*/
const lastStreamedBodyRef = useRef<string | null>(null)
const onChangeRef = useRef(onChange)
onChangeRef.current = onChange
const onSaveShortcutRef = useRef(onSaveShortcut)
onSaveShortcutRef.current = onSaveShortcut
/**
* The frontmatter to re-attach to the body on save. For a collaborative doc it lives in the CRDT
* (config map, seeded/updated server-side), so a server edit that changes it is reflected rather
* than reverted by this editor's stale open-time copy; falls back to the locked `settledRef` copy
* before the seed lands and for non-collaborative documents.
*/
const resolveSaveFrontmatter = useCallback((): string => {
const fromDoc = collaboration?.doc
.getMap(FILE_DOC_SEED.configMap)
.get(FILE_DOC_SEED.frontmatterKey)
if (typeof fromDoc === 'string') return fromDoc
return settledRef.current?.frontmatter ?? ''
}, [collaboration])
/**
* While the file is still unnamed, name it after its leading heading: `onDeriveTitleFromHeading` is
* called (debounced) so the caller can rename the file, and `fileNameRef` lets the onUpdate handler
* read the current name without re-subscribing. See {@link isUntitledName}.
*/
const onDeriveTitleFromHeadingRef = useRef(onDeriveTitleFromHeading)
onDeriveTitleFromHeadingRef.current = onDeriveTitleFromHeading
const fileNameRef = useRef(file.name)
fileNameRef.current = file.name
const deriveTitleTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null)
/**
* Read in the RAF tick so an already-scheduled tick still sees the latest edit kind (it can change
* between sessions within one turn, e.g. an append followed by a rewrite).
*/
const streamIsIncrementalRef = useRef(streamIsIncremental)
streamIsIncrementalRef.current = streamIsIncremental
const streamOperationRef = useRef(streamOperation)
streamOperationRef.current = streamOperation
/** The live agent-stream shadow replica, held for the current stream and freed on settle/unmount. */
const agentStreamSessionRef = useRef<AgentStreamSession | null>(null)
/** True once this client has announced candidacy in the agent-stream election for the current stream. */
const agentAnnouncedRef = useRef(false)
const router = useRouter()
const routerRef = useRef(router)
routerRef.current = router
@@ -290,8 +471,27 @@ export function LoadedRichMarkdownEditor({
}
}
/**
* Extensions: the shared module set for the local path, or a per-instance set
* carrying this document's Collaboration + CollaborationCaret. Built once (collab
* is decided at mount), since `useEditor` fixes the extension set at creation.
*/
const [extensions] = useState<Extensions>(() =>
collaboration
? createMarkdownEditorExtensions({
placeholder: PLACEHOLDER,
embeds: true,
collaboration: {
doc: collaboration.doc,
awareness: collaboration.awareness,
user: collaboration.user,
},
})
: EXTENSIONS
)
const editor = useEditor({
extensions: EXTENSIONS,
extensions,
editable: isEditable,
enablePasteRules: false,
autofocus: streamingAtMountRef.current ? false : autoFocus ? 'end' : false,
@@ -442,14 +642,188 @@ export function LoadedRichMarkdownEditor({
return false
},
},
onUpdate: ({ editor }) => {
onUpdate: ({ editor, transaction }) => {
const md = postProcessSerializedMarkdown(editor.getMarkdown())
lastSyncedBodyRef.current = md
onChangeRef.current(applyFrontmatter(settledRef.current?.frontmatter ?? '', md))
onChangeRef.current(applyFrontmatter(resolveSaveFrontmatter(), md))
// While the file is still untitled, name it after its leading heading once typing settles — but
// only for the LOCAL user's own edits. `isChangeOrigin` is true for a remote Yjs change (a peer
// typing); bail BEFORE touching the timer so a remote edit never cancels or reschedules the local
// user's pending rename (and every client doesn't schedule the same rename from a peer's
// not-yet-synced heading). It is false for local edits and non-collaborative surfaces.
if (isChangeOrigin(transaction)) return
// Local edit: restart the debounce. Clearing first cancels a stale rename if the heading was
// removed/changed before it fired; the timer re-derives the title from the live doc rather than a
// value captured now, so it can never name the file after a heading the user has since changed.
// `editor.isEditable` is the autosave gate (canEdit + settled + collab-ready), so a view-only
// viewer or the not-yet-editable mount seed never schedules a rename.
if (deriveTitleTimerRef.current) clearTimeout(deriveTitleTimerRef.current)
if (
!editor.isEditable ||
!isUntitledName(fileNameRef.current) ||
firstHeadingTitle(editor.state.doc) === null
)
return
deriveTitleTimerRef.current = setTimeout(() => {
const liveEditor = editorInstanceRef.current
if (!liveEditor || !liveEditor.isEditable || !isUntitledName(fileNameRef.current)) return
const title = firstHeadingTitle(liveEditor.state.doc)
if (title) onDeriveTitleFromHeadingRef.current?.(title)
}, DERIVE_TITLE_DEBOUNCE_MS)
},
})
editorInstanceRef.current = editor
useEffect(
() => () => {
if (deriveTitleTimerRef.current) clearTimeout(deriveTitleTimerRef.current)
},
[]
)
/**
* The loaded markdown to seed the shared doc from, held by pointer so the parse
* runs once at seed time rather than every render.
*/
const seedContentRef = useRef(content)
seedContentRef.current = content
/**
* The collaborative document lifecycle. In one effect because the three concerns
* are one state machine keyed off the same provider events:
* - **observe** readiness: the server seeds the doc authoritatively (content +
* `initialContentLoaded` flag in ONE Yjs update), so the client only watches for
* synced AND seeded it never imports content itself on the happy path;
* - **gate** the parent's autosave until the doc is synced AND seeded, so an
* empty/still-syncing doc can never overwrite the real file's markdown mirror;
* - **fall back** on a fatal join: seed the loaded content so it is SHOWN, but
* leave the editor read-only + gated. Every non-retryable failure (auth, access
* denied, not found, client-id conflict) either can't save or is moot, so the
* safe fallback is a read-only view of the content rather than editable-but-
* unsavable which would silently drop the user's edits.
*
* `ready` (synced+seeded) gates BOTH the editor's editability (a user must never
* type into an empty/unsynced doc) and the parent's autosave. Non-collaborative
* documents are never gated. The server seeds the doc authoritatively (content + the
* seed flag arrive together), so the client only observes readiness. `provider.joinError`
* is latched, so a fatal rejection that fired before this subscription is not missed.
*/
useEffect(() => {
const setReady = (ready: boolean) => {
// Child-local: gates editability (a user must never type into an unsynced/unseeded doc).
setCollabReady(ready)
// Parent: gates CLIENT autosave. In a collaborative session the relay persists the doc to
// markdown server-side (debounced + on last-disconnect), so the client must NOT also autosave —
// a stale keystroke saving over a server/copilot edit is the clobber the server path closes.
// Only the non-collaborative (solo) path client-autosaves.
onCollabReadyChange(collaboration ? false : ready)
}
if (!collaboration) {
setReady(true)
return
}
const { provider, doc } = collaboration
if (!editor) {
setReady(false)
return
}
const config = doc.getMap(FILE_DOC_SEED.configMap)
// Readiness LATCHES so a post-seed `synced` flap can't re-gate a new file's agent stream — see
// {@link nextCollabReadiness} for the full rationale. `offlineSeed` marks a local (read-only) seed.
let syncedOnce = false
let offlineSeed = false
const seedFromLoaded = () => {
if (config.get(FILE_DOC_SEED.flag) === true) return
offlineSeed = true
doc.transact(() => {
editor.commands.setContent(
parseMarkdownToDoc(splitFrontmatter(seedContentRef.current).body),
{ contentType: 'json', emitUpdate: false }
)
config.set(FILE_DOC_SEED.flag, true)
})
}
if (!provider) {
setReady(false)
return
}
const report = () => {
const synced = provider.synced
const seeded = config.get(FILE_DOC_SEED.flag) === true
const next = nextCollabReadiness(syncedOnce, { synced, seeded, offlineSeed })
syncedOnce = next.syncedOnce
setReady(next.ready)
}
const onJoinError = (error: JoinFileDocError) => {
if (error.retryable === false) seedFromLoaded()
}
// A server edit that changes ONLY the frontmatter (e.g. copilot) updates the config map but not
// the body fragment, so TipTap's `onUpdate` never fires and the autosave draft would keep the
// stale open-time frontmatter — an explicit save could then revert the live change. Re-attach the
// new frontmatter to the current body and push a fresh draft whenever it changes on its own.
let lastFrontmatter = config.get(FILE_DOC_SEED.frontmatterKey)
const syncFrontmatter = () => {
const current = config.get(FILE_DOC_SEED.frontmatterKey)
if (current === lastFrontmatter) return
lastFrontmatter = current
// Null body ref ⇒ no body has synced yet (e.g. this fired before the seed's own `onUpdate`);
// that path re-attaches the frontmatter itself, so there is nothing to do here.
if (lastSyncedBodyRef.current !== null) {
onChangeRef.current(
applyFrontmatter(typeof current === 'string' ? current : '', lastSyncedBodyRef.current)
)
}
}
provider.on('synced', report)
provider.on('join-error', onJoinError)
config.observe(report)
config.observe(syncFrontmatter)
report()
if (provider.joinError) onJoinError(provider.joinError)
return () => {
provider.off('synced', report)
provider.off('join-error', onJoinError)
config.unobserve(report)
config.unobserve(syncFrontmatter)
// Report NOT ready on teardown — the safe direction. If this effect ever re-runs while mounted
// (a future dep change), briefly gating autosave off is harmless; reporting `true` here could
// ungate it while the doc is unready.
onCollabReadyChange(false)
}
}, [collaboration, editor, onCollabReadyChange, setCollabReady])
/**
* Owns editability for the collaborative lifecycle: `useEditor`'s `editable` is only the initial
* value, and the streaming/settle effect only moves content in collab mode (never toggles
* editability) so re-apply here whenever collaboration readiness (synced + seeded) or an agent
* stream flips `isEditable`.
*/
useEffect(() => {
if (!editor || !collaborationEnabled) return
if (editor.isEditable === isEditable) return
// Defer out of the render/commit phase. `isEditable` flips from collab readiness (synced + seeded),
// which is driven by a Yjs `config.observe` firing synchronously inside `Y.applyUpdate` — so this
// effect can run while React is mid-render. `setEditable` dispatches a TipTap transaction that the
// React binding commits with `flushSync`, which throws ("cannot flush while rendering") in that
// window. A microtask runs right after the current commit, before paint; re-check liveness/value
// since either can change before it fires.
let cancelled = false
queueMicrotask(() => {
if (cancelled || editor.isDestroyed) return
if (editor.isEditable !== isEditable) editor.setEditable(isEditable)
})
return () => {
cancelled = true
}
}, [editor, collaborationEnabled, isEditable])
/**
* Wire the `/Image` slash command to the hidden picker (per-editor storage, since the extension set is
* shared across instances). Reads only refs, so the handler stays stable across the editor's life.
@@ -472,6 +846,8 @@ export function LoadedRichMarkdownEditor({
const pendingStreamBodyRef = useRef<string | null>(null)
const streamRafRef = useRef<number | null>(null)
const lastStreamParseAtRef = useRef(0)
const settleRunSeqRef = useRef(0)
const pendingCollapseRef = useRef(false)
useEffect(() => {
if (!editor) return
const syncEditorBody = (body: string) => {
@@ -482,9 +858,169 @@ export function LoadedRichMarkdownEditor({
emitUpdate: false,
})
}
// Editor view mutations flush synchronously through the @tiptap/react binding (setContent mounts
// the React node views via flushSync — tiptap#3764), so calling them directly in this effect body
// throws "flushSync ... cannot flush while rendering" when the effect runs mid-render. Defer to a
// microtask (after commit, before paint). The streaming rAF tick below already runs off-render.
//
// Tag each run: a deferred mutation applies only if it is still the latest run (this effect has
// several early-return exits, so a run-token beats a per-exit cleanup flag) and the editor is
// alive. This drops a superseded run's microtask when React ran the next pass — a newer stream or
// settle — before the microtask flushed, so it can't overwrite the newer state.
const runSeq = ++settleRunSeqRef.current
const runOffRender = (mutate: () => void) => {
queueMicrotask(() => {
if (runSeq !== settleRunSeqRef.current || editor.isDestroyed) return
mutate()
})
}
// Collaborative surface: stream by applying a minimal CRDT diff into the live Y.Doc each frame
// (never `setContent`, which would replace the shared doc and wipe peers). Each diff renders
// locally and broadcasts to every peer, so the stream is smooth here and on other clients (e.g.
// the standalone Files page) alike. This branch moves content only — editability for the collab
// lifecycle is owned by the reactive effect above.
if (collaborationEnabled) {
if (isStreaming) {
wasStreamingRef.current = true
// Apply streamed diffs only after the shared doc has SEEDED (synced + seed flag, i.e.
// `collabReady`). Applying onto an unseeded (empty) doc would let the later seed CRDT-merge into
// it — transient garble. In the common case the doc is long seeded before an agent edit begins;
// in the rare stream-before-seed race we wait, and since `collabReady` is an effect dep this
// re-runs and applies once it lands (the read-only placeholder shows the base content meanwhile —
// see `showPlaceholder`).
if (!collabReady) return
// Announce candidacy in the single-writer election (see the tick) so only one tab/window applies
// this stream. The shadow is opened lazily in the tick, only when THIS client actually leads — so a
// non-leader builds none, and a client that takes leadership mid-stream (a handoff) seeds its shadow
// from the CURRENT doc, already carrying the prior leader's ops, never a stale base.
if (!agentAnnouncedRef.current) {
agentAnnouncedRef.current = true
if (collaboration) announceAgentApplying(collaboration.awareness)
}
const body = splitFrontmatter(content).body
if (body === lastStreamedBodyRef.current) return
pendingStreamBodyRef.current = body
if (streamRafRef.current !== null) return
const tick = () => {
const pending = pendingStreamBodyRef.current
if (pending === null || pending === lastStreamedBodyRef.current) {
streamRafRef.current = null
return
}
// Hold a from-scratch rewrite (`update`) until settle so the open doc doesn't collapse to a
// partial rewrite mid-stream (matching `main`). `append`/`patch`/`create` apply each frame — the
// shadow reconcile is peer-safe, and base-less `append` fragments no longer reach the client (the
// server fail-closes them), so there is nothing here to string-prefix or wipe-guard against.
if (streamOperationRef.current === 'update') {
streamRafRef.current = null
return
}
// Single-writer election: only the leader (min clientID among clients announcing they apply this
// stream) writes it into the shared doc, so multiple tabs/windows watching the same live copilot
// stream don't each insert it and duplicate content. A non-leader renders the leader's ops via
// Yjs; re-checked each frame, so a co-leader stops the moment awareness propagates. (The pick-up
// direction — a successor beginning to write after the leader tab closes — waits for the next
// content frame to run a tick; a stream that already delivered its last frame is covered by
// settle and the durable write, so at worst a brief end-of-stream display lag, never a loss.)
// Bounded residual (accepted): if two tabs start the SAME stream within the awareness-propagation
// window they briefly both lead and duplicate a frame or two — a rare, transient, never-persisted
// glitch (SYNC_NO_PERSIST keeps it out of storage; the durable edit_content write reconciles the
// final doc). Resumes are sequential (the second tab sees the first's announcement), so the common
// multi-tab case elects cleanly.
if (
collaboration &&
!isAgentStreamLeader(collaboration.awareness, collaboration.doc.clientID)
) {
// Not (or no longer) the leader: discard any shadow this client holds. A shadow only tracks
// ITS OWN reconciles, so one kept across a leadership loss goes stale as the interim leader
// advances the shared doc; reusing it on a later regain would re-emit ops for content already
// present (duplication). Dropping it here means a regain rebuilds a FRESH shadow from the
// current doc via the `??=` below — upholding "a non-leader holds none."
if (agentStreamSessionRef.current) {
endAgentStream(agentStreamSessionRef.current)
agentStreamSessionRef.current = null
}
streamRafRef.current = null
return
}
if (
pending.length > STREAM_REPARSE_THROTTLE_THRESHOLD &&
performance.now() - lastStreamParseAtRef.current < STREAM_REPARSE_THROTTLE_MS
) {
streamRafRef.current = requestAnimationFrame(tick)
return
}
const el = containerRef.current
const pinnedToBottom = el ? el.scrollHeight - el.scrollTop - el.clientHeight < 80 : false
// Open the shadow lazily HERE — only when THIS client actually leads — seeded from the CURRENT
// doc. A non-leader holds none (torn down above), so whether this client is a first-time leader
// or one REGAINING leadership, `??=` finds a null ref and rebuilds fresh from the current doc,
// already carrying the interim leader's ops (never a stale base). Defensive: a ready collab
// editor always has a ySync binding.
agentStreamSessionRef.current ??= beginAgentStream(editor)
const session = agentStreamSessionRef.current
if (!session || !applyAgentStreamFrame(editor, session, pending)) {
streamRafRef.current = null
return
}
streamRafRef.current = null
lastStreamedBodyRef.current = pending
lastStreamParseAtRef.current = performance.now()
if (!disableStreamingAutoScroll && el && pinnedToBottom) el.scrollTop = el.scrollHeight
}
streamRafRef.current = requestAnimationFrame(tick)
return
}
if (streamRafRef.current !== null) {
cancelAnimationFrame(streamRafRef.current)
streamRafRef.current = null
}
// Settle: apply the FINAL body so the Y.Doc exactly equals the streamed result — but ONLY the
// elected writer applies it (the same min-clientID election the streaming tick uses). Without this,
// N tabs watching one run each open a fresh shadow and reconcile current→final; a non-leader's local
// settle microtask runs BEFORE the leader's final propagates (a server round-trip), so both insert
// the same tail and Yjs keeps both (it does not dedupe identical text from two clients) → a
// duplicated tail. The election is reliable here (unlike the bounded startup window): the stream ran
// for seconds, so awareness is long converged. Each tab reads leadership BEFORE clearing its own
// announcement — a remote clear is a network round-trip, always slower than these local microtasks,
// so every tab sees the same announcer set and agrees on one leader. The leader reuses its
// up-to-date shadow (catching a throttled last frame) or, if it never applied mid-stream (a held
// `update`, or a pre-seed stream), opens a FRESH shadow from the current doc; a non-leader applies
// nothing (the leader's final broadcasts to it) and frees any shadow it still held. The durable
// `edit_content` write then lands as a noop diff for everyone.
if (wasStreamingRef.current && collabReady) {
wasStreamingRef.current = false
agentAnnouncedRef.current = false
const isSettleWriter =
!collaboration || isAgentStreamLeader(collaboration.awareness, collaboration.doc.clientID)
if (collaboration) clearAgentApplying(collaboration.awareness)
lastStreamedBodyRef.current = null
const heldSession = agentStreamSessionRef.current
agentStreamSessionRef.current = null
if (isSettleWriter) {
const finalBody = splitFrontmatter(content).body
const session = heldSession ?? beginAgentStream(editor)
if (session) {
runOffRender(() => applyAgentStreamFrame(editor, session, finalBody))
// Free the shadow with an UNGUARDED microtask (not `runOffRender`): a rapid follow-up stream
// can supersede the run token and drop the apply above, but the shadow must always be
// destroyed. Queued after the apply, so it frees the shadow only once that has had its chance.
queueMicrotask(() => endAgentStream(session))
}
} else if (heldSession) {
// Non-leader: it never writes the final (the leader does + broadcasts it); free any shadow it held.
queueMicrotask(() => endAgentStream(heldSession))
}
}
return
}
if (isStreaming) {
wasStreamingRef.current = true
if (editor.isEditable) editor.setEditable(false)
if (editor.isEditable) {
runOffRender(() => {
if (editor.isEditable) editor.setEditable(false)
})
}
const body = splitFrontmatter(content).body
if (body === lastSyncedBodyRef.current) return
pendingStreamBodyRef.current = body
@@ -533,31 +1069,67 @@ export function LoadedRichMarkdownEditor({
if (isInitialSettle || wasStreamingRef.current) {
wasStreamingRef.current = false
settledRef.current = lockSettled(content)
syncEditorBody(splitFrontmatter(content).body)
// `setContent` maps any pre-existing selection onto the new doc rather than clearing it — a
// select-all survives as "select everything," permanently painting every divider/image with the
// `rich-leaf-in-selection` decoration (keymap.ts) until the user clicks elsewhere. This must run
// on every settle regardless of whether `setContent` ran just above: the last streaming tick
// already syncs `lastSyncedBodyRef` to the final body before settle, so `body` usually already
// equals it here — collapsing only inside that `if` would skip the common streamed-content case
// entirely. `setTextSelection` (not `.focus()`) so this never steals DOM focus from whatever the
// user is doing outside the editor.
editor.commands.setTextSelection(editor.state.doc.content.size)
editor.setEditable(canEdit && settledRef.current.verdict)
if (isInitialSettle && autoFocus) editor.commands.focus('end')
const settledVerdict = settledRef.current.verdict
const shouldFocus = isInitialSettle && autoFocus
// A settle owes a selection collapse. Track it as a ref, not just inline in this microtask: if a
// newer run bumps the token before this microtask fires, this settle's task is dropped — but the
// debt survives, and the run that supersedes it (settle OR the steady-sync path below) clears it.
pendingCollapseRef.current = true
// One ordered microtask: set body → collapse selection → re-apply editability. The collapse is
// load-bearing and runs on every settle even when the body is unchanged — setContent maps a
// pre-existing selection onto the new doc, so a prior select-all survives as "select everything",
// permanently painting every divider/image with the rich-leaf-in-selection decoration (keymap.ts)
// until the user clicks away. setTextSelection (not .focus()) never steals DOM focus.
runOffRender(() => {
syncEditorBody(splitFrontmatter(content).body)
pendingCollapseRef.current = false
editor.commands.setTextSelection(editor.state.doc.content.size)
editor.setEditable(canEdit && settledVerdict && collabReady)
if (shouldFocus) editor.commands.focus('end')
})
return
}
syncEditorBody(splitFrontmatter(content).body)
if (settledRef.current) editor.setEditable(canEdit && settledRef.current.verdict)
}, [editor, content, isStreaming, canEdit, autoFocus, disableStreamingAutoScroll])
const settled = settledRef.current
runOffRender(() => {
syncEditorBody(splitFrontmatter(content).body)
// Honor a collapse a superseded settle owed but never applied (its microtask was dropped when this
// run bumped the token), so a post-stream select-all can't keep the leaf-in-selection decoration.
if (pendingCollapseRef.current) {
pendingCollapseRef.current = false
editor.commands.setTextSelection(editor.state.doc.content.size)
}
if (settled) editor.setEditable(canEdit && settled.verdict && collabReady)
})
}, [
editor,
content,
isStreaming,
canEdit,
autoFocus,
disableStreamingAutoScroll,
collaborationEnabled,
collabReady,
])
useEffect(
() => () => {
if (streamRafRef.current !== null) cancelAnimationFrame(streamRafRef.current)
if (agentStreamSessionRef.current) {
endAgentStream(agentStreamSessionRef.current)
agentStreamSessionRef.current = null
}
lastStreamedBodyRef.current = null
agentAnnouncedRef.current = false
},
[]
)
// Show the read-only placeholder (the already-fetched markdown) whenever a collaborative doc has not yet
// seeded — including during an agent stream that begins before the seed lands. Streamed diffs are held
// until `collabReady` (see the streaming effect), so before then the editor is empty; the placeholder
// shows the base content until the seed swaps it in, avoiding both a blank frame and a garbled merge.
const showPlaceholder = collaborationEnabled && !collabReady
return (
<div
ref={containerRef}
@@ -582,9 +1154,20 @@ export function LoadedRichMarkdownEditor({
if (images.length > 0) void insertImagesRef.current(images, at)
}}
/>
{showPlaceholder && placeholderHtml && (
// Instant read-only content while the collaborative doc seeds; the editor stays mounted-but-
// hidden below so it renders the seeded doc before the swap. Same layout box → no reflow.
<div
className='rich-markdown-prose mx-auto w-full max-w-[48rem] px-8 py-6'
dangerouslySetInnerHTML={{ __html: placeholderHtml }}
/>
)}
<EditorContent
editor={editor}
className='mx-auto flex w-full max-w-[48rem] flex-1 flex-col px-8 py-6 selection:bg-[var(--selection-bg)] selection:text-[var(--text-primary)] dark:selection:bg-[var(--selection-dark)] dark:selection:text-white'
className={cn(
'mx-auto flex w-full max-w-[48rem] flex-1 flex-col px-8 py-6 selection:bg-[var(--selection-bg)] selection:text-[var(--text-primary)] dark:selection:bg-[var(--selection-dark)] dark:selection:text-white',
showPlaceholder && placeholderHtml && 'hidden'
)}
/>
</div>
)
@@ -0,0 +1,63 @@
/**
* @vitest-environment jsdom
*/
import { Editor } from '@tiptap/core'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { createMarkdownEditorExtensions } from './editor-extensions'
import { firstHeadingTitle } from './title-heading'
function editorWith(markdown: string): Editor {
const editor = new Editor({ extensions: createMarkdownEditorExtensions({ placeholder: '' }) })
if (markdown) editor.commands.setContent(markdown, { contentType: 'markdown' })
return editor
}
describe('firstHeadingTitle', () => {
beforeEach(() => {
vi.stubGlobal(
'ResizeObserver',
class {
observe() {}
unobserve() {}
disconnect() {}
}
)
Element.prototype.scrollIntoView = vi.fn()
document.elementFromPoint = vi.fn(() => null)
})
it('returns the leading H1 text', () => {
const editor = editorWith('# Q3 Planning\n\nbody')
expect(firstHeadingTitle(editor.state.doc)).toBe('Q3 Planning')
editor.destroy()
})
it('returns the text of any leading heading level', () => {
const editor = editorWith('## Sub title')
expect(firstHeadingTitle(editor.state.doc)).toBe('Sub title')
editor.destroy()
})
it('returns null when the first block is a paragraph, not a heading', () => {
const editor = editorWith('just text\n\n# later heading')
expect(firstHeadingTitle(editor.state.doc)).toBeNull()
editor.destroy()
})
it('returns null for an empty leading heading', () => {
const editor = editorWith('')
editor.commands.setContent({ type: 'doc', content: [{ type: 'heading', attrs: { level: 1 } }] })
expect(firstHeadingTitle(editor.state.doc)).toBeNull()
editor.destroy()
})
it('returns null for a whitespace-only leading heading (trim boundary)', () => {
const editor = editorWith('')
editor.commands.setContent({
type: 'doc',
content: [{ type: 'heading', attrs: { level: 1 }, content: [{ type: 'text', text: ' ' }] }],
})
expect(firstHeadingTitle(editor.state.doc)).toBeNull()
editor.destroy()
})
})
@@ -0,0 +1,9 @@
import type { Node as ProseMirrorNode } from '@tiptap/pm/model'
/** The text of the document's leading heading (any level), or null when the first block isn't a heading. */
export function firstHeadingTitle(doc: ProseMirrorNode): string | null {
const first = doc.firstChild
if (!first || first.type.name !== 'heading') return null
const text = first.textContent.trim()
return text.length > 0 ? text : null
}
@@ -61,6 +61,13 @@ interface UseEditableFileContentOptions {
* the at-rest baseline, never while an agent stream is in flight. Stable reference required.
*/
normalizeBaseline?: (raw: string) => string
/**
* Extra gate on autosave (and draft persistence). When `false`, saving is
* suppressed even when otherwise eligible the collaborative editor uses it to
* hold saves until the shared document is synced AND seeded, so an empty or
* partially-synced doc can never overwrite the real file. Defaults to `true`.
*/
canAutosave?: boolean
}
interface EditableFileContent {
@@ -136,6 +143,7 @@ export function useEditableFileContent({
saveRef,
discardRef,
normalizeBaseline,
canAutosave = true,
}: UseEditableFileContentOptions): EditableFileContent {
const onDirtyChangeRef = useRef(onDirtyChange)
const onSaveStatusChangeRef = useRef(onSaveStatusChange)
@@ -234,7 +242,7 @@ export function useEditableFileContent({
[workspaceId, file.id, markSavedContent]
)
const autosaveEnabled = canEdit && isInitialized && !isStreamInteractionLocked
const autosaveEnabled = canEdit && isInitialized && !isStreamInteractionLocked && canAutosave
const { saveStatus, saveImmediately, isDirty, discard } = useAutosave({
content,
@@ -249,9 +257,16 @@ export function useEditableFileContent({
),
})
// When the client can't autosave it isn't the durability owner: the collaborative editor holds
// `canAutosave` permanently false because the relay persists the doc server-side (debounced + on
// last-disconnect), so `savedContent` never advances and raw `isDirty` would latch true after any
// local OR remote keystroke — surfacing a spurious "Unsaved changes" navigation prompt whose
// "Discard" discards nothing real. With nothing the user can save, there is nothing to warn about.
const isDirtyForCaller = canAutosave && isDirty
useEffect(() => {
onDirtyChangeRef.current?.(isDirty)
}, [isDirty])
onDirtyChangeRef.current?.(isDirtyForCaller)
}, [isDirtyForCaller])
useEffect(() => {
onSaveStatusChangeRef.current?.(
@@ -299,6 +314,6 @@ export function useEditableFileContent({
hasContentError: streamingContent === undefined && Boolean(error) && !isInitialized,
saveStatus,
saveImmediately,
isDirty,
isDirty: isDirtyForCaller,
}
}
@@ -77,8 +77,11 @@ import {
isPreviewable,
isTextEditable,
} from '@/app/workspace/[workspaceId]/files/components/file-viewer'
import { FileDocAvatars } from '@/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/collaboration/file-doc-avatars'
import { FileDocRoomProvider } from '@/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/collaboration/file-doc-room-context'
import { FilesListContextMenu } from '@/app/workspace/[workspaceId]/files/components/files-list-context-menu'
import { ShareModal } from '@/app/workspace/[workspaceId]/files/components/share-modal'
import { useWorkspaceFilesRoom } from '@/app/workspace/[workspaceId]/files/hooks/use-workspace-files-room'
import {
filesFilterParsers,
filesFilterUrlKeys,
@@ -86,6 +89,12 @@ import {
filesSortParams,
filesUrlKeys,
} from '@/app/workspace/[workspaceId]/files/search-params'
import {
DEFAULT_UNTITLED_NAME,
deriveMarkdownFileName,
isUntitledName,
uniqueMarkdownName,
} from '@/app/workspace/[workspaceId]/files/untitled-title'
import { useUserPermissionsContext } from '@/app/workspace/[workspaceId]/providers/workspace-permissions-provider'
import { useContextMenu } from '@/app/workspace/[workspaceId]/w/components/sidebar/hooks'
import { usePinItem, usePinnedIds, useUnpinItem } from '@/hooks/queries/pinned-items'
@@ -208,6 +217,11 @@ export function Files() {
const canEdit = userPermissions.canEdit === true
const { config: permissionConfig } = usePermissionConfig()
// Joined for the live file tree: a `workspace-files-changed` broadcast invalidates the
// browser. "Who's in this file" comes from the file-doc room (see FileDocRoomProvider),
// not from who's browsing the Files section.
useWorkspaceFilesRoom(workspaceId)
useEffect(() => {
if (permissionConfig.hideFilesTab) {
router.replace(`/workspace/${workspaceId}`)
@@ -358,6 +372,34 @@ export function Files() {
const selectedFileRef = useRef(selectedFile)
selectedFileRef.current = selectedFile
/**
* While a file is still untitled, name it after the leading heading the user types in its editor. The
* editor reports the heading text (debounced); here we re-check the file is still untitled, derive a
* unique `.md` name among its folder siblings, and rename. A no-op once the file has a real name.
*/
const handleDeriveTitleFromHeading = useCallback(
(headingText: string) => {
const currentFile = selectedFileRef.current
if (!currentFile || !isUntitledName(currentFile.name)) return
const derived = deriveMarkdownFileName(headingText)
if (!derived) return
const siblingNames = new Set(
filesRef.current
.filter(
(f) =>
(f.folderId ?? null) === (currentFile.folderId ?? null) && f.id !== currentFile.id
)
.map((f) => f.name)
)
const name = uniqueMarkdownName(derived, siblingNames)
if (name === currentFile.name) return
renameFile
.mutateAsync({ workspaceId, fileId: currentFile.id, name })
.catch((err) => logger.error('Failed to auto-name file from heading:', err))
},
[workspaceId]
)
const shareFile = shareFileId ? (files.find((f) => f.id === shareFileId) ?? null) : null
const shareModal = shareFile ? (
<ShareModal
@@ -1248,12 +1290,7 @@ export function Files() {
const existingNames = new Set(
filesRef.current.filter((f) => (f.folderId ?? null) === currentFolderId).map((f) => f.name)
)
let name = 'untitled.md'
let counter = 1
while (existingNames.has(name)) {
name = `untitled (${counter}).md`
counter++
}
const name = uniqueMarkdownName(DEFAULT_UNTITLED_NAME, existingNames)
const mimeType = getMimeTypeFromExtension('md')
const blob = new Blob([''], { type: mimeType })
@@ -1979,35 +2016,43 @@ export function Files() {
if (selectedFile) {
return (
<>
<Resource>
<Resource.Header
icon={FilesIcon}
breadcrumbs={fileDetailBreadcrumbs}
actions={fileActions}
/>
<FileViewer
key={selectedFile.id}
file={selectedFile}
workspaceId={workspaceId}
canEdit={canEdit}
previewMode={previewMode}
autoFocus={isNewFile || justCreatedFileIdRef.current === selectedFile.id}
onDirtyChange={setIsDirty}
onSaveStatusChange={handleSaveStatusChange}
saveRef={saveRef}
discardRef={discardRef}
/>
{/* The room provider scopes "who's in this file" presence to the open document: the
editor (inside FileViewer) publishes the server-authenticated roster and the
header's FileDocAvatars reads it both must be descendants. */}
<FileDocRoomProvider>
<Resource>
<Resource.Header
icon={FilesIcon}
breadcrumbs={fileDetailBreadcrumbs}
actions={fileActions}
aside={<FileDocAvatars />}
/>
<FileViewer
key={selectedFile.id}
file={selectedFile}
workspaceId={workspaceId}
canEdit={canEdit}
previewMode={previewMode}
autoFocus={isNewFile || justCreatedFileIdRef.current === selectedFile.id}
onDirtyChange={setIsDirty}
onSaveStatusChange={handleSaveStatusChange}
saveRef={saveRef}
discardRef={discardRef}
collaborative
onDeriveTitleFromHeading={handleDeriveTitleFromHeading}
/>
<ChipConfirmModal
open={showUnsavedChangesAlert}
onOpenChange={setShowUnsavedChangesAlert}
srTitle='Unsaved Changes'
title='Unsaved Changes'
text='You have unsaved changes. Are you sure you want to discard them?'
dismissLabel='Keep editing'
confirm={{ label: 'Discard Changes', onClick: handleDiscardChanges }}
/>
</Resource>
<ChipConfirmModal
open={showUnsavedChangesAlert}
onOpenChange={setShowUnsavedChangesAlert}
srTitle='Unsaved Changes'
title='Unsaved Changes'
text='You have unsaved changes. Are you sure you want to discard them?'
dismissLabel='Keep editing'
confirm={{ label: 'Discard Changes', onClick: handleDiscardChanges }}
/>
</Resource>
</FileDocRoomProvider>
<DeleteConfirmModal
open={showDeleteConfirm}
@@ -0,0 +1,18 @@
'use client'
import { ROOM_TYPES } from '@sim/realtime-protocol/rooms'
import { useQueryClient } from '@tanstack/react-query'
import { useWorkspaceInvalidationRoom } from '@/app/workspace/[workspaceId]/hooks/use-workspace-invalidation-room'
import { invalidateWorkspaceFileBrowsers } from '@/hooks/queries/workspace-file-folders'
/**
* Keeps the file browser live: joins the workspace-files room so a `workspace-files-changed`
* broadcast (fanned out by the file mutation API) invalidates the browser queries and every viewer
* refetches without waiting for staleness. Thin binding over {@link useWorkspaceInvalidationRoom}.
*/
export function useWorkspaceFilesRoom(workspaceId: string): void {
const queryClient = useQueryClient()
useWorkspaceInvalidationRoom(workspaceId, ROOM_TYPES.WORKSPACE_FILES, () =>
invalidateWorkspaceFileBrowsers(queryClient, workspaceId)
)
}
@@ -0,0 +1,77 @@
import { describe, expect, it } from 'vitest'
import {
DEFAULT_UNTITLED_NAME,
deriveMarkdownFileName,
isUntitledName,
uniqueMarkdownName,
} from './untitled-title'
describe('untitled format single-source-of-truth', () => {
// Guards against DEFAULT_UNTITLED_NAME / uniqueMarkdownName drifting from the isUntitledName regex:
// the default name and its deduped siblings must always read back as "untitled".
it('recognizes the default name and its deduped siblings as untitled', () => {
expect(isUntitledName(DEFAULT_UNTITLED_NAME)).toBe(true)
const second = uniqueMarkdownName(DEFAULT_UNTITLED_NAME, new Set([DEFAULT_UNTITLED_NAME]))
expect(second).toBe('untitled (1).md')
expect(isUntitledName(second)).toBe(true)
})
})
describe('isUntitledName', () => {
it.each([
['untitled.md', true],
['untitled (1).md', true],
['untitled (23).md', true],
['Untitled.md', false],
['untitled.txt', false],
['untitled', false],
['my notes.md', false],
['untitled draft.md', false],
['untitled ().md', false],
])('%s → %s', (name, expected) => {
expect(isUntitledName(name)).toBe(expected)
})
})
describe('deriveMarkdownFileName', () => {
it('turns heading text into a .md file name', () => {
expect(deriveMarkdownFileName('Q3 Planning')).toBe('Q3 Planning.md')
})
it('strips filesystem-illegal characters and collapses whitespace', () => {
expect(deriveMarkdownFileName('Roadmap: Q3 / Q4 *draft*')).toBe('Roadmap Q3 Q4 draft.md')
})
it('keeps hyphens and dots inside the title', () => {
expect(deriveMarkdownFileName('v1.2 - release-notes')).toBe('v1.2 - release-notes.md')
})
it('returns null when nothing usable remains', () => {
expect(deriveMarkdownFileName(' ')).toBeNull()
expect(deriveMarkdownFileName('///')).toBeNull()
})
it('does not double the extension when the heading already ends in .md', () => {
expect(deriveMarkdownFileName('README.md')).toBe('README.md')
expect(deriveMarkdownFileName('notes.MD')).toBe('notes.MD')
})
it('hard-caps the length (no ellipsis) before the extension', () => {
const result = deriveMarkdownFileName('a'.repeat(200))
expect(result).toBe(`${'a'.repeat(100)}.md`)
})
it('re-trims when the hard cap lands on a space (no "foo .md")', () => {
// 99 non-space chars + space at index 99 → truncate(100) leaves a trailing space to re-trim away.
const result = deriveMarkdownFileName(`${'a'.repeat(99)} bcd`)
expect(result).toBe(`${'a'.repeat(99)}.md`)
})
})
describe('uniqueMarkdownName', () => {
it('returns the name unchanged when free', () => {
expect(uniqueMarkdownName('notes.md', new Set())).toBe('notes.md')
})
it('appends an incrementing suffix before the extension when taken', () => {
expect(uniqueMarkdownName('notes.md', new Set(['notes.md']))).toBe('notes (1).md')
expect(uniqueMarkdownName('notes.md', new Set(['notes.md', 'notes (1).md']))).toBe(
'notes (2).md'
)
})
})
@@ -0,0 +1,57 @@
import { truncate } from '@sim/utils/string'
/**
* The name a freshly-created markdown file is given in `handleCreateFile`: `untitled.md`, or
* `untitled (n).md` when that is taken. A file keeps this "unnamed" status until it is renamed
* while unnamed, typing a leading heading names the file (one direction only; the reverse
* nameheading seed was removed as unsafe on the shared editor). See {@link isUntitledName}.
*/
export const DEFAULT_UNTITLED_NAME = 'untitled.md'
const UNTITLED_NAME_RE = /^untitled(?: \(\d+\))?\.md$/
/** Longest title kept when deriving a file name from a heading, before the `.md` extension. */
const MAX_DERIVED_TITLE_LENGTH = 100
/**
* Filename characters disallowed across the common platforms (`\ / : * ? " < > |`) plus C0 control
* characters, replaced with a space when deriving a file name from heading text.
*/
const ILLEGAL_FILENAME_CHARS = /[\\/:*?"<>|\x00-\x1f]/g
/** True when `name` is still the auto-assigned untitled markdown name (`untitled.md`, `untitled (2).md`). */
export function isUntitledName(name: string): boolean {
return UNTITLED_NAME_RE.test(name)
}
/**
* Derives a markdown file name from heading text illegal filename characters dropped, whitespace
* collapsed, trimmed, hard-capped at {@link MAX_DERIVED_TITLE_LENGTH}, and suffixed with `.md`.
* Returns null when nothing usable remains (e.g. a heading of only slashes), so the caller keeps the
* current name.
*/
export function deriveMarkdownFileName(headingText: string): string | null {
const base = headingText.replace(ILLEGAL_FILENAME_CHARS, ' ').replace(/\s+/g, ' ').trim()
if (!base) return null
// Re-trim after the hard cap: truncation can land mid-word and leave a trailing space (`"foo .md"`).
const capped = truncate(base, MAX_DERIVED_TITLE_LENGTH, '').trim()
if (!capped) return null
// A heading that already ends in `.md` (e.g. `# README.md`) must not become `README.md.md`.
return /\.md$/i.test(capped) ? capped : `${capped}.md`
}
/**
* Makes `name` unique among `existingNames` by appending ` (n)` before the `.md` extension the same
* scheme `handleCreateFile` uses for the default untitled name.
*/
export function uniqueMarkdownName(name: string, existingNames: ReadonlySet<string>): string {
if (!existingNames.has(name)) return name
const withoutExt = name.replace(/\.md$/i, '')
let counter = 1
let candidate = `${withoutExt} (${counter}).md`
while (existingNames.has(candidate)) {
counter++
candidate = `${withoutExt} (${counter}).md`
}
return candidate
}
@@ -250,6 +250,7 @@ export const ResourceContent = memo(function ResourceContent({
}
isAgentEditing={isAgentEditing}
streamIsIncremental={streamIsIncremental}
streamOperation={previewSession?.operation}
disableStreamingAutoScroll={disableStreamingAutoScroll}
previewContextKey={previewContextKey}
/>
@@ -663,6 +664,7 @@ interface EmbeddedFileProps {
streamingContent?: string
isAgentEditing?: boolean
streamIsIncremental?: boolean
streamOperation?: string
disableStreamingAutoScroll?: boolean
previewContextKey?: string
}
@@ -675,6 +677,7 @@ function EmbeddedFile({
streamingContent,
isAgentEditing,
streamIsIncremental,
streamOperation,
disableStreamingAutoScroll = false,
previewContextKey,
}: EmbeddedFileProps) {
@@ -718,8 +721,10 @@ function EmbeddedFile({
streamingContent={streamingContent}
isAgentEditing={isAgentEditing}
streamIsIncremental={streamIsIncremental}
streamOperation={streamOperation}
disableStreamingAutoScroll={disableStreamingAutoScroll}
previewContextKey={previewContextKey}
collaborative
/>
</div>
)
@@ -95,6 +95,80 @@ describe('deriveFilePreviewSession', () => {
expect(next.previewVersion).toBe(9)
})
it('ignores a re-delivered delta (same version) — no double-append (the duplication bug)', () => {
const prev = session({ previewText: 'the story so far.', previewVersion: 7 })
const replay = deriveFilePreviewSession(
prev,
{
previewPhase: 'file_preview_content',
toolCallId: 'tool-1',
toolName: 'workspace_file',
content: ' the story so far.',
contentMode: 'delta',
previewVersion: 7, // <= prev.previewVersion → a replay, must not re-append
fileName: 'deck.pptx',
},
'stream-1',
NOW
)
expect(replay.previewText).toBe('the story so far.')
expect(replay.previewVersion).toBe(7)
})
it('ignores a replayed older snapshot — no regression of accumulated text', () => {
const prev = session({ previewText: 'full accumulated body', previewVersion: 12 })
const stale = deriveFilePreviewSession(
prev,
{
previewPhase: 'file_preview_content',
toolCallId: 'tool-1',
toolName: 'workspace_file',
content: 'earlier partial',
contentMode: 'snapshot',
previewVersion: 5, // stale replay
fileName: 'deck.pptx',
},
'stream-1',
NOW
)
expect(stale.previewText).toBe('full accumulated body')
expect(stale.previewVersion).toBe(12)
})
it('re-processing the SAME delta stream N times yields the content exactly once', () => {
// Simulates a client re-render/re-subscribe replaying the stream: the accumulated text must be stable.
const deltas = [
{ v: 1, c: 'A' },
{ v: 2, c: 'B' },
{ v: 3, c: 'C' },
]
const run = (start: FilePreviewSession | undefined) =>
deltas.reduce<FilePreviewSession | undefined>(
(acc, d) =>
deriveFilePreviewSession(
acc,
{
previewPhase: 'file_preview_content',
toolCallId: 'tool-1',
toolName: 'workspace_file',
content: d.c,
contentMode: 'delta',
previewVersion: d.v,
fileName: 'deck.pptx',
},
'stream-1',
NOW
),
start
)
const first = run(undefined)
expect(first?.previewText).toBe('ABC')
const second = run(first) // replay the exact same events
expect(second?.previewText).toBe('ABC') // NOT 'ABCABC'
const third = run(second)
expect(third?.previewText).toBe('ABC')
})
it('replaces text on a snapshot and carries forward prior fileId', () => {
const prev = session({ previewText: 'old', fileId: 'file-9', previewVersion: 4 })
const next = deriveFilePreviewSession(
@@ -71,15 +71,23 @@ export function deriveFilePreviewSession(
return base
case 'file_preview_content': {
const incomingVersion =
typeof payload.previewVersion === 'number' && Number.isFinite(payload.previewVersion)
? payload.previewVersion
: (prev?.previewVersion ?? 0) + 1
// Replay-safe accumulation. A content event may be re-delivered or re-processed (a client
// re-render/re-subscribe, or a stream replay); `previewVersion` is monotonic per tool call, so only
// apply when it STRICTLY advances. Without this guard a re-delivered `delta` double-appends (the
// duplicated-tail bug) and a replayed older `snapshot` regresses the text. `base` already carries
// `prev.previewText`, so an ignored replay leaves the accumulated text untouched.
if (prev && incomingVersion <= prev.previewVersion) {
return { ...base, status: 'streaming' }
}
const previewText =
payload.contentMode === 'delta'
? (prev?.previewText ?? '') + payload.content
: payload.content
const previewVersion =
typeof payload.previewVersion === 'number' && Number.isFinite(payload.previewVersion)
? payload.previewVersion
: (prev?.previewVersion ?? 0) + 1
return { ...base, status: 'streaming', previewText, previewVersion }
return { ...base, status: 'streaming', previewText, previewVersion: incomingVersion }
}
case 'file_preview_complete':
@@ -0,0 +1,111 @@
'use client'
import { useEffect, useRef } from 'react'
import { createLogger } from '@sim/logger'
import type { RoomType } from '@sim/realtime-protocol/rooms'
import { useSocket } from '@/app/workspace/providers/socket-provider'
const logger = createLogger('WorkspaceInvalidationRoom')
/** Retry cap + base delay for a retryable join failure on an otherwise-live socket. */
const MAX_JOIN_RETRIES = 3
const JOIN_RETRY_BASE_MS = 1000
interface JoinErrorPayload {
workspaceId: string
error: string
code: string
retryable?: boolean
}
/**
* Joins a workspace-scoped, presence-free "invalidation room" over the shared socket and runs
* `onChanged` whenever the server broadcasts `${roomType}-changed` for this workspace, so the list
* refetches without waiting for staleness. Shared core behind {@link useWorkspaceFilesRoom} and
* {@link useWorkspaceTablesRoom}; event names derive from `roomType`.
*
* These rooms carry no presence "who's in a resource" comes from the per-resource room, not from
* who's browsing the section. Mutations happen server-side (HTTP + copilot) and fan out this signal.
*/
export function useWorkspaceInvalidationRoom(
workspaceId: string,
roomType: RoomType,
onChanged: () => void
): void {
const { socket } = useSocket()
// Held by ref so a caller passing a fresh closure each render never re-subscribes the socket.
const onChangedRef = useRef(onChanged)
onChangedRef.current = onChanged
useEffect(() => {
if (!socket || !workspaceId) return
const joinEvent = `join-${roomType}`
const successEvent = `${joinEvent}-success`
const errorEvent = `${joinEvent}-error`
const leaveEvent = `leave-${roomType}`
const changedEvent = `${roomType}-changed`
let retries = 0
let retryTimer: ReturnType<typeof setTimeout> | null = null
const join = () => socket.emit(joinEvent, { workspaceId })
// A fresh (re)connect gets a fresh retry budget, so a prior full exhaustion doesn't leave the
// socket unable to retry a failed re-join until the next success. Cancel any retry still pending
// from before the reconnect so it can't fire a duplicate join after this immediate one.
const handleConnect = () => {
retries = 0
if (retryTimer) {
clearTimeout(retryTimer)
retryTimer = null
}
join()
}
const handleJoinSuccess = (data: { workspaceId: string }) => {
if (data.workspaceId !== workspaceId) return
retries = 0
// Cancel any retry scheduled by a prior retryable error so it can't fire an extra
// join after we're already in.
if (retryTimer) {
clearTimeout(retryTimer)
retryTimer = null
}
}
const handleJoinError = (data: JoinErrorPayload) => {
if (data.workspaceId !== workspaceId) return
logger.warn(`Failed to join ${roomType} room`, { code: data.code, error: data.error })
if (data.retryable && retries < MAX_JOIN_RETRIES) {
retries += 1
// Clear any still-pending retry before scheduling a new one, so reconnect churn can't
// orphan a timer that fires an extra join().
if (retryTimer) clearTimeout(retryTimer)
retryTimer = setTimeout(join, JOIN_RETRY_BASE_MS * retries)
}
}
const handleChanged = (data: { workspaceId: string }) => {
if (data.workspaceId === workspaceId) onChangedRef.current()
}
// Join now if the socket is already connected; `connect` covers (re)connects.
if (socket.connected) join()
socket.on('connect', handleConnect)
socket.on(successEvent, handleJoinSuccess)
socket.on(errorEvent, handleJoinError)
socket.on(changedEvent, handleChanged)
return () => {
if (retryTimer) clearTimeout(retryTimer)
socket.off('connect', handleConnect)
socket.off(successEvent, handleJoinSuccess)
socket.off(errorEvent, handleJoinError)
socket.off(changedEvent, handleChanged)
// Leave the room, scoped to THIS workspace: the server no-ops if the socket has already
// switched to another workspace's room (so a workspace A→B switch, where B's join runs first
// and auto-leaves A, can't have A's leave evict B).
socket.emit(leaveEvent, { workspaceId })
}
}, [socket, workspaceId, roomType])
}
@@ -0,0 +1,312 @@
'use client'
import { useCallback, useEffect, useLayoutEffect, useRef, useState } from 'react'
import { createPortal } from 'react-dom'
import { getUserColor, withAlpha } from '@/lib/workspaces/colors'
import {
isCellInSelection,
type NormalizedSelection,
} from '@/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/utils'
import type { RemoteTableSelection } from '@/app/workspace/[workspaceId]/tables/[tableId]/hooks/use-table-room'
/** A measured remote selection, positioned in the grid content wrapper's space. */
interface SelectionBox {
socketId: string
userName: string
color: string
editing: boolean
top: number
left: number
width: number
height: number
/** Viewport-space top/left of the selection, for the body-portaled name label. */
viewportTop: number
viewportLeft: number
/** Resolved anchor/focus cell indices (undefined when off-window). Coverage by the local
* selection is derived from these in render (see {@link isSelectionCovered}) no DOM
* re-measure when only the local caret moves. */
anchorRow: number | undefined
anchorCol: number | undefined
focusRow: number | undefined
focusCol: number | undefined
}
interface RemoteSelectionOverlayProps {
remoteSelections: RemoteTableSelection[]
/** Column id → its rendered column index (matches the cells' `data-col`). */
columnIndexById: Map<string, number>
/** Row id → its index in the current row list, to test local-selection coverage. */
rowIndexById: Map<string, number>
/** The local user's own normalized selection, so a co-selected remote cell defers to it. */
localSelection: NormalizedSelection | null
/** The grid's scroll container (`data-table-scroll`), queried for cell rects. */
scrollElement: HTMLElement | null
}
/** The cell `<td>` for a (rowId, columnIndex), or undefined when virtualized off-window. */
function cellRect(
scrollEl: HTMLElement,
rowId: string,
columnIndex: number | undefined
): DOMRect | undefined {
if (columnIndex === undefined) return undefined
// `rowId` is a remote peer's value — escape it so a hostile id can't break the
// selector and throw (`columnIndex` is a local numeric index, already safe).
const cell = scrollEl.querySelector(
`[data-row-id="${CSS.escape(rowId)}"][data-col="${columnIndex}"]`
)
return cell?.getBoundingClientRect()
}
/**
* Whether a remote selection defers to the local one: true only when the local selection
* fully contains it (both corners inside), so a partial overlap still shows.
*/
function isSelectionCovered(
anchorRow: number | undefined,
anchorCol: number | undefined,
focusRow: number | undefined,
focusCol: number | undefined,
bounds: NormalizedSelection | null
): boolean {
return (
bounds !== null &&
isCellInSelection(anchorRow, anchorCol, bounds) &&
isCellInSelection(focusRow, focusCol, bounds)
)
}
/**
* Renders remote collaborators' cell selections over the table grid a colored
* border per user (Google-Sheets style), a darker fill while they are editing, and
* their name on hover. Mounted inside the grid's `relative` content wrapper, so
* content-space coordinates scroll with the grid automatically.
*
* Positions are measured from the live cell rects (the same `[data-row-id][data-col]`
* idiom the reveal effect uses), keyed by stable ids so each client renders under its
* own sort/scroll. A selection whose rows are virtualized off-window is simply not
* drawn. The layer is `pointer-events-none` so it never intercepts cell clicks; the
* name-on-hover is driven by hit-testing pointer moves against the measured boxes.
*/
export function RemoteSelectionOverlay({
remoteSelections,
columnIndexById,
rowIndexById,
localSelection,
scrollElement,
}: RemoteSelectionOverlayProps) {
const rootRef = useRef<HTMLDivElement>(null)
const [boxes, setBoxes] = useState<SelectionBox[]>([])
const [hoveredSocketId, setHoveredSocketId] = useState<string | null>(null)
// Latest data read by the subscribe-once effect + the pointer hit-test, so neither
// re-subscribes on every incoming selection delta.
const boxesRef = useRef<SelectionBox[]>([])
boxesRef.current = boxes
const remoteSelectionsRef = useRef(remoteSelections)
remoteSelectionsRef.current = remoteSelections
const columnIndexByIdRef = useRef(columnIndexById)
columnIndexByIdRef.current = columnIndexById
const rowIndexByIdRef = useRef(rowIndexById)
rowIndexByIdRef.current = rowIndexById
// Read only by the pointer hit-test (never in render) to skip a locally-covered box.
const localSelectionRef = useRef(localSelection)
localSelectionRef.current = localSelection
// Cached content-wrapper origin, refreshed on each measure (scroll/resize/data change),
// so the pointer hit-test never forces a layout read per mouse move.
const originRef = useRef({ top: 0, left: 0 })
const measure = useCallback(() => {
const scrollEl = scrollElement
const root = rootRef.current
if (!scrollEl || !root) return
const origin = root.getBoundingClientRect()
originRef.current = { top: origin.top, left: origin.left }
const next: SelectionBox[] = []
for (const selection of remoteSelectionsRef.current) {
const { anchor, focus, editing } = selection.cell
const anchorCol = columnIndexByIdRef.current.get(anchor.columnId)
const focusCol = columnIndexByIdRef.current.get(focus.columnId)
const anchorRow = rowIndexByIdRef.current.get(anchor.rowId)
const focusRow = rowIndexByIdRef.current.get(focus.rowId)
const rects = [
cellRect(scrollEl, anchor.rowId, anchorCol),
cellRect(scrollEl, focus.rowId, focusCol),
].filter((rect): rect is DOMRect => rect !== undefined)
if (rects.length === 0) continue
const viewportTop = Math.min(...rects.map((r) => r.top))
const viewportLeft = Math.min(...rects.map((r) => r.left))
const top = viewportTop - origin.top
const left = viewportLeft - origin.left
const bottom = Math.max(...rects.map((r) => r.bottom)) - origin.top
const right = Math.max(...rects.map((r) => r.right)) - origin.left
next.push({
socketId: selection.socketId,
userName: selection.userName,
color: getUserColor(selection.userId),
editing: editing === true,
top,
left,
width: right - left,
height: bottom - top,
viewportTop,
viewportLeft,
anchorRow,
anchorCol,
focusRow,
focusCol,
})
}
setBoxes(next)
}, [scrollElement])
// Subscribe once per scroll element: re-measure on scroll/resize, and hit-test pointer
// moves against the cached boxes/origin — no layout read per move, stays pointer-events-none.
useEffect(() => {
const scrollEl = scrollElement
if (!scrollEl) return
let raf = 0
const schedule = () => {
if (!raf)
raf = requestAnimationFrame(() => {
raf = 0
measure()
})
}
const handleMove = (event: PointerEvent) => {
const { top, left } = originRef.current
const x = event.clientX - left
const y = event.clientY - top
const hit = boxesRef.current.find(
(b) =>
x >= b.left &&
x <= b.left + b.width &&
y >= b.top &&
y <= b.top + b.height &&
// Skip a box the local selection covers — it isn't drawn, so hovering it must not
// pop a name tag over a cell with no visible remote selection.
!isSelectionCovered(
b.anchorRow,
b.anchorCol,
b.focusRow,
b.focusCol,
localSelectionRef.current
)
)
setHoveredSocketId((prev) =>
prev === (hit?.socketId ?? null) ? prev : (hit?.socketId ?? null)
)
}
const handleLeave = () => setHoveredSocketId(null)
// No measure() here — the re-measure layout effect below runs on mount and whenever
// `measure` changes (it depends on `scrollElement`), so it already covers the initial
// and scroll-element-changed measures without a redundant pass.
scrollEl.addEventListener('scroll', schedule, { passive: true })
scrollEl.addEventListener('pointermove', handleMove, { passive: true })
scrollEl.addEventListener('pointerleave', handleLeave)
const resizeObserver = new ResizeObserver(schedule)
resizeObserver.observe(scrollEl)
// Also observe the content layer (this overlay fills it): a column resize or a
// row-count change grows/shrinks the content without resizing the scroll container,
// yet moves cell rects — so measure off the content, not just the viewport.
if (rootRef.current) resizeObserver.observe(rootRef.current)
// Re-measure when rows are added/removed/reordered/virtualized (a live refetch moves
// cells without a scroll/resize) — childList only, so a cell-content edit doesn't fire.
const tbody = scrollEl.querySelector('tbody')
const rowObserver = new MutationObserver(schedule)
if (tbody) rowObserver.observe(tbody, { childList: true })
return () => {
scrollEl.removeEventListener('scroll', schedule)
scrollEl.removeEventListener('pointermove', handleMove)
scrollEl.removeEventListener('pointerleave', handleLeave)
resizeObserver.disconnect()
rowObserver.disconnect()
if (raf) cancelAnimationFrame(raf)
}
}, [scrollElement, measure])
// Re-measure when the remote selections or column layout change (listeners stay
// subscribed). Layout effect so positions update before paint — no one-frame lag as a
// peer moves. NOT keyed on `localSelection`: moving the local caret changes only which
// boxes are `covered`, which the cheap in-memory pass below handles without a reflow.
useLayoutEffect(() => {
measure()
}, [remoteSelections, columnIndexById, measure])
// Re-derived in render so it reacts to `localSelection`: when the local selection grows to
// cover the hovered box (its outline is no longer drawn) without another pointer move, the
// floating name tag must drop rather than linger over cells with no visible remote selection.
const hoveredBox = hoveredSocketId
? boxes.find(
(box) =>
box.socketId === hoveredSocketId &&
!isSelectionCovered(
box.anchorRow,
box.anchorCol,
box.focusRow,
box.focusCol,
localSelection
)
)
: undefined
return (
<>
<div ref={rootRef} className='pointer-events-none absolute inset-0 z-[8] overflow-hidden'>
{boxes.map((box) =>
// A cell the local user also has selected shows only the local selection — the
// remote box isn't drawn (its `boxes` entry still drives the hover name). The
// border is an inset box-shadow (no layout width, so it never stacks with an
// adjacent cell's border) plus a subtle fill, darker while the peer is editing.
isSelectionCovered(
box.anchorRow,
box.anchorCol,
box.focusRow,
box.focusCol,
localSelection
) ? null : (
<div
key={box.socketId}
className='absolute rounded-xs'
style={{
top: box.top,
left: box.left,
width: box.width,
height: box.height,
boxShadow: `inset 0 0 0 2px ${box.color}`,
backgroundColor: withAlpha(box.color, box.editing ? 0.22 : 0.08),
}}
/>
)
)}
</div>
{/* The name label portals to the body so it floats on top of the grid (and its
sticky header) instead of being clipped by the overlay's overflow-hidden; it's
placed in viewport space, its bottom-left tabbed onto the selection's top-left. */}
{hoveredBox &&
createPortal(
// Same chrome as the workflow-canvas presence label (see cursors.tsx): the
// identity color is the only per-user value; text color, font, radius, and
// padding all reuse the canvas tokens so tables + canvas presence look identical.
// `rounded-bl-none` tabs the label's bottom-left corner onto the selection's
// top-left, the one deviation from the free-floating canvas cursor tag.
<div
className='pointer-events-none fixed z-[60] max-w-[160px] truncate whitespace-nowrap rounded-xs rounded-bl-none px-1.5 py-0.5 font-medium text-[var(--surface-1)] text-xs'
style={{
top: hoveredBox.viewportTop,
left: hoveredBox.viewportLeft,
backgroundColor: hoveredBox.color,
transform: 'translateY(calc(-100% - 2px))',
}}
>
{hoveredBox.userName}
</div>,
document.body
)}
</>
)
}
@@ -5,6 +5,7 @@ import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } fr
import { cn, toast, useToast } from '@sim/emcn'
import { Loader, TableX } from '@sim/emcn/icons'
import { createLogger } from '@sim/logger'
import type { TableCellSelection } from '@sim/realtime-protocol/table-presence'
import { getErrorMessage } from '@sim/utils/errors'
import { useVirtualizer } from '@tanstack/react-virtual'
import { useParams } from 'next/navigation'
@@ -23,6 +24,7 @@ import { getColumnId } from '@/lib/table/column-keys'
import { columnTypeOf } from '@/lib/table/column-types'
import { TABLE_LIMITS } from '@/lib/table/constants'
import { useUserPermissionsContext } from '@/app/workspace/[workspaceId]/providers/workspace-permissions-provider'
import type { RemoteTableSelection } from '@/app/workspace/[workspaceId]/tables/[tableId]/hooks/use-table-room'
import type { BlockedTableAction } from '@/app/workspace/[workspaceId]/tables/[tableId]/lock-copy'
import { useTimezone } from '@/hooks/queries/general-settings'
import {
@@ -53,6 +55,7 @@ import { ExpandedCellPopover } from './cells'
import { ADD_COL_WIDTH, COL_WIDTH, SELECTION_TINT_BG } from './constants'
import { DataRow } from './data-row'
import { ColumnHeaderMenu, WorkflowGroupMetaCell } from './headers'
import { RemoteSelectionOverlay } from './remote-selection-overlay'
import { TableFind } from './table-find'
import { AddRowButton, SelectAllCheckbox, TableColGroup } from './table-primitives'
import type { DisplayColumn } from './types'
@@ -65,6 +68,7 @@ import {
computeNormalizedSelection,
type ExecStatusMix,
expandToDisplayColumns,
isCellInSelection,
moveCell,
ROW_SELECTION_ALL,
ROW_SELECTION_NONE,
@@ -149,6 +153,10 @@ interface TableGridProps {
workspaceId?: string
tableId?: string
embedded?: boolean
/** Remote collaborators' cell selections, rendered as presence overlays. */
remoteSelections: RemoteTableSelection[]
/** Broadcast the local viewer's cell selection to the table presence room. */
emitCellSelection: (cell: TableCellSelection) => void
/** The table's mutation locks; gates row/schema affordances alongside `canEdit`. */
locks?: TableLocks
/**
@@ -365,6 +373,8 @@ export function TableGrid({
workspaceId: propWorkspaceId,
tableId: propTableId,
embedded,
remoteSelections,
emitCellSelection,
locks,
onBlockedAction,
sidebarReservedPx,
@@ -432,6 +442,13 @@ export function TableGrid({
const columnWidthsRef = useRef(columnWidths)
columnWidthsRef.current = columnWidths
const [resizingColumn, setResizingColumn] = useState<string | null>(null)
const resizingColumnRef = useRef(resizingColumn)
resizingColumnRef.current = resizingColumn
/** True from a committed local width change until its metadata PUT settles. Keeps a
* concurrent peer-triggered definition refetch (the value-less `metadata` event forces a
* refetch that can carry the not-yet-persisted server widths) from momentarily reverting
* the just-written widths. */
const pendingWidthWriteRef = useRef(false)
const [columnOrder, setColumnOrder] = useState<string[] | null>(null)
const columnOrderRef = useRef(columnOrder)
columnOrderRef.current = columnOrder
@@ -797,6 +814,24 @@ export function TableGrid({
return expandToDisplayColumns(ordered, tableWorkflowGroups)
}, [columns, columnOrder, hiddenColumns, tableWorkflowGroups])
/** Column id its rendered index (matches the cells' `data-col`), for placing overlays.
* Only built when collaborators are present (the overlay it feeds is gated on that too),
* so solo editing never pays the map build. */
const columnIndexById = useMemo(() => {
const map = new Map<string, number>()
if (remoteSelections.length > 0) displayColumns.forEach((col, index) => map.set(col.key, index))
return map
}, [displayColumns, remoteSelections.length])
/** Row id its index in the current row list, for testing local-selection coverage.
* Only built when collaborators are present (the overlay is gated on that too), so
* solo editing never pays the O(n) map build on a refetch. */
const rowIndexById = useMemo(() => {
const map = new Map<string, number>()
if (remoteSelections.length > 0) rows.forEach((row, index) => map.set(row.id, index))
return map
}, [rows, remoteSelections.length])
const workflowGroupById = useMemo(
() => new Map(tableWorkflowGroups.map((g) => [g.id, g])),
[tableWorkflowGroups]
@@ -971,6 +1006,30 @@ export function TableGrid({
? (rowsRef.current[selectionFocus.rowIndex]?.id ?? null)
: null
// Broadcast the local viewer's cell selection to the presence room. Resolves the
// index-based selection to stable (rowId, columnId), re-running on `rows`/`displayColumns`
// too so a peer's row insert/delete/reorder re-broadcasts the shifted id under the same
// index (the emitter dedups an unchanged result). `editing` marks the active cell so
// peers darken it (the "someone is typing here" signal).
useEffect(() => {
const resolve = (coord: CellCoord | null) => {
if (!coord) return null
const rowId = rows[coord.rowIndex]?.id
const columnId = displayColumns[coord.colIndex]?.key
return rowId && columnId ? { rowId, columnId } : null
}
const anchor = resolve(selectionAnchor)
if (!anchor) {
emitCellSelection(null)
return
}
// A single-cell click leaves `selectionFocus` null; the grid treats that as a
// one-cell selection at the anchor (`focus ?? anchor`). Mirror that — otherwise the
// most common selection would never broadcast and would clear the prior outline.
const focus = resolve(selectionFocus) ?? anchor
emitCellSelection({ anchor, focus, editing: editingCell !== null })
}, [selectionAnchor, selectionFocus, editingCell, rows, displayColumns, emitCellSelection])
const { data: findData, isFetching: isFindFetching } = useFindTableRows({
workspaceId,
tableId,
@@ -1341,12 +1400,7 @@ export function TableGrid({
selectionAnchorRef.current,
selectionFocusRef.current
)
const isWithinSelection =
sel !== null &&
rowIndex >= sel.startRow &&
rowIndex <= sel.endRow &&
colIndex >= sel.startCol &&
colIndex <= sel.endCol
const isWithinSelection = sel !== null && isCellInSelection(rowIndex, colIndex, sel)
if (!isWithinSelection) {
setSelectionAnchor({ rowIndex, colIndex })
@@ -1563,7 +1617,11 @@ export function TableGrid({
const handleColumnResizeEnd = useCallback(() => {
setResizingColumn(null)
updateMetadataRef.current({ columnWidths: columnWidthsRef.current })
pendingWidthWriteRef.current = true
updateMetadataRef.current(
{ columnWidths: columnWidthsRef.current },
{ onSettled: () => (pendingWidthWriteRef.current = false) }
)
}, [])
const handleColumnAutoResize = useCallback((columnKey: string) => {
@@ -1616,7 +1674,11 @@ export function TableGrid({
setColumnWidths((prev) => ({ ...prev, [columnKey]: newWidth }))
const updated = { ...columnWidthsRef.current, [columnKey]: newWidth }
columnWidthsRef.current = updated
updateMetadataRef.current({ columnWidths: updated })
pendingWidthWriteRef.current = true
updateMetadataRef.current(
{ columnWidths: updated },
{ onSettled: () => (pendingWidthWriteRef.current = false) }
)
}, [])
const handleColumnDragStart = useCallback((columnName: string) => {
@@ -1921,28 +1983,45 @@ export function TableGrid({
return
}
if (!source) return
// After first load: only re-seed `columnOrder` when the *set of columns*
// changes (e.g. a workflow group adds/removes outputs server-side). Pure
// reorders are left alone so an in-flight optimistic drag isn't clobbered
// by a refetch returning the pre-drag order.
// After first load a collaborator (or our own committed edit) reshaped the layout.
// Re-apply it live from the active layout source (a view's config when one is active,
// else the table's own metadata), but never clobber the gesture the local user is
// mid-way through — their in-progress value leads the server's. Each field is guarded
// by reference: React Query structural sharing keeps an unchanged sub-object
// referentially stable, so an unrelated change (e.g. a peer's pin) doesn't re-apply
// widths/order.
// Width: keep the column being actively resized on its live local value.
const serverWidths = source.columnWidths
if (serverWidths && serverWidths !== columnWidthsRef.current) {
const resizing = resizingColumnRef.current
const localWidth = resizing ? columnWidthsRef.current[resizing] : undefined
if (resizing && localWidth !== undefined) {
setColumnWidths({ ...serverWidths, [resizing]: localWidth })
} else if (!pendingWidthWriteRef.current) {
setColumnWidths(serverWidths)
}
// else: a just-committed local width write is still in flight — local leads until
// its onSettled invalidation brings back the server's committed (merged) widths.
}
// Pins toggle instantly (no in-progress gesture) — apply on change.
const serverPins = source.pinnedColumns
if (serverPins && serverPins !== pinnedColumnsRef.current) {
setPinnedColumns(serverPins)
}
// Order: apply the server order live (a peer reorder or our own committed edit),
// unless a local column drag is in flight (an optimistic reorder would otherwise be
// reverted to the pre-drag order the refetch returns). Preserve our own just-appended
// ids whose patch is still in flight by appending them — `viewLayout` gets a new
// identity on every save, so a refetch/view-save predating the append must not drop
// them; `displayColumns` harmlessly skips any id with no matching column.
const serverOrder = source.columnOrder
if (serverOrder) {
if (serverOrder && serverOrder !== columnOrderRef.current && !dragColumnNameRef.current) {
const localOrder = columnOrderRef.current
if (!localOrder) {
setColumnOrder(serverOrder)
} else {
// Re-seed only when the server knows an id the local order lacks — a real
// schema change (a workflow group gained outputs). Ids present locally but
// NOT on the server are our own just-appended columns whose patch is still
// in flight: `viewLayout` gets a new identity on every save, so a refetch
// carrying the pre-append order would otherwise roll them back, and the
// append effect can't re-fire because `columns` is unchanged. Ids the
// server drops stay in the local order harmlessly — `displayColumns`
// skips any id with no matching column.
const localSet = new Set(localOrder)
if (serverOrder.some((id) => !localSet.has(id))) {
setColumnOrder(serverOrder)
}
const localOnly = localOrder.filter((id) => !serverOrder.includes(id))
setColumnOrder(localOnly.length > 0 ? [...serverOrder, ...localOnly] : serverOrder)
}
}
}, [tableData?.metadata, viewLayout, viewLayoutKey])
@@ -4253,6 +4332,15 @@ export function TableGrid({
})()}
</tbody>
</table>
{remoteSelections.length > 0 && (
<RemoteSelectionOverlay
remoteSelections={remoteSelections}
columnIndexById={columnIndexById}
rowIndexById={rowIndexById}
localSelection={normalizedSelection}
scrollElement={scrollRef.current}
/>
)}
{resizingColumn && (
<div
className='-translate-x-[1.5px] pointer-events-none absolute top-0 z-20 h-full w-[2px] bg-[var(--selection)]'
@@ -83,6 +83,25 @@ export interface NormalizedSelection {
anchorCol: number
}
/**
* Whether a (row, col) index pair falls inside a normalized selection rectangle.
* Row/col may be `undefined` (e.g. an id that didn't resolve to an index) `false`.
*/
export function isCellInSelection(
row: number | undefined,
col: number | undefined,
sel: NormalizedSelection
): boolean {
return (
row !== undefined &&
col !== undefined &&
row >= sel.startRow &&
row <= sel.endRow &&
col >= sel.startCol &&
col <= sel.endCol
)
}
/** A run of consecutive `displayColumns` rendered together in the meta header row. */
export type HeaderGroup =
| { kind: 'plain'; size: 1; startColIndex: number }
@@ -1,3 +1,4 @@
export * from './use-context-menu'
export * from './use-table'
export * from './use-table-event-stream'
export * from './use-table-room'
@@ -379,14 +379,38 @@ export function useTableEventStream({
else if (entry.event?.kind === 'dispatch') applyDispatch(entry.event)
else if (entry.event?.kind === 'job') applyJob(entry.event)
else if (entry.event?.kind === 'usageLimitReached') applyUsageLimit(entry.event)
// A collaborator's manual edit: refetch rows (debounced) so the winning
// last-write value shows live, in this client's own wire format.
else if (entry.event?.kind === 'edit') scheduleRowsInvalidate()
// A collaborator changed the table structure: mirror the local
// invalidateTableSchema set — the definition (exact, so rows stay on the
// debounce), the run-state + enrichment sibling queries under detail (a group
// delete/restructure can otherwise leave a stale running badge or enrichment
// panel), the tables list (column/row counts), and the debounced rows.
else if (entry.event?.kind === 'schema') {
void queryClient.invalidateQueries({ queryKey: tableKeys.detail(tableId), exact: true })
void queryClient.invalidateQueries({ queryKey: tableKeys.activeDispatches(tableId) })
void queryClient.invalidateQueries({ queryKey: tableKeys.enrichmentDetails(tableId) })
void queryClient.invalidateQueries({ queryKey: tableKeys.lists() })
scheduleRowsInvalidate()
}
// A collaborator changed the column layout (width/pin/order): refetch the
// definition alone (it carries the metadata) — the grid re-applies it without
// a rows refetch. Exact, so rows/run-state stay put.
else if (entry.event?.kind === 'metadata') {
void queryClient.invalidateQueries({ queryKey: tableKeys.detail(tableId), exact: true })
}
// A collaborator toggled a table lock: re-read the definition so every open
// viewer's gating updates. `exact` avoids refetching every rows page
// (rowsRoot nests under detail); no row data changed.
else if (entry.event?.kind === 'definition') {
// A lock/schema change on the definition — re-read it so every open
// viewer's gating updates. `exact` avoids refetching every rows page
// (rowsRoot nests under detail); no row data changed.
void queryClient.invalidateQueries({
queryKey: tableKeys.detail(tableId),
exact: true,
})
void queryClient.invalidateQueries({ queryKey: tableKeys.detail(tableId), exact: true })
}
// A collaborator changed the table's shared saved views (create/rename/delete/
// re-save): refetch the views list alone. Views are presentation state layered on
// the already-loaded table, so no rows/definition refetch is needed.
else if (entry.event?.kind === 'views') {
void queryClient.invalidateQueries({ queryKey: tableKeys.views(tableId) })
}
} catch (err) {
logger.warn('Failed to parse table event', { tableId, err })

Some files were not shown because too many files have changed in this diff Show More