mirror of https://github.com/simstudioai/sim.git synced 2026-09-24 15:45:35 +08:00

Files

T

Theodore Li 24a6086d95 feat(tables): fractional order keys for O(log n) row insert/delete (flag-gated, default off) (#4890 )

* feat(tables): add order_key column, fractional-indexing util, and ordering flag (off)

* feat(tables): write order_key on insert, flag-gate delete reindex + query ordering, add backfill

Flag off (default) = identical behavior. Single-insert assigns a fractional
order_key; queryRows orders by order_key when the flag is on; deletes skip the
O(N) reindex when on. Per-table-atomic backfill script populates existing rows.

* feat(tables): write order_key on all insert paths (batch, upsert, replace, import, create, copilot)

Completes the always-write-keys prerequisite: every row insert now assigns a
fractional order_key consistent with position order, so the flag can be flipped
safely after backfill. Flag off (default) still = identical behavior.

* feat(tables): insert-by-neighbor-id + orderKey on wire + client order-by-key

Inserts express intent as afterRowId/beforeRowId (O(1) key mint via the
(table_id,order_key,id) index); orderKey is returned on every row; client
reconcile/undo place by orderKey (no neighbor bump) with position fallback.
Flag off = unchanged. 205 table tests pass.

* feat(tables): resolve position-based inserts by key ordinal under the flag

Position-based callers (mothership tool, v1 API, undo fallback, transient old
clients) resolve their insert neighbor by order_key ordinal (OFFSET) when the
flag is on — positions are gappy then, so WHERE position=N would miss. Flag off
keeps the indexed position lookup. The mothership tool itself is unchanged.

* test(tables): flag-on coverage — delete skips reindex, insert mints key + no shift

* fix lint

* chore(db): regenerate order_key migration with default drizzle name

* fix(tables): address review — guard neighbor insert + mutual-exclusion + safe reconcile

- resolveInsertByNeighbor throws when the anchor row is missing (was silently
  inserting at the front) and when its order_key is null under the flag.
- insert contract: afterRowId/beforeRowId are mutually exclusive (refine).
- reconcileCreatedRow only key-sorts when every cached row is keyed, so mid-
  backfill un-keyed rows aren't yanked to the front.

* fix(kb): restore non-null guard in storage-key filter (unsafe-lint regression)

* refactor(tables): extract maxOrderKey + thread import append key

- Extract maxOrderKey(executor, tableId) helper; replaces three identical
  max(order_key) selects (single/batch insert append + import).
- Import: read the append anchor once up front and thread each batch's last
  key forward (nextImportStartOrderKey + afterOrderKey) instead of re-scanning
  max(order_key) per batch over a growing table — one scan per import, not
  one per 1k-row batch.

* fix(tables): keep insert body base omittable for v1 contract

The afterRowId/beforeRowId mutual-exclusion .refine() turned the schema into a
ZodEffects, which Zod forbids .omit() on — v1's insertTableRowBodySchema.omit({
position }) threw at module load (runtime-only; tsc misses it). Split the plain
object base out, apply the shared refine on top, and have v1 omit from the base
then re-apply it.

* fix(tables): chunk backfill order-key writes

A single UPDATE … FROM (VALUES …) over a whole large table overflows the JS call
stack while drizzle assembles the VALUES list (and would blow past Postgres's
65535 bound-param limit at ~32k rows) — large tables failed with 'Maximum call
stack size exceeded'. Write in 1000-row chunks inside the same per-table
transaction so keying stays atomic.

* fix(tables): emit orderKey in insert responses

The single-row and batch insert handlers dropped orderKey from the JSON
response even though the service returns it, so reconcileCreatedRow always fell
back to position-sorting and could place neighbor inserts wrong under the
fractional-ordering flag. Serialize orderKey alongside position.

* fix(tables): restore by orderKey, not position, under fractional flag

A saved position is the gappy column value, but under the flag insert reads
position as a visual rank (OFFSET) — so position-based restore misplaces rows.

- create-row redo now goes through the batch path carrying the saved orderKey
  (the single-insert API has no orderKey field); drop the now-unused single
  create mutation.
- resolveBatchInsertOrderKeys appends under the flag instead of feeding gappy
  positions to resolveInsertOrderKey; positions remain the flag-off path.

* perf(tables): backfill writes 5000 rows/chunk (was 1000)

5x fewer round-trips per table; ~10k bound params stays well under Postgres's
65535 ceiling and far below the single-statement size that overflows the stack.

* fix(tables): drop rowNumber from table trigger payload

position is gappy under the fractional-ordering flag, so rowNumber (= row.position)
no longer reflects a contiguous visual rank. Rather than compute-on-read, remove
it from the trigger payload, output schema, and column-execution input.

Also pin isTablesFractionalOrderingEnabled=false in update-row.test.ts so its
flag-off position-shift assertions are deterministic regardless of local env.

* chore(db): format generated 0226 migration metadata

biome check . flagged the drizzle-generated _journal.json and 0226_snapshot.json;
apply the formatter so packages/db lint:check passes in CI.

* docs(triggers): drop rowNumber from table trigger outputs

rowNumber was removed from the table trigger payload; remove it from the
documented output fields to match.

* test(tables): remove flag-on fractional-ordering unit suite

Flag-on behavior is covered by manual large-table verification; the heavily-
mocked DB-chain suite added little signal.

2026-06-05 17:18:33 -04:00