Waleed c7130c6bf9 fix(tables): fix bulk ops truncation for tables larger than one page (#4532)
* fix(tables): fix bulk ops truncation for tables larger than one page

Bulk operations (column-header delete, select-all copy/cut/delete/run)
were silently truncated to the first 1000 rows because handlers only
iterated the loaded pages from useInfiniteQuery.

Fix:
- Extract tableRowsInfiniteOptions factory (infiniteQueryOptions) so
  the hook and imperative drain share the same typed cache key
- Add background drain via useEffect watching hasNextPage/isFetchingNextPage
  — chains fetchNextPage until getNextPageParam returns undefined
- Add ensureAllRowsLoaded to use-table: reads cache via getQueryData +
  calls fetchNextPage in a while loop until the last page is partial
- Await ensureAllRowsLoaded at every kind:'all' bulk-op entry point in
  table-grid (column delete, copy, cut, action-bar delete/run)
- Add chunkBatchUpdates to send updates in MAX_BULK_OPERATION_SIZE=1000
  chunks so server validation never rejects oversized batches
- Fix undo-redo: make executeAction async and chunk clear-cells,
  update-cells, and delete-column cell-restore with mutateAsync loops

Tests: 41 passing across use-table, tables queries, and use-table-undo

* chore(tables): remove extraneous comments

* fix(tables): add missing useEffect import; chunk range-selection delete and cut

* fix(tables): add hasRunningGroupExecution and import it

* fix(tables): define mergePagePreservingIdentity helper used in polling cache merge

* fix(tables): define ROWS_POLL_INTERVAL_WHILE_RUNNING_MS constant used in polling loop

* fix(tables): capture rowSel before await in delete handler; handle clipboard NotAllowedError

* fix(tables): abort cut on clipboard NotAllowedError to prevent silent cell deletion

* fix(tables): push undo before chunkBatchUpdates to survive partial chunk failures

* fix(tables): use text input for number cells; idle poll backoff; csv error toast; column-cut drain

* fix(tables): audit fixes — column-copy drain, polling scope, merge identity

- Fix polling tick: move Promise.all inside else-branch so dirty[] stays in
  scope; keep hasDirty=true during active mutations so the short interval
  fires while chunked batch-updates are in flight
- Add isColumnSelectionRef branch to handleCopy (mirrors handleCut fix):
  column-header Cmd+C now drains all pages before building clipboard content
- Replace String(updatedAt) comparison in mergePagePreservingIdentity with
  Date.getTime() equality — handles ISO vs +00:00 timezone variants
- Remove redundant batchUpdates.length > 0 guards at chunkBatchUpdates
  callsites (empty-array case is handled inside the function)
- Export _mergePagePreservingIdentity for unit testing
- Add 6 unit tests covering mergePagePreservingIdentity edge cases

* improvement(tables): cleanup — extract components, stabilize callbacks, fix ref sync

* improvement(tables): remove polling, eager drain, and parallelize batch updates

- Drop the per-page polling loop — SSE stream already patches execution
  cell state in real time and invalidates on buffer prune; polling was
  redundant and burned CPU/network on every open table
- Remove eager mount drain (fetchNextPage loop in use-table.ts); scroll
  handler and ensureAllRowsLoaded handle progressive/on-demand loading
- Parallelize chunkBatchUpdates with a 3-worker pool instead of serial
  chunks, reducing bulk-op round-trips by ~3x
- Delete mergePagePreservingIdentity and its tests (no longer called)

* chore(tables): remove stale comments and dead defensive code

- Fix chunkBatchUpdates JSDoc to reflect parallel dispatch (was "sequentially")
- Inline CHUNK_CONCURRENCY=3, single-use constant needs no abstraction
- Drop stale "Polls while any cell is in flight" from useTableRows JSDoc
- Remove two generic "Validation errors surfaced by caller" comments
- Remove ASCII separator line from workflow group mutations section
- Remove dead `if (!variables) return` guard in useImportCsvIntoTable onSettled
  (TanStack v5 always provides variables to onSettled)

* fix(tables): run-all selection sends all rows to server, not just loaded pages

When rowSelection.kind === 'all', selectedRunScope now flags allRows: true.
The action-bar run handlers pass rowIds: undefined to the server when allRows
is set, matching the server contract (missing rowIds = run all eligible rows).
Stop likewise routes through scope: 'all' instead of per-row cancels.

Previously, selecting all rows and clicking Run would silently only run the
rows loaded in the current infinite-query cache (potentially one page of 1000
on a 5000-row table).

* test(tables): remove background-drain describe block (behavior intentionally removed)

* fix(tables): surface CSV import error toast; remove dead hasRunningGroupExecution

* fix(tables): make runWithoutRecording async-aware so undoRedoInProgress covers full undo execution

* feat(tables): render URL cells with favicon and clickable link

* feat(tables): clickable URL cells with favicons using tldts
2026-05-09 13:21:42 -07:00

Sim Logo

The open-source platform to build AI agents and run your agentic workforce. Connect 1,000+ integrations and LLMs to orchestrate agentic workflows.

Sim.ai Discord Twitter Documentation

Ask DeepWiki Set Up with Cursor

Build Workflows with Ease

Design agent workflows visually on a canvas—connect agents, tools, and blocks, then run them instantly.

Workflow Builder Demo

Supercharge with Copilot

Leverage Copilot to generate nodes, fix errors, and iterate on flows directly from natural language.

Copilot Demo

Integrate Vector Databases

Upload documents to a vector store and let agents answer questions grounded in your specific content.

Knowledge Uploads and Retrieval Demo

Quickstart

Cloud-hosted: sim.ai

Sim.ai

Self-hosted: NPM Package

npx simstudio

→ http://localhost:3000

Note

Docker must be installed and running on your machine.

Options

Flag Description
-p, --port <port> Port to run Sim on (default 3000)
--no-pull Skip pulling latest Docker images

Self-hosted: Docker Compose

git clone https://github.com/simstudioai/sim.git && cd sim
docker compose -f docker-compose.prod.yml up -d

Open http://localhost:3000

Sim also supports local models via Ollama and vLLM — see the Docker self-hosting docs for setup details.

Self-hosted: Manual Setup

Requirements: Bun, Node.js v20+, PostgreSQL 12+ with pgvector

  1. Clone and install:
git clone https://github.com/simstudioai/sim.git
cd sim
bun install
bun run prepare  # Set up pre-commit hooks
  1. Set up PostgreSQL with pgvector:
docker run --name simstudio-db -e POSTGRES_PASSWORD=your_password -e POSTGRES_DB=simstudio -p 5432:5432 -d pgvector/pgvector:pg17

Or install manually via the pgvector guide.

  1. Configure environment:
cp apps/sim/.env.example apps/sim/.env
# Create your secrets
perl -i -pe "s/your_encryption_key/$(openssl rand -hex 32)/" apps/sim/.env
perl -i -pe "s/your_internal_api_secret/$(openssl rand -hex 32)/" apps/sim/.env
perl -i -pe "s/your_api_encryption_key/$(openssl rand -hex 32)/" apps/sim/.env
# DB configs for migration
cp packages/db/.env.example packages/db/.env
# Edit both .env files to set DATABASE_URL="postgresql://postgres:your_password@localhost:5432/simstudio"
  1. Run migrations:
cd packages/db && bun run db:migrate
  1. Start development servers:
bun run dev:full  # Starts Next.js app and realtime socket server

Or run separately: bun run dev (Next.js) and cd apps/sim && bun run dev:sockets (realtime).

Copilot API Keys

Copilot is a Sim-managed service. To use Copilot on a self-hosted instance:

  • Go to https://sim.ai → Settings → Copilot and generate a Copilot API key
  • Set COPILOT_API_KEY environment variable in your self-hosted apps/sim/.env file to that value

Environment Variables

See the environment variables reference for the full list, or apps/sim/.env.example for defaults.

Tech Stack

Contributing

We welcome contributions! Please see our Contributing Guide for details.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Made with ❤️ by the Sim Team

Languages
TypeScript 77%
MDX 20.8%
JavaScript 1.9%
CSS 0.1%