Waleed
19230bfcd7
* feat(connectors): hand source files to the document pipeline instead of extracting them A connector that extracted text itself stranded the document on a second, weaker parser. The shared pipeline routes PDFs to OCR — the only way a scanned page is readable at all — and owns every other format's parser, but its OCR branch is gated on `mimeType === 'application/pdf'` and connector documents were stored as `text/plain`, so a connector PDF could never reach it. The same file dragged into the UI was read by OCR; synced through a connector it got the local parser. `ExternalDocument` can now carry the source file itself, and SharePoint and OneDrive hand over anything the knowledge base can parse rather than extracting it. The sync engine stores those bytes under the file's own name and type, so the pipeline parses them exactly as it would an upload of the same file. Formats that are already text stay on the text path: HTML still reduces to plain text and the rest are UTF-8 decodes, so nothing already indexed changes representation. The MIME type is derived from the extension rather than the source's own declaration, so a provider that omits or mislabels it cannot strand a PDF on the non-OCR path. Re-syncing an existing document now rewrites `mimeType` too, which is what lets one stored as connector-extracted text stop declaring `text/plain`. This removes the duplicate extraction path rather than leaving both in place: `extractConnectorText` is text-only, and the guard against fabricated content moves to the pipeline where parsing now happens. That guard still matters — `DocParser` and `PptxParser` never throw, returning a placeholder sentence or scraped archive bytes on a legacy binary or an image-only deck — so a `degraded` result now fails the document with the same actionable message it produced before, naming the modern container for legacy formats. The in-flight byte budget already accounted for this: `estimateOpSizeBytes` reads the true source size from listing metadata, so batching reserved against the real file all along and merely over-reserved while only text was stored. * fix(knowledge): guard every parser against empty output, not just the file parsers Moving connector parsing into the pipeline exposed a gap on the OCR branch. OCR reads a scanned page with no recoverable text as empty, and the empty-content guard lived inside the file-parser path, so such a document chunked to nothing and reported success — the same silently-complete-but-useless outcome the guard exists to prevent. The check now sits above the parser choice and covers OCR too. Also preserves a source file's extension when its name is too long for a storage key. The extension is what picks the parser; a truncated name would still parse correctly by falling back to the display name, but only by luck. * fix(knowledge): validate the stored artifact against the parser registry Ten of the formats a connector now hands over — docm, dotx, xlsm, xlsb, xltx, pptm, potx, odt, ods and odp — parse fine but are deliberately not offered as upload types. `resolveStoredArtifactExtension` gated on the upload allowlist, so it rejected every one of them and processing failed with `Unsupported file type`. They worked before only because the connector extracted them itself and stored the result as text. The question the gate is asking is whether a parser can read the stored object, which the parser registry answers; the upload allowlist answers a different question about what we accept from a user. Also matches the sibling comment style in the object literal it sits in, and teaches two test mocks the newly imported symbol. * fix(knowledge): carry the MIME type through hydration A listing stub is built before the file is fetched and declares `text/plain` for everything, so a hydrated PDF kept claiming plain text at the top level. Nothing broke today only because storage reads `sourceFile.mimeType` — which is exactly what makes it a trap: anything later reaching for `extDoc.mimeType`, the obvious field, silently loses the OCR routing this change exists to restore. The merge is now `mergeHydratedDocument` rather than an inline spread, so what hydration must carry is a stated contract with a test behind it instead of a literal that is easy to under-specify — which is how the field was missed.
A workspace to build, deploy and manage AI agents and workflows.
Quickstart
Cloud-hosted: sim.ai
Self-hosted
git clone https://github.com/simstudioai/sim.git && cd sim
bun install
bun run setup
Capabilities
- Connect 1,000+ integrations and every major LLM
- Add Slack, Notion, HubSpot, Salesforce, databases, and more
- Build agents visually, conversationally, or with code
- Ingest files, knowledge bases, and structured table data
- Monitor runs, logs, schedules, and workflow activity
One workspace, every surface
Chat and workflows are just the start — tables, files, and knowledge all live in the same workspace.
Tables — a database, built in |
Files — one store for your team and every agent |
Knowledge — your agents' memory |
Self-hosting
bun run setup is an interactive wizard: it provisions the database, generates secrets, writes your .env files, connects a Chat API key, and starts Sim the way you choose:
- Local dev — run from source to contribute or hack on Sim
- Docker Compose — a self-contained instance for testing self-hosting
- Kubernetes (Helm) — deploy to a local cluster
When it finishes, open http://localhost:3000.
Reconfigure an optional capability without rerunning the full wizard:
bun run setup status
bun run setup email
bun run setup storage
bun run setup sandbox
bun run setup jobs
bun run setup cache
bun run setup knowledge
bun run setup llm
bun run setup integration slack
bun run setup status detects the effective local-dev, Docker Compose, or current-context
Helm configuration and reports configured, missing, or invalid capabilities and OAuth
integrations without printing credential values. This is separate from bun run sim status,
which reports whether installed services are running and healthy.
Manage your install with bun run sim:
bun run sim start | stop | restart # bring your install up / down / cycle
bun run sim update # pull/rebuild and apply Compose images
bun run sim status # what's installed and healthy
bun run sim logs # follow logs
bun run sim doctor # diagnose configuration problems
bun run sim down # remove containers (data kept)
bun run sim reset # archive .env and wipe managed data
sim detects how you're running (Docker Compose, local dev, or Kubernetes) and acts accordingly.
Prefer a bare sim? Run bun link once — but note sim lands in ~/.bun/bin, which Homebrew's bun doesn't add to your PATH, so you may need export PATH="$HOME/.bun/bin:$PATH" in your shell profile.
Sim also supports local models via Ollama and vLLM. See the self-hosting docs for details.
Chat API Keys
Chat is a Sim-managed service. bun run setup connects a Chat API key for you — sign in when it opens your browser and the key is stored automatically. To view, create, or revoke keys later, go to sim.ai/selfhost/settings/chat-keys.
Environment Variables
See the environment variables reference for the full list, or apps/sim/.env.example for defaults.
Tech Stack
Next.js · Bun · PostgreSQL · Drizzle · Better Auth · Tailwind — and the rest of the stack
- Framework: Next.js (App Router)
- Runtime: Bun
- Database: PostgreSQL with Drizzle ORM
- Authentication: Better Auth
- Schema Validation: Zod
- UI: Shadcn, Tailwind CSS
- Streaming Markdown: Streamdown
- State Management: Zustand, TanStack Query
- Flow Editor: ReactFlow
- Docs: Fumadocs
- Monorepo: Turborepo
- Realtime: Socket.io
- Background Jobs: Trigger.dev
- Remote Code Execution: E2B
- Isolated Code Execution: isolated-vm
Contributing
We welcome contributions! Please see our Contributing Guide for details.
License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
