mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
Waleed
ef225f99ef
* fix(connectors): index Office documents and PDFs from SharePoint and OneDrive
The SharePoint and OneDrive connectors filtered their listings against a
12-item plain-text extension whitelist, so a document library of .docx, .pdf
or .xlsx files synced as "success, 0 documents" — no document, no failed row,
and no log line, which is indistinguishable from a wrong folder path. Both
whitelists had been unchanged since the connectors shipped, and Sim already
parses all of these formats for a manually uploaded knowledge base document.
Adds a shared `extractConnectorText` in connectors/utils that routes binary
document formats through the same `parseBuffer` the upload path uses, so the
OOXML zip-bomb guard and each parser's extraction limits apply. The
previously-accepted text formats stay on their exact existing path: sending
.csv through CsvParser would silently reformat every already-indexed connector
document on its next re-index.
Also logs a per-page count of files skipped for an unsupported extension.
Unsupported files are counted rather than turned into failed document rows, so
a library full of images does not fill the knowledge base with noise.
* fix(connectors): never index a degraded document extraction
`DocParser` and `PptxParser` never throw by design — on a legacy OLE `.doc`/`.ppt`
or a deck with no extractable text they return a placeholder sentence or scraped
ZIP internals so an interactive upload still shows the user something. Verified
against real OOXML fixtures: an image-only `.pptx` yields 1.9KB of
`[Content_Types].xml…` as "content", and a legacy `.ppt` yields "Unable to
extract text from PowerPoint file."
A connector sync would embed that into the vector index at scale, so it needs to
tell a real extraction from a fabricated one. Adds a declared `degraded` flag to
`FileParseMetadata`, set by exactly those two fallback paths, rather than having
callers sniff `extractionMethod`. `DocParser`'s plaintext branch stays unflagged:
a text file misnamed `.doc` is a genuine extraction.
`extractConnectorText` now raises `ConnectorTextExtractionError` when a parsed
format comes back degraded or blank, and SharePoint/OneDrive surface it as a
skipped document via the existing `markSkipped` path — so the file appears in the
knowledge base as a failed row telling the user to re-save it as DOCX/PPTX/XLSX,
instead of being silently dropped or indexed as junk.
The upload path is unaffected; it ignores the new flag.
* fix(file-parsers): register the document variants the parsers already handle
A document library holds whole format families, not just the headline extension
of each. These all extract correctly with the libraries already installed — they
were simply never registered, so every one of them was reported as an
unsupported file type:
docm dotx (WordprocessingML — mammoth reads word/document.xml regardless of
the package content type)
xlsm xlsb xltx ods (SheetJS reads every workbook container natively)
pptm potx (PresentationML)
odt odp (OpenDocument, via a new OpenDocumentParser)
Verified against real fixtures built with jszip and SheetJS rather than assumed:
officeparser identifies a Buffer by sniffing content with `file-type`, not by the
name we pass, so the routing had to be measured. `ods` goes to the spreadsheet
parser rather than OpenDocumentParser so its output keeps per-sheet structure.
`rtf` is deliberately excluded: nothing bundled extracts it, and DocParser's
plaintext branch would pass its control words through as if they were prose.
Converts the registry from `require()` inside per-parser `try/catch` blocks that
only logged to static imports. Every parser dependency is a regular, non-optional
one, so a resolution failure should fail loudly — the old form produced a silently
**empty** registry in which every format became `Unsupported file type`, with an
empty "Supported types are:" list as the only clue. The heavy extraction libraries
are still deferred inside the individual parsers, and connectors now import the
registry lazily so the ~60 connectors that never touch a file do not pull SheetJS.
Adds registry.test.ts, which exercises the real module: index.test.ts mocks
`@/lib/file-parsers` itself, so it validated its own fake routing table and the
real registry had no coverage at all. The new test gates every member of
SupportedFileType on having a registered parser that supports buffer parsing.
* fix(file-parsers): resolve the parser registry through a Map, not object keys
The registry rewrite switched extension lookup from
`Object.keys(parsers).includes(ext)` to a bracket read on an object literal,
which also resolves inherited keys. `PARSERS['constructor']` therefore returned
`Object` — truthy, with no parse methods — so a caller-supplied extension of
`constructor` fell through to "does not support buffer parsing" instead of being
rejected as an unsupported type, and `parseFile` would have raised a TypeError.
It also disagreed with `isSupportedFileType`, which used `Object.hasOwn` and
correctly returned false for the same input.
A Map has no prototype chain to walk, so lookup and support check now agree by
construction. `isSupportedFileType` also guards a non-string argument, which the
try/catch it replaced used to absorb.