mirror of
https://github.com/Tencent/WeKnora.git
synced 2026-11-03 22:43:26 +08:00
nullkey
6492bf6a15
Rework the Feishu datasource connector from whole-document export to the docx blocks API, so embedded content that export silently dropped is now ingested, and add the multi-item ingestion infrastructure it needs. Feishu connector: - Embedded online sheets and bitables are read via their APIs and inlined as Markdown tables — searchable text, instead of being lost or (post-export) rendered as unsearchable images. Bitable date/datetime cells render in the table's timezone (configurable via Settings["timezone"], default GMT+8) rather than a raw epoch-ms integer; a datetime formatter keeps the time component. - Uploaded attachments (>2KB, parseable types) become standalone knowledge sub-items so their text is parsed and retrievable. - Embedded images become standalone sub-items routed through the KB's VLM OCR pipeline, gated by the KB's multimodal setting: the image's external_id always stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image) but the bytes are only downloaded/ingested when multimodal is enabled. - Sub-documents (child wiki nodes) sync as their own nodes. - The three paginated Feishu loops (document blocks, bitable fields, bitable records) break on an empty page so a malformed has_more response cannot spin until the task deadline. Datasource ingestion infrastructure: - Multi-item subtree reconciliation: a source node fans out into a parent plus "<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the sweep deletes prior children absent from SubtreeKeep. A per-child keep-set means a single failed attachment can't wipe its siblings, and dupIsSameNode guards the sweep so a file_hash-only dedup collision against a different node cannot delete this node's children. - config.MultimodalEnabled mirrors the KB's VLM state to the connector per run (never persisted); an image a non-VLM KB rejects is skipped, not failed. - A composite partial expression index (text_pattern_ops) backs the sweep's metadata external_id prefix scan. Verified end-to-end against a live Feishu wiki: docx text, native tables, embedded sheets/bitables (with timezone-correct dates), attachments, images (VLM OCR), and sub-documents all sync, parse, and are retrievable.