mirror of
https://github.com/Tencent/WeKnora.git
synced 2026-08-29 02:04:30 +08:00
Greenplumwine
439af21f9b
* feat(datasource): add Feishu/Lark Drive (云盘) connector Add a new Drive (云盘) data source alongside the existing Feishu wiki connector, letting users sync documents/files under a user-supplied Drive folder_token. Shares the feishu connector package (Client/Config/Region/ export/download/error-handling) with the wiki connector; only resource enumeration and fetch dispatch differ. Backend: - types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive - region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL - types.go: driveFile / driveFileListResponse / partialDriveFileListError / feishuDriveCursor - client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut expansion via shortcut_info, visited cycle guard, root-folder rejection) - drive_connector.go: DriveConnector implementing StreamingConnector (FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring the wiki connector's three cursor semantics (resume fast-path, failure-does-not-advance-cursor, toSyncCursor snapshot), plus ListResources / ResolveResourceAncestors (top-down shared traversal, best-effort) / fetchDriveFileContent / parseDriveResourceID - container.go + connector.go: register two Drive instances + metadata Incremental detection uses modified_time returned directly by the list API (verified) - no batch_query/metas call needed. Shortcuts are expanded at list-parse time via shortcut_info; target_type is never "folder". Frontend: - DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive, Drive root folder_token input + load button before the lazy-load tree, edit-mode prefill from resource_ids - datasourceIcons.ts: reuse feishu/lark icons - i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive input group Co-Authored-By: Claude <noreply@anthropic.com> * fix(datasource): refine Feishu Drive connector UI, channel and root folder A batch of fixes from end-to-end verification of the Feishu/Lark Drive connector, all aimed at matching the wiki connector's behavior: Channel (knowledge "source" label): - Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show "飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki connector's "飞书" (ChannelFeishu). - fetchDriveFileContent / appendDriveFileListFailureItems select the channel by region via a new driveChannel() helper. - ingestItem now prefers metadata["channel"] over ds.Type so the connector-supplied channel value reaches the knowledge base. - doc-content.vue channelLabelMap maps feishu_drive/lark_drive. Frontend Drive picker UX: - loadDriveRoot persists the new folder_token in edit mode too (previously skipped updates in edit mode, so listResources read the old token). - The folder_token input is now always visible alongside the tree (not a switch); the tree area shows a "load a folder first" placeholder until the first successful load. - extractDriveFolderToken accepts a bare token or a Feishu Drive folder URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token. - Edit mode auto-loads when a saved folder_token exists, and reveals pre-existing selections via revealExistingSelections. - Classify Drive list errors (403/forbidden, 401/auth, not-found) into actionable i18n hints. Root folder name & selection: - Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta) to resolve the root folder's human-readable name; the list API only returns children. Best-effort, falls back to the token. - Fix root folder ExternalID to be the bare rootFolderToken (no ":token" suffix) so it matches the saved resource_id form and the direct children's ParentID, which previously broke selection matching on edit. Co-Authored-By: Claude <noreply@anthropic.com> * fix(datasource): sync a single selected Drive file without folder error When a user selects a single file (not a folder) in the Drive picker, the resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/ FetchIncremental previously passed fileToken straight to ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu list API returns 1061002 (params error) for a file token, failing the whole sync with "all fetched items failed". The wiki connector resolves a single selected node via GetWikiNode; Drive has no single-file meta API (folder meta only works on folders, and metas/batch_query requires doc_type as input - a chicken-and-egg). Instead, listDriveFilesForResource walks the root folder subtree (the file's parent) and filters to the selected fileToken, yielding the full driveFile (type/name/modified_time) needed for fetchDriveFileContent. A sub-folder selection (fileToken is itself a folder) still walks that sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token, so no filtering is needed there. Co-Authored-By: Claude <noreply@anthropic.com> * ci: add build-test-images workflow for tar.gz artifact export Manual-only workflow_dispatch that builds linux/amd64 Docker images for the current branch and exports them as .tar.gz artifacts downloadable from the Actions run page. No Docker Hub push, no secrets required. - build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz - build-app: builds weknora-app image (CGO + duckdb), exports tar.gz - Input choice: build both / app only / ui only - Artifacts retained 7 days Co-Authored-By: Claude <noreply@anthropic.com> * ci: add build-test-images workflow for tar.gz artifact export Manual-only workflow_dispatch that builds linux/amd64 Docker images for a selected branch and exports them as .tar.gz artifacts downloadable from the Actions run page. No Docker Hub push, no secrets required. - build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz - build-app: builds weknora-app image (CGO + duckdb), exports tar.gz - Input choice: build both / app only / ui only - Artifacts retained 7 days Placed on the default branch so the workflow appears in the Actions list; trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive). Co-Authored-By: Claude <noreply@anthropic.com> * docs(datasource): add feishu shared-blocks extraction design spec Co-Authored-By: Claude <noreply@anthropic.com> * docs(datasource): add feishu shared-blocks implementation plan Co-Authored-By: Claude <noreply@anthropic.com> * feat(datasource): sync feishu drive docx via blocks API with export fallback - Extract wiki/drive shared helpers from connector.go into shared.go - Make fetchDocxWithBlocks a package-level function taking docxFetchInput - Drive docx now renders via blocks API (Markdown + attachment/image sub-items), falling back to the export API on blocks errors or empty renders, mirroring the wiki connector - fetchDriveFileContent returns item slices and takes multimodalEnabled Co-Authored-By: Claude <noreply@anthropic.com> * fix(frontend): label feishu drive knowledge as 飞书云盘 in list The knowledge list source column (DocumentListView.getSourceInfo) only whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs fell through to the default "上传" label while the wiki connector's "feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive → Lark 云盘 (i18n keys already existed), and add feishu_drive to the source filter options in KnowledgeBase.vue. Co-Authored-By: Claude <noreply@anthropic.com> * fix(frontend): polish feishu drive resource-step UI in datasource editor - Replace the two stacked t-alert banners under the folder_token input: shareHint becomes an always-visible form-desc line, rootNotSupportedHint moves into a help-circle tooltip next to the label. - folder_token is now required: red asterisk on the label, inline error status + tips on the input (replacing the global MessagePlugin) when empty on 加载/下一步, cleared on input. - Step 2 选择范围 no longer scrolls as a whole: the token input stays fixed and the resource region (placeholder / loading / empty / tree) fills the remaining drawer height, with the tree scrolling internally. Co-Authored-By: Claude <noreply@anthropic.com> * docs(datasource): add feishu drive datasource integration guide User-facing setup guide for feishu_drive/lark_drive: app creation, required scopes (drive:drive:readonly / drive:export:readonly / docx:document:readonly), sharing the folder to the app's group, the four-step wizard, supported file types, sync semantics and FAQ. Also clarify in extractDriveFolderToken's comment that pasted folder URLs are matched by path and work for Lark links too. Co-Authored-By: Claude <noreply@anthropic.com> * refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine. Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs. Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages. Co-Authored-By: Claude <noreply@anthropic.com> * refactor(datasource): split feishu connector into core/wiki/drive packages Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references. container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core). Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes). Co-Authored-By: Claude <noreply@anthropic.com> * test(datasource): migrate feishu tests to core/wiki/drive packages Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script. All tests pass: core 4s, wiki 13s, drive 3s. Co-Authored-By: Claude <noreply@anthropic.com> * refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations). Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test. All tests pass: core 3.5s, wiki 13s, drive 3s. Co-Authored-By: Claude <noreply@anthropic.com> * fix(datasource): repair feishu cleanup fallout Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range. All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes. Co-Authored-By: Claude <noreply@anthropic.com> * feat(frontend): feishu drive data source UI and i18n Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU. Co-Authored-By: Claude <noreply@anthropic.com> * chore: add trellis journal merge=union rule Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal. Co-Authored-By: Claude <noreply@anthropic.com> * refactor(datasource): rename feishu/drive drive_connector.go to connector.go Aligns naming with feishu/wiki subpackage (connector.go). Co-Authored-By: Claude <noreply@anthropic.com> * fix(frontend): allow data:image base64 URIs in sanitized markdown DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/..."> in uploaded markdown files had their src stripped during sanitizeHTML — the file preview showed broken images while the chunk view (minio:// via hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so base64 images render in preview and in LLM answers that embed base64 images. Co-Authored-By: Claude <noreply@anthropic.com> * feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path The blocks API path renders image blocks as empty ![图片]() placeholders and fans images into separate knowledge items, breaking image↔document association across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async export API (.docx binary -> docreader), so images are parsed inline and bound to the parent document via parent_chunk_id (same as a regular docx upload). Default (unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki and drive connectors since they share FetchDocxWithBlocks. Co-Authored-By: Claude <noreply@anthropic.com> * feat(datasource): default FEISHU_DOCX_PARSE_MODE to export Switch the feishu docx parse default from blocks to export so embedded images are parsed inline and bound to the parent document via parent_chunk_id (retrieval / wiki / agent all associate images). Operators who need the blocks behaviour (faster, keeps docx attachments, no docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks. - shared.go: default unset -> export; update comment to match - .env.example / docker-compose.yml: default export - tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so they don't break under the new default Co-Authored-By: Claude <noreply@anthropic.com> * docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout - 飞书云盘接入说明: add section on docx parse mode (export default vs blocks), permission differences, export tradeoffs - 数据源导入开发: add env var + mode comparison - 数据源导入开发文档: rewrite content fetch flow (blocks/export paths), add pros/cons table, fix source-file table to core/wiki/drive layout Co-Authored-By: Claude <noreply@anthropic.com> * fix: 修复合并main后丢失新增的国际化问题 * fix(i18n): 修复误删的国际化内容 * docs: 删除误提交的文件 --------- Co-authored-by: Claude <noreply@anthropic.com>