SanitizeForLog is for log output and rewrites tabs/newlines/control
characters. Applying it to req.Password during public registration
changed the credential that was hashed, so login with the original
password failed. Match the invite registration path and pass the
password through unchanged.
Fixes#2521
Introduce a new entrypoint script for the Nginx container to ensure proper startup behavior. Update the Dockerfile to use a stable-alpine Nginx base image and include the entrypoint script, enhancing the deployment process for the documentation site.
Update the Nginx configuration to listen on port 8081 instead of the default port 80. This change is necessary for serving the documentation site correctly in the current deployment setup.
Introduce a Dockerfile for building the documentation site using VitePress and an Nginx configuration for serving the built assets. The Dockerfile sets up a multi-stage build process, while the Nginx config handles routing and caching for the /docs/ path. Update VitePress configuration to set the base path for asset links. This enhances the deployment process for the WeKnora documentation site.
Follow up on #2567 by documenting the upgrade workflow, syncing JA/KO
READMEs and installation guide, and clarifying that WEKNORA_VERSION
changes require docker compose pull to avoid stale cached images.
Import and update WeKnora documentation with VitePress: getting started,
architecture, features, API reference, clients, and development guides.
Includes landing page with inline SVG illustrations, theme components,
doc check scripts, and product screenshots.
Follow up on #2567 by documenting the upgrade workflow, syncing JA/KO
READMEs and installation guide, and clarifying that WEKNORA_VERSION
changes require docker compose pull to avoid stale cached images.
Users downloading a release and running `docker compose up -d` may get
cached old images because Docker doesn't pull by default. This caused
confusion when the UI displayed an older version than the release.
- Add `docker compose pull` before `up -d` in both README and README_CN
- Enable WEKNORA_VERSION in .env.example so users can pin a release tag
Closes#2560
This commit sets up a new documentation site for WeKnora using VitePress. Key additions include:
- A structured `.gitignore` to manage ignored files.
- An `index.md` file serving as the landing page with a clear title and layout.
- A `package.json` and `package-lock.json` to manage dependencies, including VitePress and related plugins.
- Thematic styles and components for a cohesive user experience, including a custom landing page and zoom functionality for diagrams.
- A README update to guide users on local development and site structure.
This foundational setup aims to enhance user onboarding and provide comprehensive documentation for the WeKnora system.
This commit introduces a complete set of documentation for WeKnora, organized into six main sections: Getting Started, Architecture, Features, API Reference, Clients, and Development Guide. Key highlights include:
- Detailed installation instructions and quick start guides.
- In-depth architectural overview and backend design documentation.
- Comprehensive feature descriptions, including multi-tenant support, document parsing, and agent capabilities.
- API reference covering approximately 360 endpoints with usage examples.
- Client documentation for various interfaces including web, CLI, and SDK.
This documentation aims to facilitate user onboarding and provide a thorough understanding of the WeKnora system.
* chore(types): add internal ContentRewritten to types.SearchResult
* test(chunks): pin rewritten results out of the position path
Pipeline-replaced content (parent or neighbor expansion) must not trust
stale coordinates even when the replacement satisfies the length
invariant, and must not extend the merged group's range.
* fix(chunks): keep pipeline-rewritten results off the position path
Parent and neighbor expansion replace Content after retrieval. Without a
marker their stale coordinates could be incorrectly trusted.
Ensure ContentRewritten is false in chunkTrusted.
Position-aware merging still routed trimming through AppendWithOverlap,
which searches for the longest suffix match inside a window. On periodic
text (table rows, log lines) that search mistakes the repetition for the
overlap and drops real content, and with a zero overlap there is nothing
to search for in the first place.
When coordinates are trusted the overlap is already known, so verify it
character for character and cut exactly, falling back to the text matcher
only when the bodies disagree inside the overlap.
- Overlaps wider than the 400-rune text window must be trimmed once.
- Adjacent chunks built from one repeating row must keep every row.
- A length-preserving text mismatch must fall back and keep both bodies.
The merge pipeline treats revision 0 as "never edited", so the field fails
open to the position path wherever it is not carried over. Make the column
explicit for direct row scans and document the remaining JSON gap.
The heartbeat loop and the detached callback handlers close the client's
current connection, but their goroutines can outlive the connection that
spawned them: heartbeatCancel only signals, and callback handlers run on a
context deliberately detached from the connection. A late-firing goroutine
therefore closed whatever connection the reconnect loop had just opened,
leaving the client to drop messages until the next read timeout.
Pass the owning connection into both callers and close through closeConnIf,
which only acts while that connection is still active. This matches the
yunzhijia and qqbot drivers, which already scope their heartbeat loops to a
single connection.
Also fold Stop() into closeConn so it no longer holds c.mu across Close.
Stop() was the one caller where the delay actually spread: the IM service
tears channels down while holding its channel-map lock, so a TLS close_notify
write to a stalled peer stalled every other channel operation.
The receive loop's deferred cleanup now closes only through closeConnIf,
dropping a redundant second Close of the same connection, and connectAndRun
re-checks the closed flag after publishing the connection so a Stop() racing
the dial cannot leave a receive loop running until its read deadline expires.
Chunk editing made parser coordinates unreliable, so all merges were
downgraded to pure text matching, including unedited documents.
Restore position-aware merging for pairs whose coordinates are still
trustworthy, and keep the text fallback for edited or stale content so
it is never dropped on coordinates.
- Feature contracts: trusted pairs trim overlaps, join adjacent chunks
seamlessly, and stay separate across position gaps.
- Safety contracts: never drop current content on coordinates alone.
* feat(datasource): add Feishu/Lark Drive (云盘) connector
Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.
Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
/ feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
(FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
the wiki connector's three cursor semantics (resume fast-path,
failure-does-not-advance-cursor, toSyncCursor snapshot), plus
ListResources / ResolveResourceAncestors (top-down shared traversal,
best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata
Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".
Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
Drive root folder_token input + load button before the lazy-load tree,
edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
input group
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): refine Feishu Drive connector UI, channel and root folder
A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:
Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
"飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.
Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
switch); the tree area shows a "load a folder first" placeholder until
the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
actionable i18n hints.
Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
to resolve the root folder's human-readable name; the list API only
returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
suffix) so it matches the saved resource_id form and the direct
children's ParentID, which previously broke selection matching on edit.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): sync a single selected Drive file without folder error
When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".
The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.
A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.
Co-Authored-By: Claude <noreply@anthropic.com>
* ci: add build-test-images workflow for tar.gz artifact export
Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.
- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days
Co-Authored-By: Claude <noreply@anthropic.com>
* ci: add build-test-images workflow for tar.gz artifact export
Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.
- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days
Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu shared-blocks extraction design spec
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu shared-blocks implementation plan
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): sync feishu drive docx via blocks API with export fallback
- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
sub-items), falling back to the export API on blocks errors or empty
renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): label feishu drive knowledge as 飞书云盘 in list
The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): polish feishu drive resource-step UI in datasource editor
- Replace the two stacked t-alert banners under the folder_token input:
shareHint becomes an always-visible form-desc line, rootNotSupportedHint
moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
status + tips on the input (replacing the global MessagePlugin) when
empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
fixed and the resource region (placeholder / loading / empty / tree)
fills the remaining drawer height, with the tree scrolling internally.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu drive datasource integration guide
User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine
Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.
Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.
Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): split feishu connector into core/wiki/drive packages
Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.
container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).
Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).
Co-Authored-By: Claude <noreply@anthropic.com>
* test(datasource): migrate feishu tests to core/wiki/drive packages
Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.
All tests pass: core 4s, wiki 13s, drive 3s.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test
Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).
Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.
All tests pass: core 3.5s, wiki 13s, drive 3s.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): repair feishu cleanup fallout
Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.
All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(frontend): feishu drive data source UI and i18n
Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: add trellis journal merge=union rule
Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): rename feishu/drive drive_connector.go to connector.go
Aligns naming with feishu/wiki subpackage (connector.go).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): allow data:image base64 URIs in sanitized markdown
DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path
The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export
Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.
- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
they don't break under the new default
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout
- 飞书云盘接入说明: add section on docx parse mode (export default vs
blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
add pros/cons table, fix source-file table to core/wiki/drive layout
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: 修复合并main后丢失新增的国际化问题
* fix(i18n): 修复误删的国际化内容
* docs: 删除误提交的文件
---------
Co-authored-by: Claude <noreply@anthropic.com>
List create_knowledge_from_file (previously missing from the README tool
list) and the new create_knowledge_from_text in README.md and EXAMPLES.md,
and add a CHANGELOG entry under Unreleased.
Add an MCP tool that creates a knowledge entry from raw Markdown text by
calling the existing backend endpoint
POST /knowledge-bases/{id}/knowledge/manual (handler.CreateManualKnowledge).
This completes the "text" half of #323, whose "file" half was already
covered by create_knowledge_from_file.
The tool defaults to status="publish" so the created entry is chunked,
embedded and made searchable immediately — matching the API/MCP use case
where there is no UI to publish a draft. Callers may pass status="draft"
to save without indexing.
- New client method WeKnoraClient.create_knowledge_from_text.
- New @mcp.tool() create_knowledge_from_text, resolving kb_id by name or
UUID (consistent with create_session / hybrid_search).
- Add focused unittests verifying endpoint, request body shape, status
default/override, and kb_id resolution wiring.
- Bump the stdio tools-list count assertion in test_mcp_transports.py
from 29 to 30 for the newly registered tool.
Restructure the page header and footer for clearer metadata, add field-level revision diffs with correct incremental comparison (including v1 initial content), and align footer links with in-body wiki link styling.
A caller that resends CreateFAQEntry before the first call answers could
create several rows for the same standard question. Indexing runs inline and
the embedding call retries with exponential backoff, so one create can take
far longer than an impatient caller waits; meanwhile the duplicate-question
check only looked at ChunkStatusIndexed, so a sibling create still mid-flight
was invisible and every retry passed the check.
- Serialize concurrent creates of the same question behind a Redis SetNX
guard, keyed by tenant + knowledge base + question hash, with an in-process
fallback when Redis is unavailable so an outage cannot block FAQ authoring.
- Widen the duplicate check to every non-deleted status. This also fixes
ChunkStatusDefault entries, which are listed to users but never blocked
duplicates.
- Bound the indexing step of an interactive create so a degraded embedding
service cannot stretch the request out; bulk and background indexing keep
the full retry budget.
- Report a failed rollback instead of discarding the error, since a leftover
stored chunk is hidden from the list yet now matches the duplicate check.
Centralize ingestion base splitter defaults in chunker.NormalizeSplitterConfig
so the preview endpoint applies the same normalization as upload. Regenerate
Swagger for parent-child preview fields and add regression tests for single-
level preview, default parent/child sizes, and context headers.
Wiki generation sometimes rendered "Write in ." because async workers
read empty locale payloads without falling back to context or defaults.
Unify language resolution helpers and apply them at enqueue and consume
time so every wiki and related async prompt gets a non-empty language.
Harden public URL mode: stop logging signed URLs at INFO, copy messages
before rewriting history responses, reuse winning grants on insert races,
sync the Go client resource_urls query param, and add knowledge-search
coverage.
Follow-up on resource_urls=public. Review turned up one authorization gap plus a
set of correctness problems in the streaming path.
Authorization:
- Embed channels were documented as excluded, but nothing enforced it. The
delegating handlers hand the visitor's own gin context to the shared
handlers, so ?resource_urls=public — and a deployment-wide
RESOURCE_URL_MODE=public — reached anonymous traffic and produced
shareable, credential-free URLs that bypass the channel-scoped
/embed/:channel_id/files proxy. ensureEmbedSession now pins the request
context to handle mode, which covers every delegated endpoint including
future ones. The downgrade is silent so a client that forwards the
parameter keeps working.
- A knowledge-base-restricted API key is refused with 403. Such a key is
already denied /files because a raw storage path cannot be bound to its
allow-list, so handing it anonymous file URLs would reopen that hole from
the other side. The default handle mode stays available to it.
Streaming:
- Rewriter.ref holds its lock across resolution. Resolver implementations keep
an unsynchronised per-provider cache, so the previous unlocked window was a
real data race for any caller sharing a StreamRewriter between goroutines —
which its own doc comment invited. Serialising also collapses a concurrent
duplicate into one signature.
- Release the holdback buffer on every path that ends a stream while the
client is still connected: a user-requested stop, an error event, and
giving up on the event store. Only completion flushed before, so the tail
generated just before a stop was silently dropped.
- A released tail carries the metadata of the event it was cut from instead of
a bare event_id.
- The maxHeldBytes safety valve aligns to a UTF-8 boundary rather than cutting
a character in half, and an unclosed "![…](" only holds back a
whitespace-free destination under 2048 bytes, so prose that merely mentions
Markdown image syntax no longer stalls the stream behind the buffer.
Also resolves the mode before running a knowledge search instead of after, so an
invalid value no longer costs a full retrieval, and warns once per distinct bad
RESOURCE_URL_MODE value instead of once per process.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
Resolving a resource:// handle into a public URL inserted one capability row
every time. Rendering an answer resolves the same image across many chunks and
re-reading a message history resolves it again on every call, so a read endpoint
behaved like a write-heavy one.
The table stores only sha256(token), so an existing grant cannot be reused by
looking it up — the plaintext token is unrecoverable. Derive the token instead:
HMAC-SHA256(SYSTEM_AES_KEY, resource id + time window). The same resource in the
same window derives the same token, so the row can be found by its hash and
reused; only the first request in a window inserts. The window is half the TTL,
so a reused grant always has at least TTL/2 of life left.
This keeps the storage properties intact: the table still holds only hashes and
the plaintext cannot be reconstructed from a dump without SYSTEM_AES_KEY, while
authorization stays entirely on the row, so expiry and revocation still decide
whether a token resolves. Without SYSTEM_AES_KEY, grants stay random and
per-request as before. A stable URL per window also lets client and CDN caches
hit.
Stop deleting revoked grants during cleanup. A revoked row has to survive until
its expiry, otherwise a token derived for the same resource and window would
re-create it and revive the access that was just revoked. grant.revoked_at has
no writer yet, so this changes nothing today, but the derived token makes the
tombstone load-bearing.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
In public resource URL mode, building the payload consumes the chunk into the
holdback buffer, so returning between build and write would drop it. Reusing
emitStreamEvent also removes the duplicated flush-before-completion logic.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
Adds a cross-cutting "文件与图片引用" section to docs/api/README.md covering the
parameter, the deployment default, the endpoints it applies to, and the two
caveats that matter in practice: it needs APP_EXTERNAL_URL (or a publicly
reachable storage backend) to produce a link at all, and the links it produces
are time-limited but anonymously readable.
Also annotates the affected endpoints for Swagger and refreshes the generated
files for those parameters only, leaving unrelated drift in the committed
swagger output alone.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
Includes an end-to-end ContinueStream test that asserts the full SSE wire
output. It caught a leak: the references event carries its results twice, and
the copy in Data holds the typed []*SearchResult that CopyData did not
traverse, so handles reached a caller that asked for public URLs. CopyData now
handles the typed slice, []string and map[string]string forms.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
API responses reference stored files as opaque resource:// handles, so an
integrating app had to make a second authenticated call to /files for every
image before it could render anything.
Add an opt-in that resolves those references server-side into time-limited
HTTP(S) URLs, using the same mechanism the IM channels already rely on:
- per request: ?resource_urls=public (default: handle, unchanged)
- per deployment: RESOURCE_URL_MODE=public
Applied to the chat SSE endpoints (knowledge-chat, agent-chat,
continue-stream), message history load, and knowledge-search. Streamed answers
buffer a trailing incomplete reference so a handle split across two deltas is
still rewritten. References that cannot become an HTTP URL (for example local
storage with no APP_EXTERNAL_URL) stay handles, so clients keep the /files
fallback. Embed channels are deliberately excluded: their visitors are
anonymous.
SSE payloads share SearchResult pointers and metadata maps with the stream
replay buffer and the message being persisted, so those are rewritten as copies.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
wizardchen