mirror of
https://github.com/Tencent/WeKnora.git
synced 2026-10-28 03:29:30 +08:00
d46fa25e41a9cf65011a341ba1b6cbba7e9897ed
28
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 wizardchen
|
d46fa25e41 |
feat(plugin): dynamic form options, OAuth connections and resource limits
Plugin config forms can now ask the plugin while they are filled in:
- x-options {name, dependsOn, search} loads a field's choices from
/v1/options/{name}. POST /api/v1/plugins/:id/options merges the form's
values with the stored secrets (system, tenant or instance scope) so a
redacted token still works. Go and Python SDKs, OpenAPI and conformance
cover the endpoint.
- x-oauth turns a string field into an account connection. WeKnora runs
the authorization code flow (PKCE, client credentials from the plugin's
system config), keeps the tokens sealed in plugin_oauth_connections and
the field holds "oauth:<id>". The invoker swaps references for fresh
access tokens at call time, refreshing near expiry. Callback:
/api/v1/plugin-oauth/callback, one-time states in Redis or memory.
- SchemaForm gains an options widget (dependsOn reload, remote search)
and an OAuth connect widget; connector, web search, workspace and
platform config forms provide the plugin behind them.
Host plugins are capped per runtime.resources on Linux: RLIMIT_DATA for
memory (WEKNORA_PLUGIN_MEMORY_DEFAULT when undeclared) and, with a
delegated cgroup v2 (WEKNORA_PLUGIN_CGROUP), memory.max and cpu.max.
|
||
|
wizardchen
|
39ffd07597 |
feat(plugin): remote runtime
A plugin can now run as a service the administrator deploys elsewhere and registers by URL (runtime.type: remote). Installing asks for the URL, issues a 32-byte signing secret (stored sealed, shown once) and keeps both across upgrades. The admin API and the plugin detail drawer change the URL and rotate the secret. - internal/plugin/remote: an Activator that checks the service's health and manifest against the package, signs every call with the secret, health-checks it and reports outages to the reconciler; after an outage it checks the manifest again. - Calls route to the host or remote runtime by where the plugin loaded. Remote plugins get Host API access only when WEKNORA_PLUGIN_HOST_API_URL names an address they can reach. - The reconciler reloads a plugin when its URL or secret changes, and now retries a failed activation with backoff (30s doubling to 10m) instead of leaving it failed until someone changes it. That also covers host plugins that failed to start. A plugin that failed is unloaded when disabled. - SavePlugin updates the new columns; a test keeps the update column list in step with the table. - Migrations: plugins.remote_url, plugins.remote_secret. |
||
|
wizardchen
|
5fa64e200a |
feat(plugin): Host API with plugin tokens and a key-value store
Plugins can call back into WeKnora. Each call to a plugin that asked for Host API scopes (permissions.hostApi, accepted at install) carries the Host API URL and a five-minute token bound to the plugin, its version, the calling tenant and those scopes. Tokens are HS256 JWTs with their own audience, keyed from the cluster secrets (SYSTEM_AES_KEY, else JWT_SECRET), and are checked by /api/v1/plugin-host/* itself, outside user and API-key auth. The first scope is kv: a key-value store partitioned by plugin and tenant (plugin_kv, migrations 000117 / sqlite 000036), so plugins keep state without a database of their own. Get, put with an optional TTL, delete and paged prefix listing; keys up to 256 bytes, JSON values up to 64 KB, 10,000 keys per plugin and tenant; expired entries read as missing and are swept every ten minutes. The SDK exposes it as call.Host().KVGet / KVPut / KVDelete / KVList. Host plugins reach this node on loopback, which their environment now excludes from the egress proxy. |
||
|
wizardchen
|
750856cd8b |
feat(plugin): install declarative plugin packages across nodes
System admins can now install plugins from .wkp packages (zip with plugin.yaml): inspect, install or upgrade pinned to the reviewed digest, switch on or off platform-wide, roll back to a stored version, uninstall. Endpoints live under /system/admin/plugins. Installing only writes the plugins / plugin_versions rows and stores the package in the deployment's object storage. Every node runs a reconciler that loads the active version of each enabled plugin into the registry and hands it to domain activators; a Redis broadcast wakes peers at once and a 30s pass catches anything missed. Nodes report per-plugin status to Redis, so the plugin detail shows every node. Without Redis it is a single node. Manifests gain the skills and mcpServers points and a declarative runtime for modelVendors/skills/mcpServers, plus engines.weknora range checks. Installed plugins start disabled in every workspace; tenant admins opt in. Only declarative runtimes are accepted until the plugin host lands. |
||
|
wizardchen
|
abdbcbdb0e |
feat(plugin): let workspaces enable and disable plugins
Add plugin_tenant_settings (migration 000115 / sqlite 000034) and internal/plugin/tenancy, which stores each workspace's plugin switches and answers whether a contribution is enabled. A plugin without a row is enabled; required builtins (agent tools, the default parser) cannot be turned off. A disabled plugin's contributions leave /datasource/types, /web-search-providers/types, /im-channels/platforms and /models/providers, and creating a data source, web search provider, IM channel or model on them returns 400. Existing instances keep working. Unreadable switches fail open. GET /api/v1/plugins and /plugins/contributions now carry the workspace's switch, and PUT /api/v1/plugins/:id/enabled (Admin+) changes it. The settings modal gains a Plugins page, registered through the settings registry, that lists every plugin with its contributions and a switch. |
||
|
lyingbug
|
1bf717a54d |
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage Clicking a citation used to show only the parsed chunk. It now opens the original file in the references panel, jumps to the cited place and highlights it, narrowed to the sentence the citation supports. Parsing records where each chunk sits in the original file (chunks.source_locators): PDF page and paragraph boxes from the docreader, MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and EPUB chapters aligned in Go against any parser engine's Markdown; text offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites are followed with a line-diff remapper. Scanned PDF pages get their OCR text as quotes at search time so a sentence can pick its page. The frontend renders PDFs with pdf.js and every other previewable type with the existing preview, then highlights with the CSS Custom Highlight API in a highlighter yellow that stays visible against the brand green. * fix(docreader): serialize source blocks outside main so tests import it test_docx_tables stubs docreader.parser in sys.modules, so importing docreader.main from a later test failed in the full unittest discover run. The wire conversion now lives in docreader.source_wire, which needs only the proto. * fix(chat): tighten source locators from review - Remap rewritten-line-for-line stretches by line so scanned PDF pages no longer drift onto the wrong page. - Keep the source preview mounted for citations in the file already open. - Wait for in-flight PDF page renders before looking for the text layer. - Count image rune offsets in one pass; stream xlsx rows up to maxUnits. - Start text locators after a UTF-8 BOM, as viewers decode without it. - Only re-query image children for scanned-page results. - Seek audio citations without autoplay and drop stale seek listeners. - Fix the SQLite migration's cross-reference to versioned/000113. * fix(migrations): renumber chunk source locators after main's chunk_images main took versioned 000113 and sqlite 000032 for chunk_images, so the source_locators column moves to versioned 000114 and sqlite 000033. |
||
|
|
68fe9599cd |
feat(gallery): add an image gallery tab with schema-driven filters (#3694)
* feat(image): observe image attributes and let the policy decide OCR Describe each image with the VLM as before, but ask it to also observe two attributes about the image in the same round — how much body text it carries (contain.text) and whether it presents data visually (contain.data_visual) — then let a small pure function decide whether that image is worth a second, separate OCR round. Per-class OCR was the first shape of this idea; a class (chart / photo / logo) is only an indirect proxy for 'is there text worth reading here', and the class table has to grow whenever a new kind of image shows up. The decision actually needs the observed attributes, so it reads them directly, and the attribute set is a backend registry row rather than a new class. The observation and the caption share one VLM call; OCR stays a separate call (prompt isolation, see #3291). An attribute the model does not answer for is left absent rather than filled with a conservative default, so the trace never claims an observation that did not happen, and an incomplete observation falls back to the on_unobserved policy (OCR still runs). The registry also carries the panel's display text, so a new attribute needs no frontend change. Off by default: image_attrs_enabled=false keeps today's behaviour (one caption and one OCR per image). KB config: image_processing_config.image_attrs_enabled / image_actions; per-upload override via process_config. Registry: GET /api/v1/image-attrs/schema. * feat(gallery): add image gallery tab with schema-driven filters Adds a 4th knowledge-base tab parallel to Documents/Wiki/Graph that browses every image asset in a KB. Backend: - GET /api/v1/knowledge-bases/:id/images lists image assets projected from chunks.image_info, with keyword (caption/OCR), multi-value attribute (attr_<name>=<value>), is_enabled and sort filters applied in memory, plus pagination. - Resolves the source knowledge item title (SourceName) via a batched GetKnowledgeBatch so the viewer can show a human-readable source. Frontend: - ImageGallery.vue: thumbnail grid + search/sort + schema-driven filter panel (from GET /image-attrs/schema) + single-image viewer with prev/next, attributes/caption/OCR/source. - Wired as the gallery tab in KnowledgeBase.vue; i18n keys in all 5 locales. * feat(gallery): drive filters/search/sort from a configurable attribute contract The gallery UI is no longer hardcoded: it renders from a self-describing contract served by GET /knowledge-bases/:id/gallery-config (live attribute sources, every attribute's definition with usage flags merged across the config tiers, and the caller's search activation state). - Attribute sources form an open registry (types.RegisterGalleryAttrSource); a builtin source models caption/ocr_text/timestamps/enabled so the gallery has zero special cases. - Usage resolution: user > KB > system > source, field by field; the system tier lives in system_settings (gallery.policy, JSON, no rebuild), the KB tier slot is wired in the engine but not stored yet. - Search activation is the user's own record: mode (all|custom) plus per-field on/off in users.preferences.gallery; source declarations never set defaults (unrecorded = off in custom mode). - image_info is read through a gallery-local struct, removing the compile dependency on the attribute-pipeline types; the gallery builds and runs with or without that feature deployed. - ListImages validates search_in / sort_by / attr_filters against the contract; ineligible fields are dropped, not served. * fix(frontend): inline the shared graph-drawer stylesheet (@import instead of <style src>) plugin-vue 6.0.6 keys unscoped src-style descriptors by the imported file path, so AgentStreamDisplay.vue and WikiBrowser.vue — both importing components/css/wiki-graph-drawer.less — overwrite each other during the build and the second one crashes with "reading 'scoped'". Adding the gallery changed the module order enough to make the collision hit every time. Inlining the same stylesheet keeps the emitted CSS identical and removes the shared descriptor key. * feat(gallery): render attributes from short labels with descriptions in tooltips A source could only say one thing about a value, so the filter panel had to put a full sentence inside a checkbox. Attributes and their values now carry two pieces of text: a short label for the space-constrained controls (checkbox, search field, sort option) and a description for wherever there is room (tooltip, detail panel). The contract's value list becomes an array of {value, label, description}, and the display helpers resolve each field separately: the gallery stamps the short name on a control and moves the sentence into a tooltip. Translation overlays are looked up per field, so a source that ships no gallery-specific text still picks up the attribute pipeline's own wording. * fix(gallery): let locale text win over the contract's wording The filter panel read the backend's own label before asking the locale, so a translated value never got a chance: the contract ships its own default-language wording, and it shadowed the overlay for every locale. Built keys were also off by a segment — imageAttr.<name> has to be imageAttr.<name>.label, and the gallery's own keys live under knowledgeEditor.wikiBrowser.gallery.attr — so builtin and system attributes fell through to English on a Chinese UI. * feat(gallery): top bar layout, pinned panels and per-value verdicts The toolbar moves onto its own row above the grid, so the controls no longer eat into the space the images are there for: search box, the two scope switches, sort field and direction, and the result count all sit in one flex row. The settings behind them are no longer inline either — each opens as a popover under its control and can be pinned to a right-hand rail, where the two panels take turns as tabs instead of both claiming width at once. Filtering gains a third state per attribute value. Where a value could only be picked or not picked, it can now be "off" (hide those images), "on" (show them whatever else says) or neutral (leave them alone), which is what lets one attribute's rules talk about an image another attribute tried to hide without the pair of them cancelling each other out: a forced display outranks a forced hide, and anything no rule speaks about stays visible. The switch that used to only exist inside the panel now has a toolbar home as well. "筛选：全显示" genuinely drops the attribute constraint instead of imposing it against a panel the user cannot see. Filtering used to empty the list the moment it was switched on. Any image whose attribute was never observed — every image ingested before the attribute pipeline existed — read as a mismatch for every filter, which is not narrowing a list but losing it. Absence now stays silence: an attribute no image carries places no constraint on it, until the pipeline catches up and fills the value in. Co-authored-by: WorkBuddy * feat(gallery): expose the image attribute pipeline as a gallery source The gallery's contract resolved to a single source, so the filter panel could only offer "Enabled" while the attributes the observe-and-describe pipeline fills in — contain.text, contain.data_visual — never reached the UI. The registry is a gallery attribute source like any other: every attribute it declares is offered to the gallery, filterable by default, with the config tiers able to override the usage flags at runtime. The registration lives in the attribute pipeline's own commit; this branch carried an earlier, squapshot of that work which predated the split of an attribute value into a short label and a description, so the registry's value shape and this file have been brought in line with it. Both PRs now hold byte-identical copies of the source hook, which is what lets either one land first and the other merge cleanly afterwards. Co-authored-by: WorkBuddy * fix(gallery): viewer race, pin icon, panel anchoring and scope wording Opening the viewer rendered the img element with the previous or empty source while the proxy fetch was still producing a blob, and the error that empty source fired locked the viewer in its failure state even after the real image arrived. The resolved source now lands before the flag is consulted, and a token keeps a slow fetch of one image from overwriting a faster one that navigated past it. The pin button drew nothing: TDesign has no "pushpin" glyph, so the button rendered as an empty shell. It now uses pin / pin-filled, keeps its place beside the tab bar, and a pinned rail assigns its columns explicitly — the panel precedes the main section in the DOM, so leaving the grid to source order handed the wide column to the panel and squeezed the images into the rail. The floating panel no longer appears at a hardcoded offset: it anchors just below the arrow that summoned it, the arrow flips up while its panel is out, a second press retracts, and any click elsewhere in the toolbar closes it. When a rail is pinned the arrows turn to the requested tab instead of spawning a second surface. The scope words (全部 / 全显示 / 自定义) become plain status text rather than a second dropdown — they name the current mode, and only the arrow beside them opens the panel that changes it. Co-authored-by: WorkBuddy * fix(web): align the OCR-on-unobserved switch with the other setting rows The switch sat bare inside its setting-row while every other switch on the page is wrapped in setting-control, which is what reserves the right column and centers the control vertically. With the row laid out from the top, the bare switch hugged the row's upper edge and crowded the description text — visibly out of place next to its neighbors. Wrapping it in setting-control gives it the same right-aligned, centered slot as the rest. Co-authored-by: WorkBuddy * fix(web): move the OCR-on-unobserved row out of the vertical schema row The row lived inside setting-row-vertical, whose descendant rule stretches every setting-info and setting-control beneath it to 100% width. Two full-width flex children side by side cannot fit, so the control overflowed the container and the switch landed past the right edge — visibly further out than the switch on the row above. The row and its note now sit beside the vertical block instead of inside it, under their own condition, so the switch shares the ordinary row layout and its right edge lines up with the "image attribute observation" switch above. Co-authored-by: WorkBuddy * fix(gallery): applying a verdict now activates the filter scope Setting an off/on verdict while the panel's master switch was off silently ignored it: buildParams only attaches attr_rules in custom scope, and setVerdict never flipped the scope. A verdict the user just expressed now switches the scope to custom; clearing the last meaningful verdict returns to all, since custom with no rules constrains nothing. * fix(gallery): search checkboxes stay visible and idle panels read as dimmed The search panel hid its field checkboxes while the master switch was on, so the only way to reach them was to flip the switch first. They now sit next to the switch like the filter panel's verdicts do: dimmed in all-field mode (where they carry no weight) yet clickable, and touching one drops the mode to custom. The filter panel's verdict rows dim the same way while the filter switch is off. * fix(gallery): bulk verdict row, neutral default mark, panel polish The clear-filters button never matched what the panel is for, so it is now a bulk verdict row: one set of -/off/on buttons that applies the chosen position to every value of every value-typed attribute, with the default mark turned into a language-neutral dash. The panel head drops its second divider (the tab bar already draws one) and the tabs follow the toolbar's order: search left, filter right. * fix(gallery): drop the all-fields caption and center the order arrow The caption above the search checkboxes showed and hid with the master switch, shoving the checkboxes down on every flip; it said nothing the dimmed state doesn't, so it is gone. The order button's arrow is an inline svg, which sits on the text baseline and reads as riding high; it now centers against the label. * fix(gallery): center the order arrow for real vertical-align never applied: the icon is a flex item inside the button's inline-flex text wrapper, and the wrapper's default stretch pins the explicitly-sized 16px icon to the top of the 22px line box. align-items: center on the wrapper is what actually centers it. * fix(gallery): shorten panel tab labels in en and ru The two-word labels overflowed the 300px panel and forced the tab bar into scroll arrows; single-word labels fit in every locale that was affected. * docs: drop the image pipeline rewrite from this branch The rewritten 'Go 侧接手后' paragraph belongs to the image attribute observation work (#3635), which this branch carries as a prerequisite and which already documents the caption_ocr / observation_driven split. Keeping a competing rewrite of the same paragraph on the gallery branch makes it collide with upstream, which extended that paragraph with the custom_instructions and inline HTML table notes. Restoring the shared wording keeps the gallery branch free of this file so upstream's edits merge cleanly. * style: gofmt the gallery attribute types Struct field tags in the resolved gallery config were one column narrow, so gofmt rebroke the alignment block. Whitespace only. * style(gallery): use design tokens in the gallery styles The style guard ratchet fails on any new violation. ImageGallery.vue shipped with 47 TDesign token fallbacks, 21 literal font sizes, 4 literal radii and a 0.15s transition; all of them now read the --app-* / --td-* tokens, which is what the guard asks for and what dark mode depends on. * lint(gallery): fix the golangci-lint findings in the gallery code lll: the three over-long swagger @Param lines and one registry row were shortened. ineffassign: ctx was assigned before the access check and then reassigned, so the first value was dead; it is now declared once, after the check. errcheck: the three unchecked c.Error calls in ListImages. revive: the AttrSpec / AttrValue doc comment block was attached to the wrong type and split into one block each. staticcheck S1016: the gallery registration converts the registry value instead of restating its fields. * lint(gallery): wrap the image query select and check the last c.Error * style: reformat the wrapped image select in chunk.go * fix(gallery): address review findings on ordering, KB switch and blob lifetime - sortImageAssets: swap operands for desc instead of negating, keeping a strict weak ordering so ties page stably - ImageGallery: reload on knowledgeBaseId change, drop stale list responses, fetch thumbnails in parallel with one cached blob per URL, revoke blobs on unmount, default sort to builtin:created_at - KnowledgeBase: non-wiki KBs with a stale wiki/graph tab fall back to documents again - handler: move ListImages godoc out of ListKnowledgeFolders' block and document attr_filters / attr_rules - DecideOCR: a payload with no OCR conditions uses the built-in policy instead of silently skipping OCR * docs(parsing): describe the caption_ocr and observation_driven image pipelines Carried over from #3635, which this PR supersedes. * fix(gallery): load local:// and other storage handles through the KB file proxy The gallery's own scheme check only caught resource://, provider:// and storage://, so local:// (the default storage), minio://, cos:// and s3:// images went straight into <img src> and never rendered. Resolve through isProviderFileURL, the scheme list chat and wiki already use, and lock the routing with a test. * fix(kb-editor): keep API-customised OCR conditions on save and upload The KB editor always wrote the registry default into image_actions.ocr.on, so any unrelated edit replaced a knowledge base's custom OCR conditions with the default table; the upload dialog did the same for the per-document override. Keep a stored custom list and fall back to the registry default only when there is none, and show the conditions actually in effect instead of the default table in both panels. * perf(gallery): de-duplicate, filter, sort and page images in SQL Every gallery request used to read every image-bearing chunk of the KB, OCR text included, decode it in Go and page in memory. A single query now expands the image_info arrays, keeps one row per image URL, applies the keyword / attribute / verdict / enabled filters and the sort, and returns only the page with the filtered total. The expansion carries narrow scalars only; filters and JSON sort keys are evaluated on the winning entry of each image, and the page's JSON is read back for its rows alone. Postgres runs the query with JIT off, since the fixed 100-rows-per-call estimate of jsonb_array_elements otherwise prices it ~100x high and compilation dominates. On 20k images with 2 KB of OCR text each (40k chunks), a page drops from ~750 ms to ~100-200 ms on postgres and ~200 ms on sqlite; keyword search over OCR text stays a full scan (~400-600 ms) until the text is indexed. The filter semantics are pinned against both databases. * perf(gallery): serve the gallery from a trigger-maintained chunk_images table Reading images out of chunks.image_info meant parsing every image chunk of the knowledge base on each request, so a page still cost time in proportion to the KB. chunk_images (versioned/000113, sqlite/000032) keeps one row per image_info entry with caption, OCR text and attributes as columns; triggers on chunks keep it in step with every write path, and the migration backfills existing chunks. Copies of one image stay separate rows and the query lists the copy on the most recently updated chunk through an indexed anti-join, so writes decide nothing and concurrent writers need no locking. On postgres, caption and OCR text get pg_trgm indexes when the extension is available. On 20k images with 2 KB of OCR text each (40k chunks): postgres first page 10 ms, page 500 41 ms, keyword 2 ms, filters 23 ms sqlite first page 3 ms, page 500 39 ms, keyword 147 ms, filters 51 ms down from ~100-600 ms. Writing those 40k image chunks costs 7.0 s instead of 1.3 s on postgres (1.9 s instead of 1.0 s on sqlite), mostly the trigram indexes; status and enable-flag updates patch rows in place. --------- Co-authored-by: tayadwx <tayadwx@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com> |
||
|
lyingbug
|
967ed09706 |
feat(models): add system admin model catalog console (#3662)
* feat(models): add system admin model catalog console System administrators can now maintain the model catalog from Settings → System administration → Model catalog instead of editing models.json on every replica. Administrator changes are stored as a versioned overlay in the database (new model_catalog_config table), layered between the deployment file and explicit per-model settings, and synchronised to other instances within about 5 seconds. Backend - GET/POST preview/PUT /system/admin/model-catalog with optimistic version + deployment baseline checks (409 on conflict), 20-version history and audit logging. - Console overlays reject credentials, headers, env interpolation and non-inline icons; those stay deployment-owned. - Catalog views include each chat model's resolved thinking levels and the vendor defaults, so the console never re-implements level inheritance. Frontend - Model list in the shared data-table style with provider/type filters, "modified only" filter and source tags (built-in / deployment / admin). - Edit drawer saves and applies immediately: display name, context window, max output, input modalities, reasoning, thinking levels, embedding dimension and picker visibility, each with its inherited default and a layer comparison table. - Add-model drawer, JSON editor with server-side change check before publishing, and version history with one-step restore. - Model settings gets a system-admin shortcut to the catalog; model editor reloads providers when opened so published catalogs apply. * fix(models): harden the admin model catalog console - Reject console base_url, base_urls, url_patterns and auth: moving a vendor's default URL would send its deployment api_key (and keys of rows without their own base URL) to another host. With every interpolated field now deployment-owned, '$' in names and descriptions is plain text. - Drop deployment base URLs from catalog views; they may carry interpolated tokens. - Remember a stored version this replica cannot compile and skip it until the version changes, instead of re-reading and recompiling every poll. - Resolve vendor default thinking levels with reasoning forced on, like per-model levels. - Compute builtin/deployment views once; preview returns only the candidate catalog and normalized overlay. - Share deployment file lookup via runtime.ReadDeploymentOverlay. - The model editor force-refreshes its own type instead of resetting the provider store for every type; the refreshed list wins in the id index. |
||
|
jjj-n
|
b86821f551 | feat(im): add per-channel reply locale (#3455) | ||
|
nullptr-errorandhailongzhao
|
22cbcdfa57 |
feat(sandbox): run unpinned Lite sessions on the host OS sandbox (#3549)
* feat(sandbox): resolve workspace paths through WorkspaceLayout Keep remote sessions on the /workspace contract, while file, shell, and artifact code follow the session layout instead of hardcoded paths. * fix(sandbox): fail closed on host layout lookup and restore the shell blacklist Treat host vs remote as an explicit origin, refuse inspect outside ReadRoots, and skip collection when the output tree is missing or equals the project root. * fix(sandbox): close leftover fail-open holes in the layout path Keep host copy, collection, and prompt text from falling back to /workspace when a lookup fails or an adapter returns unclean roots. * fix(sandbox): satisfy golangci-lint on the layout helpers Wrap new copy and test lines to 120 columns, and add the exported comments revive requires. * fix(sandbox): gofmt the read_file layout copy wrap * feat(sandbox): add host as an OS-enforced backend type Host is not a named/pinnable config: each command is a fresh local process, unlike Docker, Cube, and E2B. * feat(session): persist Lite host workspace directory Empty means an auto-allocated folder. Standard edition leaves the column unused so the shared session model still loads. SQLite also gains the tenant skill tables Lite shell_exec already expects. * feat(localsandbox): adapt the host OS sandbox to session interfaces File tools stay outside Seatbelt and are bounded by PathGuard. A revoked project directory fails closed instead of allocating a replacement workspace. * feat(sandbox): run unpinned Lite sessions on the host backend Lite opts into host explicitly; web stays disabled when no named remote config exists. Host workspaces are not destroyed or auto-committed. * feat(desktop): pick and persist host project directories The native folder dialog is the authorization path; typed paths are rejected. settings.sandbox.host tells the client when this edition can use it. * fix(sandbox): fail closed on Lite host layout lookup misses Unpinned host sessions were described as /workspace when pin read or sandbox resolve failed. Prompts and attachment staging now share the same fallback so Lite never pretends the remote tree exists. * fix(frontend): keep settings.sandbox.host in sync with the backend keys TestDeploymentCapabilityKeysMatchFrontend requires the lists to match. Host stays fail-closed like docker so a missed probe does not show Lite-only UI. * fix(lint): satisfy golangci-lint on the host sandbox wiring Check c.Error, wrap lines over 120 columns, put context first on versionsWorkspace, and document exported adapter methods. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> |
||
|
Leo Dawnandhailongzhao
|
56d075e213 |
fix(sandbox): resolve a session's pinned sandbox in the workspace that owns it (#3471)
* fix(sandbox): resolve a session's pinned sandbox in the workspace that owns it
tenant_sandbox_configs is keyed by (tenant_id, id), so a config id is not an
address on its own. sessions.sandbox_config_id stored only the id and every
reader took the workspace from the ambient request context.
That holds inside a chat turn, which already runs under the agent's workspace
(WithExecutionTenant in the QA handler). It breaks for a shared agent
everywhere else, because a shared agent runs on ITS OWNER's sandbox config
while the session belongs to the borrower, and these paths run from plain
requests whose only tenant is the session owner's:
* DeleteSession resolved nothing and returned after a warning. Cube and E2B
create session sandboxes with onTimeout=pause, so the MicroVM was not
reclaimed later either: it stayed paused, billing, with nobody holding its
id. ReapOrphanSandboxes has no production caller, so nothing recovered it.
* The terminal and desktop panels failed on both branches - attaching to the
pinned sandbox and provisioning a new one - with ErrSandboxConfigNotFound,
which terminalErrorFrame maps to a bare INTERNAL close reason. The panel is
offered unconditionally and already forwards agent_source_tenant_id, so
this was a dead end the user could reach in one click.
* Fork read the pin through PinnedSessionSandbox, got no manager, and
silently degraded to SNAPSHOT_UNSUPPORTED, losing the sandbox state.
* The fork snapshot lease recorded the session's workspace next to the
lending workspace's config id, so the reaper could not resolve it either.
Record the owning workspace beside the pin and make the pair travel together
as SandboxPin. Readers now take the workspace from the pin and fall back to the
session's own, which is what every non-borrowed pin (and every row written
before this column) resolves to - so behaviour for own agents is unchanged.
The provision path has no pin yet, so the WebSocket handler passes the agent's
workspace along with its config: terminalProvisionPin already resolved the
shared agent, it just discarded the tenant resolveAgent handed back.
Backfill in the migration is limited to sessions whose pinned config does not
exist in their own workspace - the only ones that can have borrowed it - using
the lending workspace recorded on their newest shared-agent message. Lite has
no shared spaces, so the SQLite mirror adds the column without a backfill.
* fix(sandbox): backfill the pin owner from the config row and self-heal legacy pins
Three gaps in the session sandbox pin owner:
Backfilling from messages.agent_tenant_id names the wrong workspace. The
pin is sticky to the session's FIRST sandbox, while the newest message
carrying agent_tenant_id may belong to a later shared agent from another
workspace. Writing that tenant leaves teardown resolving a config the
workspace does not own - the very failure the column exists to fix.
tenant_sandbox_configs.id is a global UUID primary key, so joining the
config row itself names the workspace that actually owns the pin.
The column was INTEGER while every other tenant id is BIGINT/uint64.
Postgres INTEGER is 32-bit signed, so a large tenant id would not fit.
Rows the SQL backfill cannot reach keep sandbox_config_tenant_id = 0
forever: the reader falls back to the caller's workspace but never
records what it learned. The next chat turn already runs under the
workspace that can resolve the config, so persisting it there is what
lets a later DELETE, panel open or fork find the sandbox. recordOwner
only writes when the lookup actually succeeded and the stored owner is
still unset - stamping the caller's tenant onto a failed lookup would
brand a sticky pin left by a previous shared agent with the wrong
workspace.
Also renames the lentSandboxConfg typo and corrects the fork degrade
reason in a comment: a lent config that resolves to nil fails
BoundSandboxID, so the fork degrades to SANDBOX_GONE, not
SNAPSHOT_UNSUPPORTED.
* fix(sandbox): require a named backend before recording a pin's owner
The self-heal wants to persist an owner "that actually resolved this config",
but a nil error does not prove that. resolveTenantSandboxForConfig returns a
disabled manager with no error twice before it ever reads the config row: for
the workspace kill switch, and for an empty or deployment-default config id.
That reopens the leak this column exists to close. The SQL backfill correctly
leaves sandbox_config_tenant_id at 0 whenever the pinned config belongs to the
session's own workspace, which is most existing rows. One turn of a shared
agent lent by a workspace whose admin has script execution turned off then
short-circuits on the kill switch, returns nil, and stamps that lending
workspace onto a config it does not own. The next DELETE resolves the pin there,
finds nothing, and strands the paused MicroVM.
The pin branch runs before agentConfigID is read, so the shared agent does not
even need a sandbox config of its own to trigger it.
A named backend (Cube/E2B/Docker) is the proof, and it is the same condition
that is already allowed to CREATE a pin further down: what may write a pin may
update whose it is. A config that exists but resolves to type "disabled" is
skipped as well - ownership is genuine there, but such a config never creates a
sandbox, so no teardown can go missing.
The deployment-default pin was inert either way (both workspaces short-circuit
to a manager with no DestroySession), but it is covered too so a later
relaxation of the proof cannot quietly reopen the kill-switch case.
Both tests fail without the guard.
---------
Co-authored-by: hailongzhao <hailongzhao@tencent.com>
|
||
|
lyingbug
|
9e0c28fbc9 |
feat(artifacts): let users delete generated files (#3447)
* feat(artifacts): let users delete generated files
技能生成的文件此前只能列出、预览和下载，没有任何删除入口，后端
handler/service/repository 三层都没有实现。DeleteSession 里那句注释
("a dedicated GC job or explicit hard-delete API is the right place to
reclaim storage") 留的就是这个位置。
Tombstone, not a row delete. message_artifacts 新增 deleted_at，删除只
打标记不删行。position 是下载接口寻址用的下标（msg.Artifacts[index]），
物理删掉中间一行会让它后面的文件整体前移，已有链接下到错的文件；而
ArtifactCollector 的去重集就是从这张表读的，行没了之后沙箱里 mtime 未变
的同名文件下一轮就会被重新采集回来，删了又出现。墓碑同时挡住这两条。
loadArtifactsByMessage 因此刻意保留墓碑，展示侧用 MessageArtifacts.Live
过滤；NewMessageArtifactRecords 带上 DeletedAt，任何一次消息重写都能把
墓碑原样写回去。
Bytes. 字节是真回收的，走 catalog.Release 引用计数：一份文件可能同时被
知识库条目和后续回答持有，只有所有持有者都放手才 DeleteFile。删多个版本
时同一个对象可能来自多条消息，每条消息的绑定都要释放，字节只删一次。
回收发生在打标记之后，且只针对本次调用真正认领到的行——并发的第二次删除
认领不到任何行，也就不会去动别人正在回收的对象。释放失败一律保留字节：
留个孤儿 blob 以后还能靠墓碑上的 url 回收，删错了则是别人的文档里的图片
凭空消失。
Scope. 会话内面板每个版本单独一行，删的就是点的那一版；产物库一行代表
一个文件（带版本数），默认连同该文件在这个会话里的所有版本一起删，否则
删完这一行还在、只是显示上一版。权限比下载严一档：下载认共享智能体的
只读访问，删除只认会话归属人。
Inline references. 正文里的 resource:// 引用渲染成置灰加删除线的卡片，
不可点击，和「模型引用了不存在的文件」的 --pending 态区分开。图片类产物
也降级成卡片：走 renderImage 只会拉取失败，留下一张永远加载不出来的图。
墓碑必须留在传给渲染器的产物列表里——句柄形式与知识库图片同形，过滤掉
之后句柄对不上本条回答，会被当成外部受保护图片去拉一个已经不存在的
blob，结果是裂图。组件里因此拆出 liveArtifacts 只给计数和文件列表用。
Also: 产物库是 offset 分页的无限列表，删一行会让后续页整体前移漏掉一条，
改成按已显示行数取下一页并按 key 去重；access.ResolveMessageArtifact 与
MessageReferencesFile 不再让墓碑授权访问；SessionArtifacts 过滤墓碑，
避免新回答把引用绑到已删文件上；版本澄清文案不再把已删版本算进版本数。
docs/swagger.* 未重新生成：仓库里那份已经落后若干功能，重生成会带进约
1000 行无关漂移，另开 PR 同步。
* fix(artifacts): close the delete holes found in review
Regeneration. loadKnownSet 把墓碑的 ContentHash 也放进了去重集，于是
knownSameContent 会把「删除后技能重新生成、字节恰好相同」当成 fork
checkout 那种 restore：不上传、不挂回本轮回答，文件再也回不来。墓碑现在
只登记 (path, mtime)，不进 hash 也不进 byPath —— 前者才是「沙箱文件没动
过就别复活」需要的键，后者是「同内容恢复」的短路，两件事不该共用。
RecordRestoredArtifactMtime 也不再改墓碑：它的 mtime 就是那把锁。
Message file proxy. /sessions/:id/messages/:message_id/files 走
MessageReferencesFile，而正文检查在最前面且无条件为真。删除不会重写回答，
resource:// 句柄还留在正文里，于是这条代理会继续把已删文件交出去——包括
交给共享智能体的只读访客，也包括「字节因为知识库还持有而没被回收」的情形。
现在先看 artifacts：命中本条消息的墓碑直接 false，命中 live 直接 true，
其余（知识库图、附件）才回落到正文。同一文件挂在多个位置时只要还有一个
live 就仍然可取。
Cross-session bytes. CreateForked 复制产物行时连同存储 URL 一起复制，但
不会给副本重新 Bind，所以 catalog 的引用计数看不到分叉。在父会话里删除会
把分叉还在用的字节删掉。无 catalog 部署、以及存的是裸 provider 路径的行，
同样不在计数覆盖内。回收前新增一道 CountLiveArtifactsByURL：只要还有任何
未删除的行指向同一对象就保留字节，查不出来也保留。配套加了 url 的部分索引。
Atomicity. SoftDeleteSessionArtifacts 原来逐条 UPDATE 不带事务，删多版本
时中途失败会留下「一部分已隐藏、字节没回收、接口 500、用户重试 404」。
现在整批放进一个事务。writeMessageArtifacts 是整表替换，拿着删除前快照的
并发写会让墓碑复活而字节已在回收，现在按 position 把已存在的 deleted_at
合并回去——删除是唯一不允许被过期写覆盖的属性。
all_versions 两个路由改用同一个解析器（strconv.ParseBool），此前一个是
== "true"、另一个是 != "false"，显式传 1/TRUE 在两边语义不同；缺省值仍
按路由区分。
测试：收集器三条（未动不复活 / 重新生成要回来 / live 的同内容恢复仍短路）、
access 三条（正文不能复活已删授权、多副本只要有 live 就可取、非产物引用
仍走正文）、service 七条（归属人校验、他人会话拒绝、并发认领为空、live
行仍在则保留字节、guard 失败保留字节、多版本按对象归并 owner、无
source_path 不扫全会话）、repository 三条（墓碑抗过期写、事务回滚、跨会话
URL 计数）、handler 一条（两个路由的 all_versions 解析矩阵）。其中
「重新生成要回来」在改回旧实现时确认会失败。
client/ 没有 artifacts 相关 API（只有一个 SSE 事件类型常量），无需同步。
|
||
|
lyingbug
|
03d9d66a1f |
feat(agent): size agent history by the window and calibrate token estimates (#3413)
* feat(agent): size agent history by the window and calibrate token estimates Agent history was capped at history_turns (default 5) because compaction could not survive a turn. With checkpoints persisted, load history by tokens instead, and let compaction bound the session. Budget. The loader drops the oldest turns that do not fit, and what it drops is lost: neither replayed nor summarized. A budget equal to the compaction threshold (the earlier attempt, since reverted) let the loader trim first, so whenever one turn was larger than the system prompt plus the new question the request never crossed the threshold; nothing was summarized and the session became a sliding window with no checkpoint. agent.HistoryTokenBudget is now the whole window: overflow reaches the first round's compaction, which summarizes it into a new checkpoint. A turn-by-turn regression test drives the real loader, budget and compactor and fails as soon as a stored turn after the checkpoint is missing from the loaded history; restoring the old budget fails it. Summarizer input. History can now reach the window, so the part being summarized can outgrow one request. An overflowing request degrades to a raw archive that is never persisted, so every later turn would fail the same way. Prepare keeps the newest messages that fit, the prompt states how many were left out, and file paths still come from all of them. Calibration. cl100k counts Chinese at about a token a character where Qwen and DeepSeek spend about 0.6. The estimator now carries a scale (clamped to 0.5-1.5, text only), learned from consecutive requests in a turn: they share tools, system prompt and earlier messages, so the provider's growth between them prices exactly the appended messages, without mixing in how a provider renders tool schemas. The turn's scale is stored on its usage (context_token_scale); the next turn's loader prices history with it and hands it to the engine. Loading, the first compaction check and compaction itself share one estimator, so they agree; calibrating only the trigger would let the loader drop turns again, which the regression test covers. Also: history is priced as the engine sends it (redacted KB/Wiki results cost one line); a checkpoint older than the rows read is matched by the assistant rows' (created_at, id) order; a read ending on a turn's question keeps that turn; a compaction discarded for freeing under 5% still persists its checkpoint; a checkpoint write matching no row is an error; a partial index backs the newest-checkpoint lookup. Agent mode no longer reads history_turns: the field leaves AgentConfig, the editor hides it in smart-reasoning mode, and quick-answer keeps its turn cap. * fix(agent): address review of window-sized history loading Summarization. The summarizer call had a fixed 60s total timeout. With history loaded up to the window, a request could carry ~175k tokens and be cut off while prefilling or writing normally; the degraded result left no checkpoint, so the next turn reloaded the same history and failed the same way. Summarization now streams and is cancelled only after no output for the engine's stall timeout (120s by default, LLMCallTimeout overrides), with the overall ceiling left to the transport, as for the engine's own rounds and as in Codex and Pi. A stopped turn is not retried. When the summarizer still fails, the raw archive is persisted as a checkpoint marked degraded so the loop is broken, and the next compaction refines it as the previous summary. The archive is held to the summary budget, newest messages first, so a degraded compaction always frees room. Loading. The loader priced turns as sent but returned and held them as stored, so wiki pages stored in full and sent as one line could pile up in memory. It now returns the as-sent history and releases each turn's stored rows once replayed, keeping about a page of rows at a time. Index. The partial checkpoint index is replaced by a composite (session_id, created_at DESC, id DESC) index serving the backwards page, the newest-checkpoint lookup and the recent-messages reads. It is built with CREATE INDEX CONCURRENTLY so writes to messages are not blocked; the migration holds that single statement, since a multi-statement query runs as an implicit transaction. A PostgreSQL integration test, skipped without WEKNORA_MIGRATION_TEST_POSTGRES_DSN, applies the chain, runs 000106 down and up through golang-migrate, and checks both queries use the index without a sort; the SQLite migration test checks the same plans. Calibration. It no longer waits for a total token count, and a usage with only a scale is persisted. A turn that measures nothing carries forward the scale it started from, so the newest turn always has the latest one. |
||
|
wizardchen
|
2a40cc37c0 |
fix(migrations): renumber the context checkpoint migration to 000105
#3406 (000104_skill_served_version) and #3407 (000104_message_context_checkpoint) each took version 104 and were merged eleven minutes apart. golang-migrate refuses to load a directory with a duplicate version ("duplicate migration file"), so every PostgreSQL deployment of main fails to migrate. Renumber the later of the two. It is safe for every deployment that could exist: one that migrated between the two merges applied the skill migration as 104 and now applies this one as 105, and after the second merge main could not load its migrations at all, so no deployment applied this file as 104. The migration is ADD COLUMN IF NOT EXISTS either way. Add a test that loads both migration directories. Pull request CI runs on the merge with main, so a second PR taking an already merged number now fails before it lands. |
||
|
lyingbug
|
62f7da5eea |
feat(agent): persist compaction checkpoints across turns (#3407)
* feat(agent): persist compaction checkpoints across turns
Agent compaction only lived inside the turn that ran it: the summary was
discarded at turn end and LoadAgentHistory rebuilt raw history from the
messages table, so every later turn past the threshold paid for another
summarization of nearly the same history.
A compaction whose history summary ends exactly on a stored turn is now
written onto that turn's assistant message (messages.context_checkpoint).
LoadAgentHistory starts from the newest checkpoint: the summary replaces
that turn and everything before it, and later turns are replayed verbatim.
The compactor already treats a leading summary as the previous summary, so
checkpoints fold forward incrementally.
History messages carry their turn's assistant message ID (chat.Message
TurnID, engine-internal) so compaction can tell whether its cut lands on a
stored-turn boundary. Nothing is persisted when the cut ends inside a stored
turn, covers only the live turn, or the history summary degraded to a raw
archive; a split turn's prefix summary is never persisted because that turn
is replayed whole next time. Writes are column-scoped and best-effort, and a
failed lookup falls back to history without a checkpoint.
Storing the checkpoint on the covered turn keeps the boundary implicit, so
session forks copy it with the turn and deleting the turn drops it. The
HistoryTurns cap still applies, now to the turns after the checkpoint.
* test(skill): wait for the background install in the name-conflict test
InstallSkill queues the install on a goroutine, and the fixture row already
starts at installing, so reading the status right after the call raced the
mock install and failed whenever it landed on ready first. Wait for the run
to finish on the reused row instead, as the other install tests do.
* feat(agent): size agent history by tokens instead of turn count
Agent history was capped at history_turns (default 5) because every turn
used to rebuild raw history and compaction could not persist. With
compaction checkpoints persisted, the cap is no longer what keeps a session
inside the window, and it threw away context that would have fit.
LoadAgentHistory now loads the newest checkpoint summary plus the newest
turns after it that fit a token budget: the run's compaction threshold
(agent.HistoryTokenBudget, the window less the reply reserve). History
that large already triggers the first round's compaction, which persists a
new checkpoint, so compaction rather than a turn count bounds the session.
Loading more would only hand that compaction a summarizer input larger
than the model can read.
Rows are paged backwards by (created_at, id) and the read stops at the
checkpoint or once the budget is full, so a long session is never read
whole. Kept turns stay contiguous, the newest turn is kept even when it
alone exceeds the budget (compaction splits it), and a turn cut by the read
is left out rather than replayed without its original question.
Agent mode no longer reads history_turns: AgentConfig.HistoryTurns is
removed, the editor hides the input in smart-reasoning mode and explains
the automatic behavior, and the smart-reasoning builtins drop the setting.
Quick-answer (KnowledgeQA) keeps its turn cap since it has no compaction.
* Revert "feat(agent): size agent history by tokens instead of turn count"
This reverts commit
|
||
|
lyingbug
|
6db8fdd601 |
feat(artifacts): artifacts library page backed by a message_artifacts table (#3384)
* feat(artifacts): add an artifacts library with its own message_artifacts table
Skill-generated files were only reachable from inside the conversation that
produced them. This adds a sidebar entry and a page that lists them across
every conversation the user can see.
Storage: artifacts move out of the messages.artifacts JSONB column into a new
message_artifacts table (PostgreSQL 000103, SQLite 000023), backfilled on
upgrade. The table is now the only source of truth; the old column is kept and
the down migration writes the table back into it. The message repository
loads artifacts for every message it returns and writes them on create,
update (when non-nil) and fork, so callers still use Message.Artifacts.
mod_time is stored as RFC 3339 text because the collector matches files on
nanosecond mtimes, which PostgreSQL timestamps would truncate.
API: GET /api/v1/artifacts lists the latest version of each file (grouped by
session + source path, with version_count), scoped like the session list,
with keyword / file type filters and paging. Downloads keep using the
per-session endpoint.
Frontend: "产物/Artifacts" sidebar entry (hidden without sandbox support), a
library page with type tabs, search, date groups, infinite loading, preview
drawer, download and jump-to-conversation; strings in all five locales. The
sidebar's top-menu paths are now one shared list.
* fix(artifacts): keep the restore hash guard and count versions by URL
RecordRestoredArtifactMtime again only stamps rows whose content hash matches
the restored content (and ignores empty hashes), and never rewrites the hash,
matching MessageArtifacts.WithRestoredMtime. Without the guard a restore of
v1's content also moved v2's mtime and relabelled v2 with v1's hash.
The library's version_count now counts distinct URLs per (session, source
path). A later answer that references an earlier file stores another row with
the same URL, which COUNT(*) reported as an extra version.
* fix(artifacts): scope the library to web chats and clear the legacy column
The library used the session list's default scope, which counts sessions
without an owner as legacy tenant-level rows. IM sessions are created without
an owner, so every member saw the file names and titles of every IM chat
(downloads then 404 for non-admins). The library now applies the same "web"
bucket predicate as the home sidebar, extracted from QueryPaged so the two
cannot drift: no IM mapping (including cleared ones), no embed marker, no
API-key owner.
Migrations 000103 / SQLite 000023 now clear messages.artifacts after the
backfill. Message loads still SELECT *, so the stale copy was read on every
history load. Only rows that carried artifacts are rewritten; the column stays
for instances on the old binary during a rolling upgrade, and the down
migration still rebuilds it from the table.
* fix(migrations): keep the 000103 backfill alive on malformed artifact JSON
A single file_size that is not an integer ("12.5", "") or a created_at that
does not parse ("garbage", month 13) raised a cast error and failed the whole
migration, blocking startup. Casts now go through pg_temp helpers that return
NULL on bad input, falling back to 0 and the message's created_at; valid rows
are unchanged. SQLite casts never raise; its migration test now covers the
same malformed values.
* docs(artifacts): describe the web-only library scope and the cleared legacy column
|
||
|
lyingbugandClaude Fable 5.1
|
76a19f14c4 |
feat(mcp): add built-in MCP server with per-workspace endpoints (#3371)
* feat(mcp): add built-in MCP server with per-workspace endpoints WeKnora previously exposed itself to MCP clients only through the Python package under mcp-server/, which binds one process to one API key and therefore to one workspace, and mirrors the REST API tool-for-tool. This adds a native MCP server surface to the Go backend: - New mcp_endpoints table (PostgreSQL 000100, SQLite 000020). A workspace can publish any number of endpoints, each with its own bearer token (SHA-256 stored, rotatable, shown once), knowledge-base allowlist, tool allowlist, default agent and per-minute rate limit. - Management API under /api/v1/mcp-endpoints (Viewer reads, Admin writes, API keys need manage_channels) plus a tool catalog endpoint. - internal/mcpserver hosts one mcp-go Streamable HTTP server in stateless mode at /mcp/:endpoint_id. tools/list is filtered per endpoint and tools/call re-checks the allowlist, applies rate limiting and touches last_used_at. MCPEndpointAuth resolves the token and injects the workspace, an mcp_endpoint principal and an API-key style scope derived from the endpoint, so existing knowledge-base and capability checks apply unchanged. - Redesigned tool set (12 tools in four groups): retrieval and reading reuse the agent's native tools; ask runs the default or a named agent synchronously and returns a cited answer with a session_id for follow-ups; wiki tools cover search, read and index; write tools are off by default. Knowledge bases accept ids or names. - Frontend: new "MCP Server" integrations tab (tab id mcpserver, to avoid clashing with the existing MCP-client settings section) with endpoint cards, a create/edit drawer, a one-time token dialog and ready-to-paste snippets for Cursor/VS Code/Claude Desktop, Claude Code and mcp-remote. Locales updated in all five languages. - Deployment capability integrations.mcpserver wired on both sides. - Docs: website-docs MCP page now leads with the built-in server; the Python mcp-server README carries a deprecation notice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(mcp): wire the MCP server into DI/router and tighten authorization The previous commit lost eight staged files to the repo pre-commit hook, which resolves ROOT to the main checkout and re-adds the main-checkout copies of tracked files into the worktree index. This commit restores them (container providers, RouterParams and route registration, context key, principal, clone table entry, deployment capability key, SQLite parity test) and addresses the review findings: - Knowledge-base scope is authorized, not trusted: endpoint allowlists and every tool call resolve each knowledge base through access.ResolveKB for the calling principal (ownership or organization share). Foreign IDs are rejected on create/update and dropped at call time. - Write tools mint explicit editor grants via access.KBAccess.WithGrant before calling the knowledge service, so add/update/delete go through the same RequireKBWrite path as the REST API. add_document also checks the grant itself. - ask no longer accepts agent_id. It runs only the endpoint's configured agent; tenant-owned agents and user-facing builtins qualify, internal builtins (wiki fixer, skill installer) are refused both on save and at call time. Web search is forced off for MCP calls. - Tool annotations match behaviour: ask is not read-only, add_document is a mutation. - CORS allows MCP-Protocol-Version / Mcp-Session-Id / Last-Event-ID and exposes Mcp-Session-Id for browser-based MCP clients. - Rate limiter runs its local cleanup goroutine; SQLite migration gains the deleted_at index for parity with PostgreSQL; non-ASCII endpoint names get unique mcpServers keys derived from the endpoint id. - Swagger annotations on the management handler. - Tests: KB isolation (service and server), write grants, endpoint-agent restriction, tool annotations, and a router test that mounts the public /mcp/:endpoint_id surface and checks the deployment capability. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(mcp): renumber migrations after rebase onto main main took 000100 / 000020 for chunks_index_diet, so the mcp_endpoints migrations move to 000101 (PostgreSQL) and 000021 (SQLite); the SQLite parity test and docs follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(mcp): run document tools under the knowledge base owner tenant Shared knowledge bases resolved through access.ResolveKB were only half usable: documents of an organization-shared knowledge base are stored under the owner tenant, so the document-level tools that queried by the calling tenant found nothing, and add_document would have created rows under the caller's tenant inside someone else's knowledge base. - knowledgeInScope loads documents tenant-agnostically and then requires the document to belong to an allowed knowledge base and to that base's tenant. - scopedKBContext replaces grantContext: it resolves the grant, switches the execution tenant to the knowledge base owner (KBAccess.Context) and swaps TenantInfo to the owner so storage/model settings resolve there. list_documents, read_document and every write tool run under it. - Locale text and docs no longer claim clients can pass agent_id. - Go SDK gains MCP endpoint methods (client/mcp_endpoint.go) with docs, and the Swagger artifacts are regenerated. - Tests: shared knowledge base read/write under the owner tenant, spoofed document tenant rejected, viewer share cannot write. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(mcp): renumber migrations after rebase onto main main now carries 000101 / 000021 (knowledge_profiles), so the mcp_endpoints migrations move to 000102 (PostgreSQL) and 000022 (SQLite); the SQLite parity test expects version 22 and the docs follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(mcp): guard last_used touch without a repository and wrap SDK line The transport tests build the server without a repository; the background last_used_at write then dereferenced a nil interface and, on the slower CI runner, took the test binary down. touchLastUsed now returns early when no repository is wired, and the tests inject a recording repository and assert the touch instead. Also wraps a 123-column SDK signature for the client module's lll check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
lyingbug
|
5574ebc011 |
feat(knowledge): derive knowledge base descriptions from document profiles (#3370)
* feat(knowledge): derive knowledge base descriptions from document profiles Knowledge bases only had a hand-written description, which most users leave empty, so agents bound to several knowledge bases had nothing to route on and searched all of them. This adds a generated description that is derived from per-document data instead of maintained by a model. Document profile: the summary call now returns JSON with a 2-3 sentence summary plus a gist, 3-5 topic keywords, a document type and one typical question (knowledges.profile). Input is capped at 8k chars and output at 1024 tokens; plain-text replies from custom templates still work and simply carry no profile. The summary chunk embeds the gist and short summary. Knowledge base aggregate: a database-only fold over those rows (counts per file type, tag, normalized topic and doc type; stratified samples of titles and questions; a hash of the inputs). Deleting, moving or re-parsing a document changes nothing but the rows, so the next rebuild is exact without any model judgement. Description text: one small model call over the aggregate yields a gist, merged topics and typical questions (knowledge_bases. generated_profile). The call is skipped when the aggregate hash is unchanged; failures keep the previous text and record the error; an empty knowledge base clears it. The manual description is never touched, and the agent runtime context renders both so the model can pass knowledge_base_ids instead of fanning out. Refreshes are requested from summary task exit, post-process completion, deletion, cross-KB moves and enabling the option, debounced per knowledge base in 30 s windows via kb:profile on the summary queue. POST /knowledge-bases/:id/profile/generate regenerates synchronously. profile_config (enabled, model_id, custom_instructions) opts a document knowledge base in; the settings dialog shows the generated card with regenerate and adopt-as-description actions in five locales. Search results now log hits per chunk type so the value of indexing summary chunks can be measured. Migrations 000101 (PostgreSQL) and 000021 (SQLite) add the columns; the SQLite schema test expects the new version and columns. * fix(knowledge): keep generated descriptions out of duplicates and refresh clones DuplicateKnowledgeBase copies the source row through JSON, which carried generated_profile onto a knowledge base that has no documents yet, so an empty duplicate advertised the source's topics. The duplicate now starts without a generated description and keeps profile_config so it derives its own once documents arrive. CopyKnowledgeBase builds the target field by field and skipped profile_config, and the clone task never asked for a description after copying the documents, so a clone stayed undescribed until its next upload. The target now carries a copy of profile_config and both clone paths request a refresh on completion. * fix(knowledge): refresh the loaded clone target and carry document profiles ProcessKBClone requested the description refresh with the reservation stub it builds before CopyKnowledgeBase runs. For the main create path that stub has neither a type nor profile_config, so the refresh helper rejected it and nothing was enqueued. The refresh now uses the loaded target returned by CopyKnowledgeBase. cloneKnowledge copied the summary text but not the structured profile, so even a scheduled refresh would have aggregated titles without topics until every clone was re-summarized. Cloned documents now carry an independent copy of the profile. |
||
|
lyingbugandClaude Fable 5.1
|
68d99d7d21 |
perf(chunk): speed up chunk ingestion and batch updates (#3369)
* perf(chunk): speed up chunk ingestion and batch updates Chunk writes were dominated by avoidable work in the repository layer: - UpdateChunks built one CASE WHEN id = ? chain per column, so a batch of N rows evaluated 5N branches per row (O(N²)). On PostgreSQL it now joins against a VALUES list keyed by the primary key (O(N), batched by 1000); other dialects run one small UPDATE per row inside a single transaction. - CreateChunks copied Content into SourceContent for every new row, doubling insert volume. SourceContent is only needed once a chunk is edited and the edit path already backfills it lazily, so it is left empty on create. Batch size raised from 100 to 500 and the insert (including the SQLite seq_id pre-assignment) now runs inside one transaction. - New UpdateChunkFieldsByIDs applies uniform field values with a single UPDATE ... WHERE id IN per batch. FAQ import uses it to flip status after indexing instead of shipping every row's content back, and resolves tag names per batch instead of per entry. - Drop three chunks indexes that only slow down writes: chunk_type (low cardinality), content_hash (never used as a filter) and (tenant_id, knowledge_id) (covered by idx_chunks_knowledge_enabled). Migrations 000100 (PostgreSQL) and 000020 (SQLite) with down scripts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf(chunk): rewrite chunk batch writes and FAQ import status updates Go changes belonging to the chunk write-path optimization. The previous commit on this branch only carried the migrations because the repo's pre-commit hook re-staged files from the main checkout when run inside a git worktree, silently dropping the staged Go changes. - UpdateChunks: O(N²) CASE chains replaced by a VALUES join on PostgreSQL (batched by 1000) and per-row updates in one transaction elsewhere. - CreateChunks: stop copying Content into SourceContent, batch 500 rows, run the insert and the SQLite seq_id pre-assignment in one transaction. - New UpdateChunkFieldsByIDs for uniform, tenant-scoped field updates. - FAQ import flips status with UpdateChunkFieldsByIDs and resolves tag names once per batch instead of once per entry. - Drop the gorm index tag on ContentHash to match migration 000100; SQLite schema test expects migration version 20. - SQLite tests for the new and rewritten repository methods. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
42e6163981 |
feat(session): fork conversations from a historical user message (#3304)
* feat(session): add fork schema and history queries 为会话分叉增加数据模型与迁移，并提供截至分叉点的消息查询以及会话内的 request_id 配对。 * feat(sandbox): add git, fork snapshots, and bootstrap hook 沙箱镜像安装 git，Docker 快照使用独立的 fork 命名空间，并在首次创建会话沙箱时接入可选 bootstrap。 * feat(session): checkpoint workspace at the end of each turn 每轮结束后对沙箱 /workspace 打 git checkpoint，并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。 * feat(session): fork conversations and reap orphan snapshots 支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap，并在失败或删除未打开的分叉时回收孤儿快照。 * feat(chat): add session fork entry, prefill, and badges 在用户消息上提供分叉入口，进入新会话时预填原问题，并展示降级提示与来源角标。 * style: add missing newline so session_fork_delete_test.go is gofmt-clean * fix(session): prune later git history when rolling a fork workspace back reset --hard 只还原工作区，分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc，避免错误基线。 * style(session): clear golangci-lint findings on the session-fork diff Pre-push lints the whole PR against main; wrap long lines, check c.Error, and fix gofumpt/revive/staticcheck so the merge can be pushed. * fix(chat): show the fork button on hover without shifting the thread Overlay the action so revealing it no longer expands the user bubble and janks the messages below. * fix(chat): drop the fork landing banners Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread. * fix(agent): dedupe artifacts by content hash, not size Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only. * feat(session): allow forking from an assistant answer User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists. * fix(session): resolve parent attachments from a forked chat Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss. * fix(agent): rewrite sandbox refs after a content-hash skip Fork restores change mtime on unchanged files, so Collect returns empty and named citations were left as sandbox: paths. Resolve those names against session artifacts, notify only real uploads, and stamp restored mtime only on the matching content hash. * fix(session): drop a dead fork snapshot and retry checkpoint rewrite Create from a missing template now abandons bootstrap so the next Resolve uses a normal sandbox. A failed checkpoint rewrite no longer marks the bootstrap consumed, which previously made nested forks SANDBOX_REPLACED. * fix(session): load forked parent attachments into the prompt ResolveForPrompt used session-scoped lookup, so copied parent attachment IDs passed Get but never reached the model. Use the same resolveInSession path as preview. * fix(chat): keep the user-message fork button visible The control was hover-only, which hid it on touch. Always show it, and drop the unused degrade tooltip. --------- Co-authored-by: hailongzhao <hailongzhao@tencent.com> Co-authored-by: wizardchen <wizardchen@tencent.com> |
||
|
lyingbug
|
d7ccd5ba51 |
fix: align client sessions and integration request handling (#3305)
* fix: align client sessions and integration request handling * fix: preserve trusted Doris redirects and prepare browser tests * fix: align proxy address checks and integration test coverage |
||
|
wizardchen
|
17f893865d |
fix(memory): rank memories by the query, not by importance
Long-term memory chose its candidates before it looked at the question. Recall listed 400 items by `importance DESC, valid_from DESC`, loaded the vectors belonging to that list, and scored those; extraction did the same with a 200-item pool. Importance is uncorrelated with relevance, so the vector stage could only re-order what importance had already picked, and a memory below the cut was unreachable no matter how well it matched. The capacity cap goes up to 2000, so everything past the 400th most important memory was permanently unrecallable, and `memory_item_embeddings.vector` being a BYTEA blob meant pgvector was never involved. The store now answers "which of this person's memories is this question about": - Migration 000095 (and the SQLite mirror) add `embedding halfvec` plus a `(tenant_id, subject_id, model_id, dims)` index. The column is conditional on the vector extension, so a deployment whose retrieval driver excludes postgres keeps working. No HNSW: an approximate index is filtered after the fact, and one subject holds at most a couple of thousand rows, so exact ordering inside the scope is both correct and a few milliseconds. - `SearchItemsByVector` ranks in SQL over every vector the subject has, with the status, expiry and kind filters joined in so reaching past the candidate list cannot reach past the filters. SQLite and pgvector-less PostgreSQL fall back to scoring in process, still over the whole scope. - Recall fuses the lexical ranking with those hits and grows the candidate pool with matches the pool did not hold, which is what lets the search escape it. The lexical pool is now the capacity cap, so it can no longer be a second, invisible limit under the one that decides what stays active. - Extraction picks the memories it shows the model the same way. This matters more than recall: the memory a new statement contradicts is not reliably an important one, so an importance-ordered window hid exactly the collision that stops a duplicate being written. - Vectors are written to the searchable column in the upsert's transaction; rows predating the column are moved by `SyncVectorColumn` during maintenance, which costs no model calls. The embedding backfill goes from 50 to 200 per run — at 50, a subject at the capacity cap took over a month to become fully searchable. - Traces and logs report `outside_pool`, the number of semantic matches the lexical pool did not contain. It was structurally zero before, so it is the number to watch when asking whether semantic recall does anything. Tests cover whole-scope ranking, the lifecycle and kind filters, pool growth, the pool-size invariant, and extraction seeing a low-importance memory it would otherwise duplicate. A pgvector integration test (WEKNORA_MEMORY_TEST_POSTGRES_DSN) exercises the migration, the halfvec write, the SQL ranking and the column sync. |
||
|
lyingbug
|
fd7fa6a7d4 |
fix(memory): preserve extraction progress and atomic memory updates (#3224)
* fix(memory): preserve extraction progress and atomic memory updates * fix(memory): bound extraction failures and persist indexed session progress |
||
|
lyingbug
|
7e53c31671 |
feat(browser): connect local Chrome with task controls and display modes (#3180)
* feat(browser): integrate BrowserSkill local task tabs and account pairing * feat(browser): refine task routing, lifecycle, and display modes * fix(browser): reclaim temporary task tabs and groups * test(database): cover browser authorization migration schema |
||
|
|
1e36e071ec |
feat(mcp): add per-tool enable toggle (#2989)
* feat(mcp): add per-tool enable toggle * style: format migration test * fix(mcp): keep enable and approval updates from overwriting each other Partial policy writes now touch only the supplied columns, and a missing tenant fails closed so a disabled tool cannot slip through. --------- Co-authored-by: myysophia <myysophia@users.noreply.github.com> Co-authored-by: wizardchen <wizardchen@tencent.com> |
||
|
ochan.kwon
|
894cbec48a |
feat(agent): report and attribute per-turn LLM token usage
Accumulate token usage across every LLM round of an agent turn and expose the aggregate everywhere a turn is observable: - AgentState.TurnUsage accumulates each round's reported usage, including the final-answer synthesis call; TokenUsage.Accumulate keeps the cache counters as prompt subsets and recomputes the combined cache status (unsupported survives only while every call was itself unsupported). - The agent completion event carries the aggregate, and the complete stream event exposes it both as data.usage and as a typed StreamEvent.Usage that buildStreamResponse promotes into the previously-unused StreamResponse.Usage wire field. - The assistant message persists it as a nullable usage column on both migration targets (postgres 000085 jsonb, sqlite 000012 text), for web and IM channels alike, so history reads still carry the turn's cost after the live stream is gone. - "[LLM Usage]" log lines gain ", session_id=..., principal=type:id" when the context carries them; calls outside sessions stay byte-identical. Usage from failed retry attempts is not accumulated: the per-call usage log remains the ground truth, while the turn aggregate serves as the operational signal. |
||
|
wizardchen
|
71fe7f31ee |
fix(migrations): 补齐 SQLite 迁移与 versioned 的索引与数据迁移缺口
对齐 000005/010/011 的邀请索引、tag_id 迁入与 MCP principal 唯一索引， 并扩展测试覆盖分享链接、OAuth upsert 与 v4 标签迁移路径。 |
||
|
icatw
|
b888172dce |
fix(migrations): 补齐 Lite 模式缺失的 SQLite 版本化迁移
migrations/sqlite 落后于 migrations/versioned，导致 Lite 模式全新数据库 首次建号/建空间失败（tenants.api_principal_config、system_settings、 task_pending_ops 等缺失）。新增 000005-000011 七个 SQLite 迁移， 按 versioned 迁移主题一一补齐： - 000005 <- 000034/000054: messages.attachments、tenant_invitations token/accepted_count - 000006 <- 000041: task_pending_ops / task_dead_letters（与 #2433 同一 DDL） - 000007 <- 000053: system_settings、users.is_system_admin - 000008 <- 000055/000056: knowledge_processing_spans、knowledges.pending_subtasks_count - 000009 <- 000060: embed_channels.allow_memory - 000010 <- 000063: knowledge_tag_relations - 000011 <- 000064: tenants.api_principal_config、mcp_oauth_tokens principal 列 测试：新增 internal/database 迁移测试，覆盖全新库迁移到 v11 与 v4->v11 升级保数据两条路径；go test ./internal/database 全绿。 关联 #2158；与 #2433 范围重叠（任务表部分 DDL 保持一致）。 |