Commit Graph
11 Commits
Author SHA1 Message Date
wizardchen 3ab8809435 feat(plugin): plugins bring IM platforms
contributes.imChannels is open to third-party plugins. A plugin IM
platform (its qualified ID, e.g. acme.zulip/zulip) joins the channel
editor's list with its instanceSchema as the credential form; secrets are
redacted and merged like the builtin platforms'. Channels connect over
webhooks: WeKnora relays each callback to the plugin, which answers the
platform (challenges, acks) and hands back the user's message; the
channel's agent answers and the plugin sends the reply. Long-connection
platforms are not supported yet.

The IM platform columns widen to 160 characters (versioned 000122) for
qualified IDs. Go and Python SDKs gain IM channels.
2026-09-26 16:29:51 +08:00
wizardchen fb0c612220 feat(plugin): pipeline hooks in knowledge Q&A
contributes.pipelineHooks join up to three stages of the knowledge Q&A
pipeline: rewriteQuery may replace the question retrieval uses,
filterResults may drop and reorder retrieved passages (it answers IDs, so
it cannot add any), answer may append Markdown after the finished answer,
streamed or not, fallbacks included. One internal chatpipeline plugin
adapts them; each call waits 5 seconds at most and a hook that fails is
skipped. Only workspaces with the plugin on are hooked.

Go and Python SDKs gain pipeline hooks.
2026-09-26 16:20:11 +08:00
wizardchen 30dc887202 feat(plugin): plugins contribute chunkers
contributes.chunkers adds chunking strategies a knowledge base can pick
(plugin:<plugin>/<chunker>). The plugin answers spans of the text in
characters; WeKnora cuts the text itself, so a chunker cannot rewrite
content. It runs as a tier ahead of the profiler's choice: an error, a
timeout, a span outside the text or the plugin being off in the workspace
falls back to the builtin tiers. With parent-child chunking the plugin cuts
the parents only.

Go and Python SDKs gain Chunker; the strategy picker and the chunking
preview list and show plugin chunkers.
2026-09-26 16:08:37 +08:00
wizardchen 9c20afa259 feat(plugin): finish P2 - sync trigger, event gaps, off hints, icons, editors, netns
- Host API scope "datasources": a plugin lists the workspace's data
  sources of its own connectors and starts incremental syncs (skipped
  while one is running). Go Host.DataSources / SyncDataSource, Python
  data_sources / sync_data_source. The Jira example's issues webhook uses
  it to sync the affected projects at once, checking Jira's X-Hub-Signature
  when a secret is set.
- Events: deleting a whole knowledge base now publishes knowledge.deleted
  per document; documents the wiki drain completes and stalled documents
  housekeeping fails publish ingested / failed.
- Instance lists (data sources, web search, models, IM channels) tag
  instances whose plugin is switched off in the workspace, and name
  plugin types from the contribution listing when the type list no
  longer has them.
- Plugin cards show icons: the package icon as manifest.iconData (a data
  URI filled when the package opens), builtin model vendors' logos, and
  the app's brand logos for other builtins.
- Instance editors: connectors and webSearch contributions may name an
  editor page shown below the generated form; it reads the instance and
  sets values through the new form.set bridge call and "values" event.
  Plugin connector settings now stay editable while stored credentials
  are kept.
- WEKNORA_PLUGIN_NETNS=1 (Linux) runs host plugins in their own user and
  network namespace through the plugin-sandbox helper, which relays the
  egress proxy and the Host API over unix sockets; plugins cannot bypass
  the proxy. Starting fails, clearly, where user namespaces are refused.
2026-09-26 13:54:34 +08:00
wizardchen e84081a4f6 feat(plugin): agent tools served by plugins, with result views
A plugin with code can now serve MCP tools itself: an mcpServers
contribution without mcp.url. Messages travel as plugin calls
(POST /v1/mcp/{id}, JSON-RPC inside the envelope), so tool calls carry the
workspace's context and configuration (OAuth fields swapped for tokens)
and the same authentication as every other call; servers are stateless.

- pluginapi MCP types and helpers (TextResult, StructuredResult,
  ToolError); Go p.Tool, Python @plugin.tool; OpenAPI, conformance
  (initialize + tools/list), and served-manifest checks.
- WeKnora side: internal/mcp takes registered transports; the plugin
  transport calls through the invoker, and plugin-served servers appear
  per workspace as MCP services with transport "plugin". Structured
  content is no longer dropped by the MCP client.
- toolViews on mcpServers map a tool's structuredContent to a result view:
  table, cards, kv, markdown, json, or one of the plugin's pages (mount
  mcpServers/<id>, context {tool, arguments, result}). MCP tool results
  carry the view as display_type plugin_tool_view; the chat renders it
  with PluginToolView and prefers it over the generic MCP call view.
- Jira example: search_issues and get_issue tools on a workspace
  connection (tenant config with x-oauth and x-options), shown as a
  table and as Markdown.
2026-09-26 13:04:46 +08:00
wizardchen d46fa25e41 feat(plugin): dynamic form options, OAuth connections and resource limits
Plugin config forms can now ask the plugin while they are filled in:

- x-options {name, dependsOn, search} loads a field's choices from
  /v1/options/{name}. POST /api/v1/plugins/:id/options merges the form's
  values with the stored secrets (system, tenant or instance scope) so a
  redacted token still works. Go and Python SDKs, OpenAPI and conformance
  cover the endpoint.
- x-oauth turns a string field into an account connection. WeKnora runs
  the authorization code flow (PKCE, client credentials from the plugin's
  system config), keeps the tokens sealed in plugin_oauth_connections and
  the field holds "oauth:<id>". The invoker swaps references for fresh
  access tokens at call time, refreshing near expiry. Callback:
  /api/v1/plugin-oauth/callback, one-time states in Redis or memory.
- SchemaForm gains an options widget (dependsOn reload, remote search)
  and an OAuth connect widget; connector, web search, workspace and
  platform config forms provide the plugin behind them.

Host plugins are capped per runtime.resources on Linux: RLIMIT_DATA for
memory (WEKNORA_PLUGIN_MEMORY_DEFAULT when undeclared) and, with a
delegated cgroup v2 (WEKNORA_PLUGIN_CGROUP), memory.max and cpu.max.
2026-09-26 11:51:35 +08:00
wizardchen e095ee7444 feat(plugin): event delivery and webhooks for plugins
Events:
- A plugin lists what it wants in permissions.events: knowledge.ingested,
  knowledge.failed, knowledge.deleted or chat.answered. The admin reviews
  the list at install; a plugin that only listens needs no other
  contribution.
- A publish enqueues one plugin:event task per subscribed plugin the
  workspace has on (maintenance queue, Redis or Lite).
- The task calls the plugin's new /v1/events at least once. Retryable
  failures retry up to 10 times, keeping the event ID; other failures drop
  the event.
- The plugin host refuses a process that subscribes without serving
  events.

Where events come from:
- A document reaches completed or failed through a dozen paths but a few
  repository writes, so a knowledge repository wrapper raises
  knowledge.ingested and knowledge.failed on the transition only:
  UpdateKnowledge, UpdateKnowledgeForTransfer, UpdateKnowledgeColumns,
  FinalizeSubtask, CompleteProcessingWithoutSubtasks.
- knowledge.deleted comes from the shared delete path.
- chat.answered comes from where RAG and agent answers are persisted. The
  stop button's early completion, which has no question, is skipped.

Webhooks:
- contributes.webhooks gives each workspace a secret URL per webhook:
  /api/v1/plugin-callbacks/{plugin}/{webhook}/{tenant}.{mac}. The MAC is
  keyed from SYSTEM_AES_KEY/JWT_SECRET.
- WeKnora answers 404 unless the URL is genuine, the webhook exists and the
  workspace has the plugin on. Calls are rate-limited per URL, bodies are
  capped at 1 MB and cookies are stripped.
- The call is relayed to the plugin's /v1/webhooks/{id} with that
  workspace's envelope; the plugin verifies the sender.
- Admins get the URLs from GET /plugins/:id/webhooks, shown with copy
  buttons in the plugin center's configure drawer. Plugins get them in
  context.webhooks when APP_EXTERNAL_URL is set.

SDKs:
- Go: OnEvent and Webhook.
- Python: @plugin.on_event and @plugin.webhook.
- Conformance, OpenAPI and READMEs are updated.

Also: examples/plugins/activity (Python) keeps a workspace activity feed
from events and a secret-checked note webhook, and shows it on a toolbox
page.
2026-09-26 11:21:55 +08:00
wizardchen 3dc1470299 feat(plugin): sandboxed plugin pages and the @weknora/plugin-ui bridge
Plugins can add pages to the app. Each is an HTML entry under ui/ in the
package:

| Point              | Shown as                      | Default minRole |
| ------------------ | ----------------------------- | --------------- |
| `pages`            | toolbox tabs                  | viewer          |
| `settingsSections` | a section under Plugins in settings | admin     |
| `kbTabs`           | a tab on each knowledge base (gets knowledgeBaseId) | viewer |

Declarative plugins may have pages too.

Isolation:
- A page runs in `<iframe sandbox="allow-scripts allow-forms allow-popups
  allow-popups-to-escape-sandbox">` without allow-same-origin, so it has an
  opaque origin and cannot read the app's storage.
- Its files come from /api/v1/plugin-ui/assets/{id}/{version}/ui/...
  (served without login, only files under ui/ of the loaded version).
- They carry a CSP with connect-src 'none', so the page cannot reach the
  network.

Bridge:
- The page talks to the app only over postMessage: `api.request`,
  `ui.toast`, `ui.confirm`, `ui.navigate`, `ui.resize`, `ui.close`.
- The app answers only its frame's window, rate-limited, and validates
  paths and sizes.
- It sends init (plugin, mount, locale, role, theme tokens, context) and
  theme/locale changes.
- api.request goes to POST /api/v1/plugins/:id/ui-request. The app checks
  the workspace has the plugin on and the caller meets the page's minRole
  (same rules as the route guards), then calls the plugin's new
  /v1/ui/request endpoint with the caller's role.

SDKs:
- Go: `p.UI` and `UIJSON`.
- Python: `@plugin.ui` and `UIResponse`.
- Both list "ui" in their manifest; conformance checks the endpoint; the
  OpenAPI spec documents it.
- packages/plugin-ui is the page side: a dependency-free ES module with
  types that connects, applies the theme as --wk-* CSS variables and keeps
  the frame sized.

Also:
- Fix the Python SDK reading Go timestamps with fewer than 6 fraction
  digits on Python < 3.11. Key-value entries tripped on it.
- examples/plugins/links: a Python plugin with all three kinds of page,
  storing links in the Host API key-value store.
2026-09-26 10:54:33 +08:00
wizardchen b0767453e4 feat(plugin): document parsers from code plugins
Plugins can contribute parser engines (contributes.parsers, with the
fileTypes they handle). Each registers under its qualified ID
(acme.ocr/ocr) next to the builtin engines and the docreader's, kept in a
separate store so the builtin plugin catalog never describes it as a
builtin. A knowledge base picks it in its parser rules like any engine;
uploads of its file types are accepted.

The engine calls POST /v1/parsers/{id}/parse with the file as base64 and
takes back Markdown plus the images it references, which the pipeline
stores and chunks as usual. A retryable plugin error (unavailable, rate
limited) fails the call so the task retries; any other error fails the
document for good. Other engines' tenant credentials are never passed to
a plugin. A plugin engine that is gone no longer falls through to the
Python docreader; the parse fails saying the plugin is not running. The
engine list honours the workspace's plugin switches and shows plugin
engines by their localized names.

SDK: p.Parser(id, ...), openapi.yaml and a conformance check.
examples/plugins/subtitles parses SRT and WebVTT into a transcript and
counts parsed files per workspace through the Host API key-value store.
2026-09-25 23:50:08 +08:00
wizardchen 5fa64e200a feat(plugin): Host API with plugin tokens and a key-value store
Plugins can call back into WeKnora. Each call to a plugin that asked for
Host API scopes (permissions.hostApi, accepted at install) carries the
Host API URL and a five-minute token bound to the plugin, its version,
the calling tenant and those scopes. Tokens are HS256 JWTs with their own
audience, keyed from the cluster secrets (SYSTEM_AES_KEY, else
JWT_SECRET), and are checked by /api/v1/plugin-host/* itself, outside
user and API-key auth.

The first scope is kv: a key-value store partitioned by plugin and tenant
(plugin_kv, migrations 000117 / sqlite 000036), so plugins keep state
without a database of their own. Get, put with an optional TTL, delete and
paged prefix listing; keys up to 256 bytes, JSON values up to 64 KB,
10,000 keys per plugin and tenant; expired entries read as missing and are
swept every ten minutes.

The SDK exposes it as call.Host().KVGet / KVPut / KVDelete / KVList. Host
plugins reach this node on loopback, which their environment now excludes
from the egress proxy.
2026-09-25 23:40:21 +08:00
wizardchen 39045543d1 feat(pluginsdk): extension protocol v1, Go SDK and conformance suite
A new module, github.com/Tencent/WeKnora/pluginsdk, so plugin authors import
the protocol without WeKnora's dependency tree:

- pluginapi: the protocol (envelope with context and the system / tenant /
  instance config scopes, error codes, NDJSON stream events, web search and
  connector messages, the host handshake line, HMAC request signing) plus
  openapi.yaml describing it for other languages.
- pluginsdk: build a plugin in Go. Register web search providers and
  connectors, then Serve: started by the plugin host it listens on the
  socket it is given, prints the handshake and accepts only the host token;
  run on its own it verifies signed requests. Envelope deadlines reach the
  handler's context, panics become internal errors, and fetches stream
  items and checkpoints.
- client: calls a plugin over a handshake socket or HTTPS, with typed
  transport errors; WeKnora's adapters and the conformance suite share it.
- conformance + cmd/weknora-plugin-conformance: checks a running plugin
  answers discovery, rejects unauthenticated and malformed calls, and
  answers every endpoint it claims with an output or a protocol error.

A test keeps openapi.yaml and the SDK routes in step. A new workflow builds
and tests the module on Linux, macOS and Windows; go-lint lints it.
2026-09-25 22:26:29 +08:00