lyingbug
04adc13272
* fix(retrieval): keep index writes and shared-KB lookups correct Index writes - Image OCR/caption chunks were indexed without IsEnabled, so every engine but Postgres stored them disabled and retrieval never returned them. The SQLite engine re-enables the affected rows at startup. - FAQ edits, similar-question additions and import merges re-index under the old SourceID and relied on the engine overwriting it. Postgres and SQLite kept the stale row (ON CONFLICT DO NOTHING); ES v8 and Qdrant mint a new document ID and kept both. FAQ re-indexing now deletes the old rows first, and Postgres/SQLite BatchSave upserts by (source_id, source_type). - Postgres: drop the GORM default tag on is_enabled, which replaced false with true on insert and indexed disabled chunks as enabled. - SQLite: BatchSave no longer attaches vectors and FTS rows to the wrong rows when part of a batch already exists, and vecTables is guarded against concurrent map writes. - CopyIndices: Postgres filters to the copied knowledge, pages by id and keeps is_enabled; SQLite copies every row of a chunk (generated questions included) with stable source IDs and enabled state. Shared knowledge bases - Chat merge: FAQ answer population and neighbor expansion look chunks up without the caller-tenant filter, like parent resolution (#3342). - Neighbor expansion never cuts the retrieval hit; the length budget goes to the text adjacent to it. - FAQ negative questions are applied whenever an FAQ KB is in scope and to chunks of shared FAQ KBs. - Search results get image_info for shared-KB chunks. - search_knowledge reads FAQ metadata from the search result instead of one tenant-scoped query per hit, so shared FAQs render their answers. * fix(retrieval): put every retrieval score on one scale Scores from different retrieval paths were ranked together while living on different scales, so ordering depended on which path produced a hit. Fusion - RRF ranked a chunk by its position in the concatenation of every vector (or keyword) list, appended in goroutine completion order. The second list's best hit ranked behind every hit of the first, and the order changed between identical requests. Each list is now ranked on its own by score and a chunk takes its best rank. - RRF is divided by its maximum, so hybrid scores land in [0, 1] like vector-only (cosine) and keyword-only (rescaled BM25) results. Raw RRF topped out near 0.016 and lost to any single-retriever hit it was ranked against across searches (document targets, model groups, FAQ calls); rerank's base-score term and MMR's relevance term had nothing to use. - Keyword-only BM25 is rescaled per list; engines' BM25 is not comparable. - FAQ iterative retrieval fuses each round like the first pass instead of sorting raw BM25 and cosine together, starts at twice the depth already searched, triggers when any single vector list is full (the old summed count never matched with more than one list) and replaces results per round. Rerank and chat pipeline - The FAQ boost is no longer capped at 1; capping made strong FAQs tie at 1.0 and fall to the tie-breaker. The composite score before boosts is kept as composite_score. - The FAQ direct-answer threshold is compared against that pre-boost score, and the FAQ that cleared it is the one marked match="exact" rather than the first FAQ. - Graph hits no longer carry a fabricated retrieval score of 1.0, which ranked them above every scored hit when rerank did not run; the composite score uses the model score in its place. Engines - OpenSearch and Weaviate return (1 + cos) / 2; their drivers now report cosine similarity and map thresholds onto the engine scale, so a threshold means the same on every engine. - Milvus L2 returned squared distance, ranked farthest-first downstream, and took the threshold as a radius; scores and radius are converted. - The multi-store normalizer no longer maps Milvus through (s + 1) / 2, which put it on a different scale from every other engine and from its own single-store results. Docs updated for fusion, normalization, FAQ boost and iterative recall. * perf(retrieval): bound rerank, agent and pipeline cost; stop failing on slow or missing models Rerank - Rerankers report the longest document a request can carry (max_document_chars / max_request_chars minus the query) and the rerank stage fits its own passages to it, trimming appended captions, OCR and generated questions first. One chunk with a large screenshot's OCR used to exceed the vendor limit and fail the whole batch, so every candidate lost its score and the threshold stopped applying. - Rerank HTTP requests default to a 60 s timeout; chat and agent contexts carry no deadline, so a stalled self-hosted rerank server held the turn open. - OCR and caption chunks no longer repeat their own text from image_info in the passage. - The chat pipeline and the agent tool rerank at most 200 candidates, best retrieval score first. - A rerank model that cannot be loaded degrades to retrieval order instead of failing every chat turn, as the search APIs already do; without rerank results the merge stage keeps only RerankTopK hits instead of enriching every one. Retrieval APIs - knowledge-search match_count and rerank.top_k are capped at 200 (unbounded values multiplied into tens of thousands of billed rerank passages); normalizedMatchCount clamps to the pool size so the over-retrieval arithmetic cannot overflow. - hybrid-search with rerank recalls at least rerank.top_k deep. Agent - search_knowledge fits its rendered output to four fifths of the tool output budget, dropping the lowest-ranked rows and telling the model how many it omitted; the model's view is rebuilt from Data, which registry truncation never reached. - Agent searches skip neighbor/parent/related enrichment rows, which were never retrieved yet tripled the rerank input. - ParallelToolCalls is enabled. The model was always told it may call tools in parallel while the engine ran them one by one; only read-only tools overlap, anything else stays a barrier. Chat pipeline - Query expansion searches keywords only (its variants are local rewrites whose embeddings repeat the main search) and is skipped when keyword matching is off. - Entity extraction, a second LLM call, is skipped for intents that need no KB retrieval and uses the rewritten query. - Partial-overlap dedup tokenizes each text once instead of per pair, and only compares token overlap between texts within 3x of each other's size. Storage - Migration 000112 adds btree indexes on embeddings(knowledge_id) and (chunk_id); document deletes, chunk re-index and enable/tag updates scanned the whole table. - Shared-KB knowledge rows are fetched in one query instead of one per ID. - Drop GORM Debug() from the ParadeDB keyword query and the knowledge batch lookup, which logged SQL and user queries on every search. Docs updated for the API limits, rerank timeout and passage fitting, agent parallel tools and output budget, and pipeline changes. * fix(retrieval): make agent tools and pipeline stages report what they did Agent tools - query_knowledge_graph now queries the graph store: entity names from the query are matched in each KB's graph, relations are returned (and shown to the model), and the entities' source chunks lead the results, with text search as a supplement. It used to run text search only, so a graph-only KB always answered "no relevant graph information". - search_knowledge reports failures instead of "no matching chunks": when every search fails the tool fails with the reason; partial failures are listed. A failed query embedding falls back to keyword search for hybrid KBs, as the chat pipeline does. - Keyword mode uses rerank to order hits, not to reject them (identifiers score low), and the empty-result hint no longer sends keyword mode back to keyword mode. - Dedup keeps siblings under one parent, no longer merges every image chunk of a document through chunk_index 0, and keeps the better-scored copy. - Search rows omit a <match> excerpt that only repeats the content; snippet terms are segmented for Chinese queries and skip stopwords, and FAQ matched questions are chosen by match strength. - read_document query mode returns next_offset when truncated and resumes from offset. - wiki_search over-fetches under document/tag scopes before filtering and says when every match was out of scope. - Prompts point to read_document(id=cN, context=k) for surrounding text. - The agent editor hides the retrieval-strategy section in agent mode, where search_knowledge uses the global retrieval settings. Chat pipeline - Intent labels are normalized; an unknown label retrieves instead of silently skipping the knowledge base, and web_search without web search enabled searches the knowledge bases. - Image turns get a 2048-token rewrite budget, and a reply cut off by the cap is salvaged field by field instead of being discarded. - History references are matched by how much of the query they cover; Jaccard against chunk-sized text never reached its threshold. - Entity search resolves chunks of org-shared KBs within the turn's KB scope, skips chunks whose document is gone (it used to panic) and caps graph chunks at 30. - Empty history is not fetched twice; fetched pages are cut on a rune boundary. Retrieval - A multi-KB search whose primary KB has no vectors embeds the query with a vector KB's model instead of failing on an empty model ID. - Rerank passages include the chunk's heading breadcrumb, which is part of what was embedded. - SQLite FTS searches the content column only and turns a lone Han character into a prefix query; TencentVectorDB ef follows TopK. Docs updated for the agent tools and pipeline stages. * fix(retrieval): address review findings on the retrieval fixes - Restore NeedsRetrieval for web_search without web search enabled: the intent prompt "web_search" is how the user learns web search is off, and running KB retrieval instead bypassed it. - query_knowledge_graph keeps the case of entity terms; Neo4j matches node names with case-sensitive CONTAINS, so lowercased terms never matched English entities. - An empty search_knowledge or query_knowledge_graph result states partial failures in its text, which is all the model sees for an empty result. - SQLite BatchSave replaces a source in one transaction, so a failed insert keeps the old rows instead of leaving the source unindexed. - Reuse stringSliceValue and an exported searchutil.ContainsChinese instead of new copies; drop an unused named return and the FAQ obsolete-source delete that the full per-chunk delete now covers. * fix(retrieval): address second review of the retrieval fixes - query_knowledge_graph: under a document/tag scope, keep only relations whose entities come from an in-scope evidence chunk (the graph namespace is the whole KB); drop chunks whose document no longer exists; report a graph or text search failure even when the other source returned hits. - search_knowledge: record the hybrid -> keyword fallback taken when the query embedding fails in mode_fallbacks, so an empty result is not read as "no semantic match". - query understanding salvage: take a cut-off value only for the image description; a truncated rewrite_query or intent is dropped instead of replacing the user's query. - Graph hits (no retrieval score) are kept past the no-rerank RerankTopK cut and the rerank candidate cap instead of always falling below them. - FAQ boost is capped at 1 again so scores stay on the [0, 1] scale; FAQs tied at the cap are ordered by their pre-boost score. - Negative-question filtering loads only candidates of FAQ knowledge bases. - read_document query continuation starts one chunk before offset so the first match keeps its preceding context at a page boundary. - graphSearchTerms and searchQueryTokens share one queryTerms splitter.
Model context boundary
modelcontext.Registry is the only application-facing boundary for values
that are shortened before an LLM call and restored afterwards. The whole
codec lives in this one package:
| File | Role |
|---|---|
registry.go |
Registry facade — the only type request lifecycles use |
handle_table.go |
handleTable[M], the generic bidirectional primitive behind every handle space |
sources.go |
source handles (cN/dN/bN/wN) and the tool-argument codec |
citations.go |
protocol prompts, <ref/> → <kb/>/<web/> expansion, citation stream expander |
model_output.go |
compact source-centric renderings of tool results |
resources.go |
durable-resource handles (res://NNNN) |
stream.go |
streamHold suffix-hold primitive and every streaming decoder |
tool_policy.go |
the single policy layer: per-tool key contracts + sourceKeySpaces dispatch |
mcp.go |
MCP bridge routing handles (msN/mtN), definition projection and envelope codec |
mcp_sources.go |
bounded citation sidecar for HTTP(S) links in successful MCP results |
handles.go |
exported HandleTable for invocation-local spaces (iN, ref-N, c000) |
leaks.go |
LeakedIdentifiers report of raw UUIDs that survive EncodeMessages, logged at every model-call site |
Identity rules
- Durable application identities: UUIDs, Wiki slugs, URLs, and
resource://...references. - Request-local model handles:
cN,dN,bN,wN,msN,mtN, andres://NNNN. - Tool-private request handles:
iNfor Wiki issues. - Ingest-call handles:
cNNNandref-N, allocated withHandleTable. - Temporary handles are decoded before tool execution, persistence, or UI delivery. They are never accepted as durable authorization or routing data.
Request lifecycle
- Create one
Registryfor the complete request or Agent execution. - Register structured source identities as they enter the context.
For Agent tools, use
EncodeToolsto project MCP routing enums and source summaries. - Call
EncodeMessagesimmediately before the model call. - Decode tool calls through
DecodeToolCallsbefore parsing or execution. - Decode complete responses with
DecodeResponse, or streaming text with oneStreamDecoderper response channel.
Unknown handle-shaped values in declared tool fields are rejected before tool execution. The same per-tool policy is applied when historical assistant tool calls and tool results are replayed, so handles neither disappear nor drift to new numbers across Agent rounds.
The registry owns codec ordering. Resource references are encoded before
source IDs so a Wiki slug such as summary/<knowledge-id> cannot be corrupted
into summary/d1.
Source addressability is separate from citation eligibility. Use
RegisterContextChunk for bound-KB directory entries. Replayed citations,
legacy tool history, and tool arguments also register navigation handles only.
ModelToolResultForTool and explicitly supplied retrieval results authorize
citations for the current execution; rereading a historical source promotes the
same handle without renumbering it. Both complete and streaming decoders drop
references that have not been backed by current evidence.
Observability
Every model-call site runs LeakedIdentifiers on the encoded messages and
logs a warning naming the role or tool whose text still carries a raw UUID.
A registered leak means the codec missed a rewrite site; an unregistered
leak means a producer emitted an identifier that never entered the registry.
Text-level compaction (CompactKnownText) stays in place as defense in depth
until those producers are fixed and the report stays quiet.
Langfuse generation observations contain the exact encoded payload sent to and
returned by the model. Agent tool spans contain both model_arguments (the
model-emitted handles) and resolved_arguments (the durable values actually
executed), plus argument_resolution and any unresolved handles. Sensitive
tools report only argument keys and resolution counts.
Langfuse is an observer of this boundary; it must not implement another handle mapping.
Built-in tool policies
Canonical ID-bearing arguments for knowledge, graph, data, web, and Wiki tools
are decoded centrally through a tool-name plus JSON-field allowlist. The
sourceKeySpaces table is the single source of truth for which keys are
ID-bearing and which handle space they register into; per-tool sourceIDKeys
contracts gate which of those keys each tool may use.
database_query.sql has an explicit policy because source handles can be
embedded inside quoted SQL values when the model filters the real
knowledges/chunks tables; unquoted SQL aliases and arbitrary prose are
never rewritten. data_analysis.sql carries no handles: the selected document
is always exposed as the fixed table dataset, so only knowledge_id is
decoded. Wiki issue IDs use the same lifecycle
via an iN handle space.
Dynamic MCP tools are intentionally opaque in both arguments and results.
Their schemas and identity semantics are controlled by the MCP server, so the
application must not guess that an arbitrary id, knowledge_id, or url
field is durable or rewrite it. Durable resource handles inside MCP arguments
still use the normal res://NNNN codec. A future MCP ID mapping must be an
explicit server/tool annotation and should plug into this registry rather than
create a parallel mapper.
Successful MCP execution results additionally receive an
external_source_candidates sidecar with up to 50 distinct observed HTTP(S)
links mapped to wN. The external payload is not rewritten, discovery schemas
are not scanned, and article IDs are never used to guess URLs. A candidate link
does not establish that its page was read: the model must select the link whose
associated result supports the claim. When no handle is supplied, the citation
protocol permits an exact supplied source URL as a Markdown link, never a
borrowed knowledge-base citation.
The application-owned MCP bridge is an explicit exception for routing fields:
discover_mcp_tools.server_id uses msN, and call_mcp_tool.tool_ref uses
mtN. Tool definition enums, service summaries, discovery results and replayed
calls share the same request registry. Only directory envelope fields are
registered; input_schema, call_mcp_tool.arguments, and external execution
results remain opaque. Arguments are restored before authorization, validation,
execution, persistence and UI events. Unknown routing handles fail closed.
For model-emitted call_mcp_tool calls, one extra JSON-string encoding of the
arguments envelope is accepted only when it contains a complete JSON object
(including {}). Normalization runs before resource-handle decoding; original
model arguments are retained for tracing. Nested business strings are unchanged,
and malformed JSON, arrays, null, and further encoding layers remain invalid.
The canonical object still passes the existing routing, schema and approval checks.
Wiki routing
WikiRouteResolver is intentionally separate. It stores server-side
slug-to-knowledge-base provenance gathered during Wiki search/read operations,
and can only return knowledge bases already present in the Agent's authorized
Wiki scope. It is routing state, not a model handle table. Reads scan every
legal Wiki KB so cached provenance cannot hide a duplicate slug; mutations
require one unique owner. New pages use link provenance, a single Wiki scope,
or a single source-document owner and reject ambiguous routing instead of
asking the model for a durable KB ID.
Document/tag-constrained Wiki and graph requests are filtered by the same
server-owned SearchTargets provenance. Uncited/global Wiki pages and graph
results outside that subset fail closed; only an explicit whole-KB target
authorizes whole-KB content.