mirror of
https://github.com/Tencent/WeKnora.git
synced 2026-10-13 00:44:45 +08:00
lyingbug
04adc13272
* fix(retrieval): keep index writes and shared-KB lookups correct Index writes - Image OCR/caption chunks were indexed without IsEnabled, so every engine but Postgres stored them disabled and retrieval never returned them. The SQLite engine re-enables the affected rows at startup. - FAQ edits, similar-question additions and import merges re-index under the old SourceID and relied on the engine overwriting it. Postgres and SQLite kept the stale row (ON CONFLICT DO NOTHING); ES v8 and Qdrant mint a new document ID and kept both. FAQ re-indexing now deletes the old rows first, and Postgres/SQLite BatchSave upserts by (source_id, source_type). - Postgres: drop the GORM default tag on is_enabled, which replaced false with true on insert and indexed disabled chunks as enabled. - SQLite: BatchSave no longer attaches vectors and FTS rows to the wrong rows when part of a batch already exists, and vecTables is guarded against concurrent map writes. - CopyIndices: Postgres filters to the copied knowledge, pages by id and keeps is_enabled; SQLite copies every row of a chunk (generated questions included) with stable source IDs and enabled state. Shared knowledge bases - Chat merge: FAQ answer population and neighbor expansion look chunks up without the caller-tenant filter, like parent resolution (#3342). - Neighbor expansion never cuts the retrieval hit; the length budget goes to the text adjacent to it. - FAQ negative questions are applied whenever an FAQ KB is in scope and to chunks of shared FAQ KBs. - Search results get image_info for shared-KB chunks. - search_knowledge reads FAQ metadata from the search result instead of one tenant-scoped query per hit, so shared FAQs render their answers. * fix(retrieval): put every retrieval score on one scale Scores from different retrieval paths were ranked together while living on different scales, so ordering depended on which path produced a hit. Fusion - RRF ranked a chunk by its position in the concatenation of every vector (or keyword) list, appended in goroutine completion order. The second list's best hit ranked behind every hit of the first, and the order changed between identical requests. Each list is now ranked on its own by score and a chunk takes its best rank. - RRF is divided by its maximum, so hybrid scores land in [0, 1] like vector-only (cosine) and keyword-only (rescaled BM25) results. Raw RRF topped out near 0.016 and lost to any single-retriever hit it was ranked against across searches (document targets, model groups, FAQ calls); rerank's base-score term and MMR's relevance term had nothing to use. - Keyword-only BM25 is rescaled per list; engines' BM25 is not comparable. - FAQ iterative retrieval fuses each round like the first pass instead of sorting raw BM25 and cosine together, starts at twice the depth already searched, triggers when any single vector list is full (the old summed count never matched with more than one list) and replaces results per round. Rerank and chat pipeline - The FAQ boost is no longer capped at 1; capping made strong FAQs tie at 1.0 and fall to the tie-breaker. The composite score before boosts is kept as composite_score. - The FAQ direct-answer threshold is compared against that pre-boost score, and the FAQ that cleared it is the one marked match="exact" rather than the first FAQ. - Graph hits no longer carry a fabricated retrieval score of 1.0, which ranked them above every scored hit when rerank did not run; the composite score uses the model score in its place. Engines - OpenSearch and Weaviate return (1 + cos) / 2; their drivers now report cosine similarity and map thresholds onto the engine scale, so a threshold means the same on every engine. - Milvus L2 returned squared distance, ranked farthest-first downstream, and took the threshold as a radius; scores and radius are converted. - The multi-store normalizer no longer maps Milvus through (s + 1) / 2, which put it on a different scale from every other engine and from its own single-store results. Docs updated for fusion, normalization, FAQ boost and iterative recall. * perf(retrieval): bound rerank, agent and pipeline cost; stop failing on slow or missing models Rerank - Rerankers report the longest document a request can carry (max_document_chars / max_request_chars minus the query) and the rerank stage fits its own passages to it, trimming appended captions, OCR and generated questions first. One chunk with a large screenshot's OCR used to exceed the vendor limit and fail the whole batch, so every candidate lost its score and the threshold stopped applying. - Rerank HTTP requests default to a 60 s timeout; chat and agent contexts carry no deadline, so a stalled self-hosted rerank server held the turn open. - OCR and caption chunks no longer repeat their own text from image_info in the passage. - The chat pipeline and the agent tool rerank at most 200 candidates, best retrieval score first. - A rerank model that cannot be loaded degrades to retrieval order instead of failing every chat turn, as the search APIs already do; without rerank results the merge stage keeps only RerankTopK hits instead of enriching every one. Retrieval APIs - knowledge-search match_count and rerank.top_k are capped at 200 (unbounded values multiplied into tens of thousands of billed rerank passages); normalizedMatchCount clamps to the pool size so the over-retrieval arithmetic cannot overflow. - hybrid-search with rerank recalls at least rerank.top_k deep. Agent - search_knowledge fits its rendered output to four fifths of the tool output budget, dropping the lowest-ranked rows and telling the model how many it omitted; the model's view is rebuilt from Data, which registry truncation never reached. - Agent searches skip neighbor/parent/related enrichment rows, which were never retrieved yet tripled the rerank input. - ParallelToolCalls is enabled. The model was always told it may call tools in parallel while the engine ran them one by one; only read-only tools overlap, anything else stays a barrier. Chat pipeline - Query expansion searches keywords only (its variants are local rewrites whose embeddings repeat the main search) and is skipped when keyword matching is off. - Entity extraction, a second LLM call, is skipped for intents that need no KB retrieval and uses the rewritten query. - Partial-overlap dedup tokenizes each text once instead of per pair, and only compares token overlap between texts within 3x of each other's size. Storage - Migration 000112 adds btree indexes on embeddings(knowledge_id) and (chunk_id); document deletes, chunk re-index and enable/tag updates scanned the whole table. - Shared-KB knowledge rows are fetched in one query instead of one per ID. - Drop GORM Debug() from the ParadeDB keyword query and the knowledge batch lookup, which logged SQL and user queries on every search. Docs updated for the API limits, rerank timeout and passage fitting, agent parallel tools and output budget, and pipeline changes. * fix(retrieval): make agent tools and pipeline stages report what they did Agent tools - query_knowledge_graph now queries the graph store: entity names from the query are matched in each KB's graph, relations are returned (and shown to the model), and the entities' source chunks lead the results, with text search as a supplement. It used to run text search only, so a graph-only KB always answered "no relevant graph information". - search_knowledge reports failures instead of "no matching chunks": when every search fails the tool fails with the reason; partial failures are listed. A failed query embedding falls back to keyword search for hybrid KBs, as the chat pipeline does. - Keyword mode uses rerank to order hits, not to reject them (identifiers score low), and the empty-result hint no longer sends keyword mode back to keyword mode. - Dedup keeps siblings under one parent, no longer merges every image chunk of a document through chunk_index 0, and keeps the better-scored copy. - Search rows omit a <match> excerpt that only repeats the content; snippet terms are segmented for Chinese queries and skip stopwords, and FAQ matched questions are chosen by match strength. - read_document query mode returns next_offset when truncated and resumes from offset. - wiki_search over-fetches under document/tag scopes before filtering and says when every match was out of scope. - Prompts point to read_document(id=cN, context=k) for surrounding text. - The agent editor hides the retrieval-strategy section in agent mode, where search_knowledge uses the global retrieval settings. Chat pipeline - Intent labels are normalized; an unknown label retrieves instead of silently skipping the knowledge base, and web_search without web search enabled searches the knowledge bases. - Image turns get a 2048-token rewrite budget, and a reply cut off by the cap is salvaged field by field instead of being discarded. - History references are matched by how much of the query they cover; Jaccard against chunk-sized text never reached its threshold. - Entity search resolves chunks of org-shared KBs within the turn's KB scope, skips chunks whose document is gone (it used to panic) and caps graph chunks at 30. - Empty history is not fetched twice; fetched pages are cut on a rune boundary. Retrieval - A multi-KB search whose primary KB has no vectors embeds the query with a vector KB's model instead of failing on an empty model ID. - Rerank passages include the chunk's heading breadcrumb, which is part of what was embedded. - SQLite FTS searches the content column only and turns a lone Han character into a prefix query; TencentVectorDB ef follows TopK. Docs updated for the agent tools and pipeline stages. * fix(retrieval): address review findings on the retrieval fixes - Restore NeedsRetrieval for web_search without web search enabled: the intent prompt "web_search" is how the user learns web search is off, and running KB retrieval instead bypassed it. - query_knowledge_graph keeps the case of entity terms; Neo4j matches node names with case-sensitive CONTAINS, so lowercased terms never matched English entities. - An empty search_knowledge or query_knowledge_graph result states partial failures in its text, which is all the model sees for an empty result. - SQLite BatchSave replaces a source in one transaction, so a failed insert keeps the old rows instead of leaving the source unindexed. - Reuse stringSliceValue and an exported searchutil.ContainsChinese instead of new copies; drop an unused named return and the FAQ obsolete-source delete that the full per-chunk delete now covers. * fix(retrieval): address second review of the retrieval fixes - query_knowledge_graph: under a document/tag scope, keep only relations whose entities come from an in-scope evidence chunk (the graph namespace is the whole KB); drop chunks whose document no longer exists; report a graph or text search failure even when the other source returned hits. - search_knowledge: record the hybrid -> keyword fallback taken when the query embedding fails in mode_fallbacks, so an empty result is not read as "no semantic match". - query understanding salvage: take a cut-off value only for the image description; a truncated rewrite_query or intent is dropped instead of replacing the user's query. - Graph hits (no retrieval score) are kept past the no-rerank RerankTopK cut and the rerank candidate cap instead of always falling below them. - FAQ boost is capped at 1 again so scores stay on the [0, 1] scale; FAQs tied at the cap are ordered by their pre-boost score. - Negative-question filtering loads only candidates of FAQ knowledge bases. - read_document query continuation starts one chunk before offset so the first match keeps its preceding context at a page boundary. - graphSearchTerms and searchQueryTokens share one queryTerms splitter.