fix(knowledge): drop whitespace tokens before SimHash to stop false-positive collisions

Root cause of the cross-file SimHash-collision incident: jieba.lcut splits
Latin-script runs word-by-word but emits every space between words as its
own token (e.g. "The quick brown" -> ["The", " ", "quick", " ", "brown"]).
compute_simhash_64_hex fed that raw token list straight into
Simhash(tokens, f=64), which treats a plain list as unweighted features —
a repeated token's hash gets counted once per occurrence, not once total.
In English/code/config-heavy text the space token can be close to half the
list, so its hash dominates the per-bit majority vote and drowns out the
real content signal.

Verified concretely: two unrelated English sentences ("The quick brown
fox..." vs "Lorem ipsum dolor sit amet...") hashed to the exact same
64-bit value before this fix, and diverged to 57.8% similarity after
filtering whitespace-only tokens. Confirmed against two real production-
pattern files (an HR reimbursement policy .docx and a docker/yaml
deployment .md, both flagged in the earlier investigation) that also
collided on the same value pre-fix.

Existing behavior for CJK-only text is unaffected — jieba doesn't emit
space tokens between CJK characters, and the "near-duplicate Chinese
paragraphs" regression test still reports 93.75% similarity. Adds a
regression test for the collision this fixes.

This explains why the standalone recompute in
scripts/repair_false_positive_simhash_duplicates.py left flagged files
unchanged (987/987 on a 171 dry-run): the algorithm itself, not stale
data, produced the collision, so recomputing without this fix reproduces
the same wrong value. Files should now get a genuinely different SimHash
on their next reparse or the next repair-script run.

src/backend/bisheng/common/utils/simhash_utils.py

@@ -8,11 +8,25 @@ def compute_simhash_64_hex(text: str) -> str:
     """Compute a 64-bit SimHash of *text*, return as 16-char lowercase hex.
 
     Uses jieba for CJK-aware tokenization. Empty/whitespace text produces "0" * 16.
+
+    jieba.lcut splits Latin-script runs word-by-word but emits every space
+    character between them as its own token (e.g. "The quick brown" ->
+    ["The", " ", "quick", " ", "brown"]). Simhash() takes an unweighted list,
+    so a repeated token's hash gets counted once per occurrence — in English/
+    code/config-heavy text, space tokens can be close to half the list,
+    letting that single low-information token's hash dominate the bit vote
+    and drown out the real content. Two unrelated documents with a similar
+    word/space ratio can then converge on the same or a near-identical
+    fingerprint regardless of what they actually say (verified: two
+    unrelated English sentences hashed identically before this filter, and
+    diverged to 58% similarity after it). Dropping whitespace-only tokens
+    removes that dominant no-signal feature without changing behavior for
+    CJK-only text (jieba doesn't emit space tokens between CJK characters).
     """
     text = (text or "").strip()
     if not text:
         return "0" * 16
-    tokens = jieba.lcut(text)
+    tokens = [token for token in jieba.lcut(text) if token.strip()]
     sh = Simhash(tokens, f=64)
     return f"{sh.value:016x}"
 

src/backend/test/knowledge/test_simhash_utils.py

@@ -48,3 +48,26 @@ def test_similarity_unrelated_text_low():
     h_b = compute_simhash_64_hex("足球运动员转会市场的经济学分析与球队预算管理")
     # Just assert it's strictly less than 1 — exact value depends on tokenization
     assert similarity(h_a, h_b) < 1.0
+
+
+def test_similarity_unrelated_latin_text_not_1():
+    """Regression test for the whitespace-token collision bug.
+
+    jieba.lcut splits Latin-script runs word-by-word but emits every space
+    between words as its own token. Before filtering those out,
+    Simhash(tokens, f=64) — an unweighted list where a repeated token's hash
+    is counted once per occurrence — let the space token's hash dominate the
+    bit vote for any text with a lot of English/code content, so unrelated
+    English/Latin sentences with a similar word/space ratio collapsed onto
+    the exact same 64-bit fingerprint (similarity == 1.0) regardless of what
+    they actually said.
+    """
+    h_a = compute_simhash_64_hex(
+        "The quick brown fox jumps over the lazy dog near the riverbank "
+        "while birds sing in the morning sun."
+    )
+    h_b = compute_simhash_64_hex(
+        "Lorem ipsum dolor sit amet consectetur adipiscing elit sed do "
+        "eiusmod tempor incididunt ut labore."
+    )
+    assert similarity(h_a, h_b) < 1.0