mirror of
https://github.com/Tencent/WeKnora.git
synced 2026-09-19 10:28:49 +08:00
The heuristic splitter relied on two regexes that were too strict for real-world Chinese technical documents: - ChineseChapterPattern required 第 / numeral / unit to be adjacent, so the very common "第 1 章 引言" form never matched. - NumberedSectionPattern required a trailing dot after the numeral, so multi-level numbering such as "1.1 文档目的" or "2.2.1 用户与权限" was missed. As a result, documents like the CHAPTER_SAMPLE shipped with the chunking debug drawer collected zero heuristic markers, and ProfileDocument fell all the way through to the character-level Legacy tier, producing chunks that ignored the document's explicit chapter structure. - Loosen ChineseChapterPattern to tolerate spaces around 第 / 数字 / 单位. - Allow the multi-level branch of NumberedSectionPattern to drop its trailing dot; keep the single-level / roman branch dot-required to avoid false positives on version strings. - Extend patterns_test.go with positive and negative cases covering the CHAPTER_SAMPLE wording plus deep-nesting and lone-numeral regressions.