mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
Waleed
220f8c8b10
* feat(knowledge): add chunking strategies and regex strict boundaries - Add Token, Sentence, Recursive, and Regex chunkers with strategy selection in create-base modal - Add opt-in strict boundaries mode for regex chunker so each match becomes its own chunk - Add chunking strategies docs page with industry references * fix(chunkers): strip capturing groups and validate strictBoundaries scope - Convert capturing groups to non-capturing in regex chunker so split() doesn't surface delimiter text as spurious chunks - Reject strictBoundaries in chunkingConfigSchema when strategy is not regex * fix(chunkers): also strip named capture groups in regex patterns Named groups (?<name>...) are still capturing groups so split() interleaves their matched text. Convert them to non-capturing alongside plain ( groups. * fix(chunkers): exclude lookbehind from named-group rewrite Tighten NAMED_GROUP_PREFIX with negative lookahead so patterns like (?<=<tag>) are not misidentified as named capture groups.