mirror of
https://github.com/Tencent/WeKnora.git
synced 2026-11-03 14:33:31 +08:00
ochan.kwon
744a367f16
Lays type-system groundwork for the upcoming OpenSearch k-NN driver (Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate: this PR ships only inert constants, schema extensions, and an unreachable normalizer case. No path activates an OpenSearch VectorStore yet — creation continues to fail with "not a valid engine type" until the activation switch lands in Phase 3 PR 3. Changes: - OpenSearchRetrieverEngineType constant ("opensearch") in internal/types/retriever.go. Not added to validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping yet (gated). - ConnectionConfig.InsecureSkipVerify (bool, default false) in internal/types/vectorstore.go, placed inside the // Common section because it is a cross-driver TLS option. Distinct from the Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify only controls verification of an already-TLS HTTPS connection. AES-GCM Value/Scan round-trips the field as plaintext (it travels alongside the encrypted Password / APIKey but is not sensitive itself). - VectorStoreFieldInfo gains four optional fields: Immutable (bool), Min/Max (*float64), Enum ([]string). All omitempty so existing UI schema entries serialize identically. The fields will drive the UI in Phase 3 PR 3 (read-only on Edit + range/enum constraints). - Six new AuditAction constants under vector_store.* and opensearch.* namespaces. Definitions only — emission lands in Phase 3 PR 3. - EngineAwareNormalizer.Normalize is restructured to group engines by the effective score range observed by the normalizer (not the theoretical raw cosine range): Range [-1, 1] (raw cosine, mapped via (score + 1) / 2): - Milvus (COSINE metric mode). Milvus docs explicitly state the COSINE metric range is [-1, 1]. Range [0, 1] (passthrough — already on the target scale): - Elasticsearch v8 / ElasticFaiss. The driver issues a cosineSimilarity(...) script_score script, and Lucene rejects negative final scores ("Final relevance scores from the script_score query cannot be negative" — ES docs); the effective range observed by the normalizer is therefore [0, 1] for IR-normalized embeddings. ES was previously grouped with Milvus and over-corrected via (score + 1) / 2, which inflated every ES result by 50% in mixed-engine RRF fusion. - OpenSearch. The k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation maps the underlying Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1] before the score reaches us. Source: github.com/opensearch-project/k-NN at src/main/java/org/opensearch/knn/index/SpaceType.java (COSINESIMIL enum, scoreTranslation method). - Weaviate. The driver requests `certainty`, defined by Weaviate as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1]. - Postgres pgvector / SQLite sqlite-vec / Qdrant / TencentVectorDB / Doris. The driver computes (1 - cosine_distance) or normalized inner_product, whose theoretical range is [-1, 1]. The IR-normalized positive-component unit vectors WeKnora targets (BGE / OpenAI text-embedding-3 / Cohere / sentence-transformers) keep the observed range in [0, 1]; negative-cosine embedding models would silently clamp to 0 downstream — explicitly documented as the IR-normalization caveat in the struct godoc. Dead enum references (ElasticFaiss, Infinity) are flagged in the godoc with a pointer to internal/types/vectorstore.go's existing "legacy/experimental, no standalone deployable instance" annotation. Their case labels are kept for switch exhaustiveness. Test coverage: - OpenSearch constant wire value + collision check against existing 10 engines + gated invariant (NOT in validEngineTypes). - ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON field deserializes as false) + round-trip + AES-GCM coexistence with encrypted Password / APIKey. - VectorStoreFieldInfo omitempty preservation + new-field serialization + *float64 pointer distinction (min=0 vs nil). - AuditAction dot-namespace convention enforcement, prefix invariants for vector_store.* and opensearch.*, no wire-string collisions, exact wire-value pins for the six new constants. - EngineAwareNormalizer: - CosineRange retains its (score + 1) / 2 coverage but now only for Milvus. - UnitInterval now covers the full passthrough group (ES / ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant / Infinity / TencentVectorDB / Doris). - New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an explicit regression guard for the score-range correction: cos=0.5 maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed). - OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift (1.0001) + defensive negative + ±Inf / NaN edges + keyword passthrough + nil-ctx safety. Backward compatibility is preserved at every layer: - All new struct fields are omitempty / pointer-tagged so existing rows and existing wire formats remain unchanged. - Normalizer's new OpenSearch case is unreachable until the driver lands. The ES regrouping changes the post-normalization value for every ES vector search result (a correctness fix, not a feature) — ES vector retrieval is currently the only production path affected. - AuditAction constants emit no audit_log rows in this PR. - engine_type=opensearch VectorStore creation still rejected (gated).