English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.
中文：打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析，保留文档结构和本地链接，不执行脚本或发起网络请求，并补充文件校验、引擎注册、编码及解析行为的回归测试。
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
Stop hardcoding DOCREADER_PDF_RENDER_PARALLELISM=4 in compose so low-core
containers use min(4,cpu). Clarify WEKNORA_LANGUAGE precedence and align
docreader JPEG quality docs with the 85 default.
- docreader: serialize all pdfium access behind a global lock. pypdfium2
is process-global and not thread-safe, so concurrent PDF uploads could
corrupt its shared state and deadlock the whole process (requests stuck
in "Parsing document" forever). Non-PDF parsers still run concurrently.
- logger: strip ANSI color escapes from file logs so on-disk logs stay
plain text while stdout keeps terminal colors.
- storage: log baseDir/resolvedPath on file-open failure and when the
multimodal service resolves a file service, so a LOCAL_STORAGE_BASE_DIR
mismatch between writer and reader is immediately visible.
- docker: set TZ (default Asia/Shanghai) for the docreader container.
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
WPS embeds images in cells using =DISPIMG("ID",mode) which may appear as plain text prefixed with "_xlfn." when processed by openpyxl. These function strings pollute the RAG-indexed document content, causing incorrect AI recall.
Add a regex filter (_is_image_function) that detects _xlfn.DISPIMG(...) and _xlfn.IMAGE(...) patterns and excludes them during row processing in ExcelParser.parse_into_text().
Refs: https://github.com/Tencent/WeKnora/issues/1779
Add a `quick` flag to `opendataloader_available` so registry listing can use
a single short-timeout health check instead of the patient retry/backoff loop.
Parsing keeps the original tolerant probe to handle a hybrid service that is
still starting up.
- Moved the _SEPARATOR_CELL regex definition to a more appropriate location in the Markdown parser.
- Implemented a fallback mechanism in the gRPC document reader to handle cases where the ReadStream RPC is unimplemented, ensuring compatibility with older versions.
- Added a readUnary method to maintain backward compatibility with the legacy unary Read RPC.
- Improved cancellation handling in the MinerUCloud and PaddleOCR-VL readers to prevent excessive API calls during context cancellation.
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:
docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
thread-safe so we fan out across processes. Configurable via
DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
image, so documents with many page images are no longer capped by the unary
gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
backward compatibility; the Go production reader switches to ReadStream.
VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
the default 90s -> 180s so dense scanned-page OCR does not time out with
"context deadline exceeded".
Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
a single large document cannot saturate the shared worker pool and block
user-facing document parsing:
image:multimodal -> "multimodal"
chunk:extract -> "graph"
question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
scanned-queue set (so cancelling a knowledge still purges its pending tasks).
- Implemented a retry mechanism for DOC to DOCX conversion to handle concurrent `soffice` invocations, ensuring each attempt uses a dedicated user profile directory.
- Added logging for each conversion attempt, including success and failure messages, to improve visibility into the conversion process.
- Adjusted the handling of temporary directories for both conversion output and user profiles, enhancing robustness against conversion failures.
- Added a regex pattern for image file extensions to the utils module for better image detection.
- Updated the BODY_XPATH in the xpaths module to prioritize matching specific content structures in web pages.
- These changes aim to improve the accuracy and efficiency of content extraction from web pages using the StdWebParser class.
docreader fails to start on a fresh `uv sync` of main:
$ uv sync --frozen
$ uv run python -m docreader.main
Traceback (most recent call last):
...
File ".../docreader/proto/docreader_pb2_grpc.py", line 19, in <module>
raise RuntimeError(...)
RuntimeError: The grpc package installed is at version 1.76.0, but the
generated code in docreader_pb2_grpc.py depends on grpcio>=1.78.0.
Please upgrade your grpc module to grpcio>=1.78.0 or downgrade your
generated code using grpcio-tools<=1.76.0.
`docreader_pb2_grpc.py:8` was regenerated with grpcio-tools 1.78+ and
hardcodes `GRPC_GENERATED_VERSION = '1.78.0'`, then at import time runs
`first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION)` and
raises if the installed runtime is older. But `pyproject.toml` still
pins:
"grpcio>=1.76.0",
"grpcio-health-checking>=1.76.0",
"grpcio-tools>=1.76.0",
…and `uv.lock` resolves to grpcio 1.76.0. So every fresh clone +
`uv sync` ends up with a docreader that crashes on import, and a
WeKnora backend that thinks docreader is "not connected" — silently
falling back to the Go-side static engine registry only.
Downstream user-visible symptom: `/api/v1/system/parser-engines`
returns just the Go-side engines. Python-only engines (e.g.
markitdown, plus any custom Python parsers) never appear in the KB
create/edit form. Document parsing falls back to mineru/builtin even
when the user intended to use docreader-backed engines.
Reproduce on main (before this commit):
cd docreader
uv sync --frozen
uv run python -m docreader.main
# → RuntimeError at import, server never starts
After this commit:
Resolved 79 packages
Updated grpcio v1.76.0 -> v1.80.0
Updated grpcio-health-checking v1.76.0 -> v1.80.0
Updated grpcio-tools v1.76.0 -> v1.80.0
...
$ uv run python -c "from docreader.proto import docreader_pb2_grpc; print('ok')"
ok
Fix is the minimum-floor bump on the three grpcio packages, plus the
lockfile update uv emits as a result (all 1.76.0 → 1.80.0, newest
stable matching the proto's floor). No proto regen, no code change —
just aligning the dep declaration with what the generated client
already requires.
The next regen of docreader_pb2_grpc.py (e.g. with grpcio-tools 1.80+)
would raise the floor again; pinning at the proto's declared floor is
the conservative choice that keeps this in sync.
Follow-up to #1359. Addresses a set of correctness and security gaps in
the initial docreader auth implementation.
- docker-compose: inject GRPC_TLS_*/GRPC_TLS_SERVER_NAME/GRPC_AUTH_TOKEN
into the WeKnora-app service. Without this the Go client never saw the
knobs, so enabling token auth on the server broke every RPC.
- client: bind tokenAuth.RequireTransportSecurity() to TLSEnabled so a
bearer token cannot be sent over an insecure channel once TLS is on.
- server: load_tls_credentials now raises TLSConfigError on misconfig
(cert/key missing, file unreadable, mTLS without CA); main.py exits 1
instead of silently downgrading to insecure.
- server: replace endswith("/Check"|"/Watch") health bypass with exact
match against /grpc.health.v1.Health/{Check,Watch}.
- server: compare tokens with hmac.compare_digest, warn on tokens < 16B.
- server: AuthInterceptor now returns an abort handler matching the
original RPC kind (unary/stream) and uses context.abort, so streaming
RPCs surface UNAUTHENTICATED instead of INTERNAL.
- internal/infrastructure/docparser/grpc_parser.go: drop the duplicated
TLS/tokenAuth block and reuse docreader/client.LoadAuthConfigFromEnv +
BuildDialOptions. Single source of truth for client-side auth.
- Add GRPC_TLS_SERVER_NAME (client SNI override) and
GRPC_MTLS_REQUIRE_CLIENT_CERT (server explicit mTLS toggle); document
the differing CA semantics between client and server in .env*.example.
- Reject half-configured client mTLS (cert XOR key) loudly.
- Fix missing trailing newline in .env.lite.example.
Verified locally: go build ./... and go vet ./... clean; auth.py
fail-fast / token paths smoke-tested.
- Updated `pydantic` from 2.12.3 to 2.13.4 and `pypdfium2` from 5.0.0 to 5.8.0 in the `docreader` requirements.
- Modified the dependabot configuration to set `open-pull-requests-limit` to 0 and added an `ignore` rule for version updates across all ecosystems, allowing only security updates.
- Adjusted settings for `server-security`, `client-security`, `frontend-security`, and `miniprogram-security` groups to streamline security update handling.
This change aims to enhance dependency management and maintain security while reducing noise from version update PRs.
- Bump base image in docker/Dockerfile.app from golang:1.24 to golang:1.26
to match `go 1.26` declared in go.mod (fixes CI build failure on
`go mod download`).
- Drop unused docreader components and their dependencies:
- Remove `docreader/ocr/` package (paddle/vlm/dummy backends are
unreferenced by the main flow; OCR/VLM is handled by the Go App).
- Remove `docreader/parser/storage.py` (dead code; image persistence
happens in the Go App via inline ImageRef bytes).
- Remove `docreader/scripts/download_deps.py` (PaddleOCR pre-download).
- Drop deps: paddleocr, paddlepaddle, openai, ollama, minio,
cos-python-sdk-v5, oss2, asyncio, pypdf2, markdown, mistletoe,
goose3, markdownify, pdfplumber, antiword, urllib3.
- Re-lock uv.lock: 145 -> 79 packages.
- Update docreader/README.md to reflect that OCR/VLM/storage are no
longer configured at the docreader level.
The Go production splitter now uses DefaultChunkOverlap = 80
(~15% of DefaultChunkSize). Bringing the Python sidecar splitter to
the same number prevents subtle behavior drift if the docreader path
ever falls back to it for chunking.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
The default DOCREADER_DOCX_MAX_PAGES=100 silently truncates large
documents, causing users to see at most ~1000 chunks regardless of
document length. Change the default to 0 (no limit) so all pages are
processed. Operators who need a cap can still set the env var.
Fixes#719
- Introduced PDFScannedParser as a fallback parser for scanned PDFs that converts pages into images for OCR processing.
- Updated PDFParser to include PDFScannedParser in the parsing chain, enhancing the document parsing capabilities for scanned content.
- Improved logging for better error tracking during PDF parsing operations.
- Updated the question generation template to clarify the role of surrounding context and main content.
- Enhanced quality rules for generated questions to better align with user search intent.
- Revised output format and added explicit instructions on what not to generate.
- Improved logging and output in the web parser for better visibility of parsed content and metadata.
- Updated DocReaderServicer to pass metadata in responses.
- Modified PipelineParser to accumulate and merge metadata from all parsers.
- Enhanced StdWebParser to extract and log the title from web page content.
- Implemented logic in knowledge service to update knowledge title based on extracted metadata.
- Updated regex patterns in MarkdownImageUtil to support alt text containing brackets and handle MIME types with hyphens.
- Implemented new functions in ImageResolver for resolving HTML <img> tags with data URIs and bare base64 content, improving image handling in markdown.
- Added comprehensive tests for various image scenarios, ensuring robust handling of data URIs and base64 images.
xiaowang