Commit Graph
39 Commits
Author SHA1 Message Date
Windfarer 54da98fc24 feat: add docx max pages env config 2026-04-02 10:31:52 +08:00
wizardchen 8e1cfaccb7 refactor(prompt_templates): improve question generation guidelines and context handling
- Updated the question generation template to clarify the role of surrounding context and main content.
- Enhanced quality rules for generated questions to better align with user search intent.
- Revised output format and added explicit instructions on what not to generate.
- Improved logging and output in the web parser for better visibility of parsed content and metadata.
2026-04-01 21:59:22 +08:00
wizardchen c4f5db7e88 feat(metadata): enhance document processing to include metadata extraction and handling
- Updated DocReaderServicer to pass metadata in responses.
- Modified PipelineParser to accumulate and merge metadata from all parsers.
- Enhanced StdWebParser to extract and log the title from web page content.
- Implemented logic in knowledge service to update knowledge title based on extracted metadata.
2026-04-01 15:49:47 +08:00
Windfarer bc0c2d23d5 add comment 2026-04-01 13:37:37 +08:00
Windfarer 6058f1cbe0 add monkey patch for docx parse error 2026-04-01 13:37:37 +08:00
wizardchen 1c9503b063 feat(docparser): enhance image resolution capabilities in markdown and HTML
- Updated regex patterns in MarkdownImageUtil to support alt text containing brackets and handle MIME types with hyphens.
- Implemented new functions in ImageResolver for resolving HTML <img> tags with data URIs and bare base64 content, improving image handling in markdown.
- Added comprehensive tests for various image scenarios, ensuring robust handling of data URIs and base64 images.
2026-03-25 22:08:29 +08:00
wizardchen 4192cfd072 refactor: simplify image handling in _resolve_images function
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.

These changes improve the clarity and efficiency of image processing and API interactions within the application.
2026-03-03 22:23:23 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
chenrui7 aba7a47b20 fix(parser): resolve chunk index mismatch in logs 2026-01-29 19:32:35 +08:00
ice 6a3c29d455 fix: correctly extract host from completion_url by handling both v1 and non-v1 endpoints 2026-01-28 14:06:43 +08:00
cc 5c7f05189e fix(parser): separate StdMinerUParser and MinerUCloudParser implementation 2026-01-23 10:28:38 +08:00
begoniezhao e655b49ed2 docs: Update README and FAQ for config, OCR, and troubleshooting 2026-01-16 16:05:31 +08:00
begoniezhao 88fd42cbc3 refactor: Restructure OCR module and centralize config 2026-01-16 16:05:31 +08:00
cc 74bf76eb3f fix(parser): provide explicit file_extension for markitdown to resolve #544 2026-01-16 11:50:42 +08:00
cc 3f2792dff4 fix(parser): provide explicit file_extension for markitdown to resolve #544 2026-01-16 11:50:42 +08:00
wizardchen 48a0f9b508 Merge remote-tracking branch 'github_public/main' 2026-01-15 14:39:35 +08:00
begoniezhao 1abdaa5d5c feat: Make OCR and task concurrency configurable 2026-01-15 10:56:09 +08:00
begoniezhao 6c90480e82 refactor: Improve chunk merge/split logic and add validation 2026-01-14 19:36:04 +08:00
begoniezhao a35999466c refactor: Improve error handling and remove unused code 2026-01-14 17:08:26 +08:00
orbisai0security 14ac60128b fix: resolve critical vulnerability V-001
Automatically generated security fix
2026-01-14 17:08:26 +08:00
begoniezhao f25ed71054 fix(doc_parser): Add secure command execution with sandbox 2026-01-14 17:08:26 +08:00
begoniezhao bd0fb766f8 refactor: Improve error handling and remove unused code 2026-01-14 14:21:32 +08:00
orbisai0security e023c26af8 fix: resolve critical vulnerability V-001
Automatically generated security fix
2026-01-12 16:11:44 +08:00
begoniezhao 91ec651ac0 fix(doc_parser): Add secure command execution with sandbox 2026-01-08 20:24:12 +08:00
wizardchen 6cf7cbcb9c feat: 添加可配置的文件上传大小限制
新增 MAX_FILE_SIZE_MB 环境变量统一控制文件上传大小,默认 50MB
2025-12-30 14:35:10 +08:00
begoniezhao 907e9a5522 feat: Add DataSchema tool for retrieving schema information from CSV and Excel files 2025-12-29 20:03:51 +08:00
begoniezhao 9ef974e4cf fix: Update storage configuration handling for improved flexibility 2025-12-05 18:02:21 +08:00
begoniezhao 6d1ac5b96d fix: Optimize the header start position 2025-12-04 16:21:41 +08:00
begoniezhao 96fd7a390f fix: Improve image decoding by using original format if available 2025-12-03 15:25:46 +08:00
begoniezhao 5d6729fc0c refactor: Introduce create_chunking_config function for better readability and maintainability 2025-12-01 18:07:43 +08:00
begoniezhao 3e31fdeefd style: add necessary comments to improve code quality
- Add docstrings and inline comments for key functions and complex logic
- Unify comment style, eliminate magic numbers and ambiguous variable names
- No functional changes, only improve maintainability
2025-12-01 17:43:26 +08:00
begoniezhao a0685a22db refactor: remove unnecessary header handling in chunk processing and update example usage 2025-11-21 15:28:14 +08:00
begoniezhao 154025f723 refactor: 优化解析器日志与API检查逻辑,简化异常处理 2025-11-20 15:05:53 +08:00
begoniezhao 587d1b2bd3 feat: 新增 CSV、XLSX、XLS 文件类型解析支持 2025-11-19 19:23:16 +08:00
begoniezhao ddbdae686f feat: 新增MarkdownTableUtil,减少md表格空格 2025-11-19 15:14:00 +08:00
begoniezhao 4fdbec17a7 feat: 新增网页解析类,优化依赖及图片编码支持 2025-11-18 22:37:01 +08:00
begoniezhao 2d66abedf0 feat: 新增文档模型类,调整配置与解析逻辑,优化日志及导入
移除日志设置与冗余代码,优化导入、类型提示及OCR后端管理
统一调整各文件模块导入路径为绝对导入
调整导入路径,移除部分导入,优化日志及注释
升级文档解析器为 Docx2Parser,优化超时与图片处理逻辑
2025-11-18 22:37:01 +08:00
begoniezhao 1fd2de5a64 refactor: 更新proto及客户端导入路径,调整Docker构建配置 2025-11-05 12:27:56 +08:00
begoniezhao c1f731e026 chore(docreader): 重新组织模块文件 2025-11-05 12:07:39 +08:00