myhloli
366fadb1c9
Promote TSV to an independent file suffix (ADR-0034)
...
- Route tsv through the shared CSV flash pipeline with an independent
file_type and metadata.file_suffix; bump docvortex pin to 0.4.9
- Add ADR-0034 documenting the content-addressed cache identity decision,
annotate ADR-0028, and update docs/next tier-group wording, telemetry
and e2e specs
- Cover detection, sync/async analyze, API server jobs and doclib
lifecycle for tsv in tests
2026-09-16 17:55:07 +08:00
myhloli
b31d3be7de
Integrate DocVortex and preserve MinerU product contracts
2026-09-07 00:43:19 +08:00
myhloli
f1c9546985
refactor: adopt DocGale native parsing and shared document pipeline
2026-09-06 03:52:53 +08:00
jinzhenj and Claude Code
e05073b785
feat: extend file type registry with tsv/shtml and more text extensions
...
- TEXT_EXTENSIONS: add text, ftxt, mdx, latex, adoc, asciidoc
- HTML_EXTENSIONS: add shtml
- CSV_EXTENSIONS: add tsv
- FILE_TYPE_BY_EXTENSION: map new extensions to keep TEXT_FILE_TYPES consistent
- MIME_TYPE_BY_EXTENSION: complete coverage including text/* and tsv
- parser/file_type: resolve .tsv/.shtml via extension sets so they parse as csv/html
Co-Authored-By: Claude Code <noreply@anthropic.com >
2026-09-02 11:01:34 +08:00
myhloli
b67bbd49c9
feat: add OFD document parsing support and update Middle JSON schema to 3.0
2026-08-28 18:35:53 +08:00
myhloli
b39de7d54d
feat: refine page range handling for PDF and non-PDF inputs, ensuring only PDF supports explicit ranges
2026-08-26 19:50:40 +08:00
myhloli
e8ae64a85a
feat: add native EPUB parsing support and update documentation
2026-08-26 15:45:51 +08:00
myhloli
c0b61cee9f
feat: add native OpenDocument parsing support
2026-08-26 13:17:49 +08:00
myhloli
ce729ca48c
feat: add native RTF parsing support
2026-08-26 12:28:55 +08:00
myhloli
5dd63a602a
feat: add support for CSV file parsing and update documentation
2026-08-26 11:36:21 +08:00
jinzhenj and Claude Opus 4.7
c92e6f8abc
feat: support legacy Office binary formats (doc/xls/ppt) end-to-end
...
myhloli 的 7 个 commit 在 backend/model 层加入 DOC/XLS/PPT 原生解析，
但上层 parser/doclib/filetypes/utils/docs/tests 未同步配套。本次补齐：
- filetypes.py: OFFICE_EXTENSIONS 加入 doc/xls/ppt，补全 MIME/FILE_TYPE 映射，
删除 LEGACY_OFFICE_EXTENSION_UPGRADES（已原生支持，不再需要"请转换为 .docx"提示）
- parser/base.py: _build_middle_json_from_legacy 补 is_full_document=True
（修复 1.0 schema 加载必崩的 bug），_LEGACY_DEFAULT_FILE_SUFFIX Literal 扩展
- parser/mineru_parser.py: _FileSuffix/_SUPPORTED_SUFFIXES 扩展，返回类型
tuple[MiddleJson, Any] → tuple[MiddleJson, ModelJson]
- utils/guess_suffix_or_lang.py: 新增 OLE2 容器识别（D0CF11E0 magic +
olefile 内部 stream 区分 doc/xls/ppt），插入 OOXML 失败后、Magika 兜底前
- doclib/services/parse_svc.py: page_count 默认值分支扩展到 doc/ppt/xls，
_unsupported_file_type_message 删除误导性"请转换"提示
- doclib/core/file_io.py: _extract_office_meta 新增 doc/ppt/xls 分支，
用 olefile 提取 OLE SummaryInformation 属性
- doclib/background/compaction.py: model_list_to_pages → model_json_to_pages
迁移（myhloli 已删除 model_list_to_pages）
- docs: CLAUDE.md/AGENTS.md/output_files.md/quick_usage.md 更新支持格式列表
- tests: 新增 test_filetypes.py + test_guess_suffix_ole2.py，
test_parse_result_contract.py 补 is_full_document 断言
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com >
2026-08-25 12:12:54 +08:00
jinzhenj
c622f21545
Clarify text file handling in doclib
2026-07-17 15:28:16 +08:00
jinzhenj
6220b07215
Refine file type constants and tier routing
2026-07-09 20:37:50 +08:00
jinzhenj
95893edf36
Refactor tier defaults and file type policy
2026-07-09 19:37:36 +08:00
myhloli