feat: update ModelJson handling and enhance document processing with new properties

This commit is contained in:
myhloli
2026-08-24 02:52:03 +08:00
parent 3eb043c974
commit 6c23c449d9
28 changed files with 632 additions and 210 deletions
+1 -1
View File
@@ -130,7 +130,7 @@ llm_aided:
cross_page_table_cell_merge: false
```
- `title_leveling` groups paragraph titles into levels 2 through 6 by document-title boundaries and runs only for whole-PDF input where `page_index_map` is `None`. It is skipped for page-selected input.
- `title_leveling` groups paragraph titles into levels 2 through 6 by document-title boundaries and runs only when `MiddleJson.is_full_document` is `true`. Page-selected input is persisted as `false` and skips title leveling.
- `cross_page_table_cell_merge` asks the LLM whether each pair of boundary-row cells continues after the existing rules identify a cross-page table.
- Table cell merge does not require whole-document input. Both features are disabled by default and share one asynchronous client, one connection configuration, and the `max_concurrency` request limit, which defaults to 16.
- `max_concurrency` must be an integer of at least 1 and can be overridden with `MINERU_LLM_AIDED_MAX_CONCURRENCY`.
+2 -2
View File
@@ -53,8 +53,8 @@ llm_aided:
```
两个功能默认关闭并共享同一套 OpenAI-compatible 连接参数。`title_leveling` 在严格
`PageInfo.blocks` 上优化标题层级,但只在 `page_index_map` 为 `None` 的整本 PDF 输入中运行;
抽页输入会跳过标题优化。`cross_page_table_cell_merge` 不受整本输入限制,只消费确定性后处理
`MiddleJson.pages` 上优化标题层级,但只在 `MiddleJson.is_full_document=true` 的整本 PDF 输入中运行;
抽页结果会跳过标题优化。`cross_page_table_cell_merge` 不受整本输入限制,只消费确定性后处理
已经写入的 `continues_prev`,再把逐单元格的 `0/1` 结果写入后表根块 `cell_merge`。两个功能
通过同一个异步客户端并发请求,`max_concurrency` 是不小于 1 的共享请求上限,默认值为 16。
标题分级结果允许 1 到 6,段落标题的 1 会归一为 2,最终严格标题层级范围为 2 到 6。前置
+28 -4
View File
@@ -4,12 +4,13 @@
读者: backend 开发者、SDK 开发者、输出开发者
范围: `doc_analyze()` 第一返回值及 `mineru/types.py` 中的公开 Middle JSON 对象
范围: `doc_analyze()` 返回值及 `mineru/types.py` 中的公开 Model/Middle JSON 对象
## 标准来源
当前事实标准由以下 Pydantic v2 模型定义:
- `ModelJson`
- `MiddleJson`
- `PageInfo`
- `Block` discriminated union
@@ -19,21 +20,44 @@
`model_validate()` / `model_validate_json()`,序列化入口是 `to_dict()` /
`to_json()`。不再提供旧 `Block(...)` 通用构造器或 dict-like 兼容接口。
本轮未迁移的 `ParseResult`、renderer、LLM 标题修正和 Structured Content
不属于本文档描述的对象链路。
本轮未迁移的 `ParseResult`、renderer 和 Structured Content 不属于本文档描述的对象链路。
`doc_analyze()` 与 `aio_doc_analyze()` 固定返回
`tuple[MiddleJson, ModelJson]`,不再把第二返回值暴露为裸 model list。
## `ModelJson`
| 字段 | 类型 | 要求 |
|------|------|------|
| `pages` | `list[list[dict]]` | 必填,保存 Analyze 产生的 raw blocks |
| `page_index_map` | `list[int]` | 必填;空列表表示整本默认顺序,非空时与 pages 等长且唯一递增 |
| `file_suffix` | `pdf/docx/pptx/xlsx` | 必填 |
| `effort` | `flash/low/medium/high/xhigh` | 必填,使用分析后实际值 |
| `parse_mode` | `txt/ocr` | 必填,使用分析后实际值 |
| `mineru_version` | 非空字符串 | 必填 |
`ModelJson.is_full_document` 是由 `page_index_map` 是否为空计算的运行时属性,
不进入 JSON;`resolved_page_indices` 同样不序列化,并把空映射展开为顺序页号。
任意非空映射均表示显式抽页,不根据映射值猜测是否覆盖整本。
文档级转换只接受完整 ModelJson:`model_json_to_pages(model_json)` 负责严格页面
对象化,`model_json_to_middle_json(model_json, llm_aided_config=...)` 负责构造
MiddleJson 并执行适用的 PDF 后处理。不再提供裸 model list 加独立页映射的转换入口。
## `MiddleJson`
| 字段 | 类型 | 要求 |
|------|------|------|
| `pages` | `list[PageInfo]` | 必填,`page_idx` 唯一且严格递增 |
| `is_full_document` | `bool` | 必填,保存整本或抽页语义,不提供默认值 |
| `file_suffix` | `pdf/docx/pptx/xlsx` | 必填 |
| `effort` | `flash/low/medium/high/xhigh` | 必填 |
| `parse_mode` | `txt/ocr` | 必填 |
| `mineru_version` | 非空字符串 | 必填 |
当 `file_suffix="pdf"` 时,每个页面顶层 block 必须具有有效 bbox。Office
顶层 block 可以没有 bbox。
顶层 block 可以没有 bbox。PDF 标题分级直接读取 `is_full_document`;缺少该字段的
旧 Middle JSON 无法通过严格解析,需要从 ModelJson 重新生成。
## `PageInfo`
+1 -1
View File
@@ -130,7 +130,7 @@ llm_aided:
cross_page_table_cell_merge: false
```
- `title_leveling`:仅在 `page_index_map` 为 `None` 的整本 PDF 输入中,以文档标题为边界分组优化 2~6 级段落标题;抽页输入会跳过该功能。
- `title_leveling`:仅在 `MiddleJson.is_full_document=true` 的整本 PDF 输入中,以文档标题为边界分组优化 2~6 级段落标题;抽页结果持久化为 `false` 并跳过该功能。
- `cross_page_table_cell_merge`:在现有规则确认跨页续表后,通过 LLM 判断边界行中各组相邻单元格是否续接。
- table cell merge 不要求整本输入;两个功能默认关闭,并通过同一个异步客户端共享连接参数和 `max_concurrency` 请求上限,默认值为 16。
- `max_concurrency` 必须是不小于 1 的整数,可通过 `MINERU_LLM_AIDED_MAX_CONCURRENCY` 覆盖。
+3 -3
View File
@@ -1,6 +1,6 @@
# Copyright (c) Opendatalab. All rights reserved.
"""MinerU Analyze 与严格 Middle JSON 的最小公开入口。"""
"""MinerU Analyze 与严格 Model/Middle JSON 的最小公开入口。"""
from .types import Block, MiddleJson, PageInfo
from .types import Block, MiddleJson, ModelJson, PageInfo
__all__ = ["Block", "MiddleJson", "PageInfo"]
__all__ = ["Block", "MiddleJson", "ModelJson", "PageInfo"]
+14 -18
View File
@@ -4,17 +4,16 @@
from __future__ import annotations
import asyncio
from typing import Any, Literal, cast
from typing import Literal, cast
from loguru import logger
from mineru.backend.analysis.contracts import AnalyzeEffort, OfficeSuffix, ParseMode
from mineru.backend.analysis.office import analyze_office
from mineru.backend.analysis.pdf.pipeline import analyze_pdf
from mineru.backend.postprocess.llm_aided import apply_llm_aided_postprocess
from mineru.backend.postprocess.pages import model_list_to_pages
from mineru.backend.postprocess.document import model_json_to_middle_json
from mineru.config import config
from mineru.types import MiddleJson
from mineru.types import MiddleJson, ModelJson
from mineru.version import __version__ as mineru_version
_SUPPORTED_FILE_SUFFIXES = {"pdf", "docx", "pptx", "xlsx"}
@@ -36,8 +35,8 @@ def doc_analyze(
image_analysis: bool = True,
page_index_map: list[int] | None = None,
file_suffix: Literal["pdf", "docx", "pptx", "xlsx"] = "pdf",
) -> tuple[MiddleJson, list[list[dict[str, Any]]]]:
"""生产 model-list,并在统一边界构造严格 MiddleJson。"""
) -> tuple[MiddleJson, ModelJson]:
"""生产严格 ModelJson,并在统一边界构造严格 MiddleJson。"""
if file_suffix not in _SUPPORTED_FILE_SUFFIXES:
raise ValueError(f"Unsupported file suffix: {file_suffix!r}")
@@ -52,22 +51,19 @@ def doc_analyze(
result = analyze_office(file_bytes, cast(OfficeSuffix, file_suffix))
_log_infer_performance(file_suffix, len(result.model_list), result.elapsed)
pages = model_list_to_pages(result.model_list, page_index_map)
if file_suffix == "pdf":
apply_llm_aided_postprocess(
pages,
config.llm_aided,
page_index_map=page_index_map,
)
middle_json = MiddleJson(
pages=pages,
model_json = ModelJson(
pages=result.model_list,
page_index_map=page_index_map or [],
file_suffix=file_suffix,
effort=result.effort,
parse_mode=result.parse_mode,
mineru_version=mineru_version,
)
return middle_json, result.model_list
middle_json = model_json_to_middle_json(
model_json,
llm_aided_config=config.llm_aided,
)
return middle_json, model_json
async def aio_doc_analyze(
@@ -77,7 +73,7 @@ async def aio_doc_analyze(
image_analysis: bool = True,
page_index_map: list[int] | None = None,
file_suffix: Literal["pdf", "docx", "pptx", "xlsx"] = "pdf",
) -> tuple[MiddleJson, list[list[dict[str, Any]]]]:
) -> tuple[MiddleJson, ModelJson]:
"""在线程中执行统一文档分析,避免阻塞调用方事件循环。"""
return await asyncio.to_thread(
doc_analyze,
+1 -1
View File
@@ -1,2 +1,2 @@
# Copyright (c) Opendatalab. All rights reserved.
"""将 raw model-list 转换为严格 Middle JSON 页对象的后处理层。"""
"""将严格 ModelJson 转换并增强为 MiddleJson 的后处理层。"""
+29
View File
@@ -0,0 +1,29 @@
# Copyright (c) Opendatalab. All rights reserved.
"""严格 ModelJson 到 MiddleJson 的文档级后处理编排。"""
from __future__ import annotations
from mineru.config import LLMAidedConfig
from mineru.types import MiddleJson, ModelJson
from .llm_aided import apply_llm_aided_postprocess
from .pages import model_json_to_pages
def model_json_to_middle_json(
model_json: ModelJson,
*,
llm_aided_config: LLMAidedConfig,
) -> MiddleJson:
"""从严格 ModelJson 构造 MiddleJson,并执行适用的文档级增强。"""
middle_json = MiddleJson(
pages=model_json_to_pages(model_json),
is_full_document=model_json.is_full_document,
file_suffix=model_json.file_suffix,
effort=model_json.effort,
parse_mode=model_json.parse_mode,
mineru_version=model_json.mineru_version,
)
if model_json.file_suffix == "pdf":
apply_llm_aided_postprocess(middle_json, llm_aided_config)
return middle_json
+11 -12
View File
@@ -8,7 +8,7 @@ import asyncio
from loguru import logger
from mineru.config import LLMAidedConfig
from mineru.types import PageInfo
from mineru.types import MiddleJson
from .llm_client import LLMAidedClient
from .table_merge.llm_cell_merge import apply_llm_cross_page_cell_merge
@@ -17,18 +17,18 @@ from .title_leveling import apply_llm_title_leveling
def _resolve_enabled_features(
config: LLMAidedConfig,
page_index_map: list[int] | None,
middle_json: MiddleJson,
) -> tuple[bool, bool]:
"""根据整本输入约束解析本次实际启用的标题与表格功能。"""
title_enabled = config.features.title_leveling and page_index_map is None
title_enabled = config.features.title_leveling and middle_json.is_full_document
table_enabled = config.features.cross_page_table_cell_merge
if config.features.title_leveling and page_index_map is not None:
logger.info("Skipping LLM title leveling because page_index_map is not None")
if config.features.title_leveling and not middle_json.is_full_document:
logger.info("Skipping LLM title leveling because the input is not a full document")
return title_enabled, table_enabled
async def _apply_llm_aided_postprocess(
pages: list[PageInfo],
middle_json: MiddleJson,
config: LLMAidedConfig,
*,
title_enabled: bool,
@@ -40,9 +40,9 @@ async def _apply_llm_aided_postprocess(
try:
tasks = []
if title_enabled:
tasks.append(apply_llm_title_leveling(pages, resolved_client))
tasks.append(apply_llm_title_leveling(middle_json.pages, resolved_client))
if table_enabled:
tasks.append(apply_llm_cross_page_cell_merge(pages, resolved_client))
tasks.append(apply_llm_cross_page_cell_merge(middle_json.pages, resolved_client))
await asyncio.gather(*tasks)
finally:
if client is None:
@@ -50,20 +50,19 @@ async def _apply_llm_aided_postprocess(
def apply_llm_aided_postprocess(
pages: list[PageInfo],
middle_json: MiddleJson,
config: LLMAidedConfig,
*,
page_index_map: list[int] | None = None,
client: LLMAidedClient | None = None,
) -> None:
"""按配置和整本输入约束同步桥接异步 LLM 后处理。"""
title_enabled, table_enabled = _resolve_enabled_features(config, page_index_map)
title_enabled, table_enabled = _resolve_enabled_features(config, middle_json)
if not title_enabled and not table_enabled:
return
asyncio.run(
_apply_llm_aided_postprocess(
pages,
middle_json,
config,
title_enabled=title_enabled,
table_enabled=table_enabled,
+6 -27
View File
@@ -1,5 +1,5 @@
# Copyright (c) Opendatalab. All rights reserved.
"""raw model-list 到严格 PageInfo 列表的唯一转换边界。"""
"""严格 ModelJson 到 PageInfo 列表的唯一转换边界。"""
from __future__ import annotations
@@ -12,29 +12,11 @@ from mineru.backend.postprocess.lists import fix_office_index_title_blocks, fix_
from mineru.backend.postprocess.page_blocks import process_page_blocks
from mineru.backend.postprocess.paragraphs import merge_para_text_blocks
from mineru.backend.postprocess.table_merge import merge_table
from mineru.types import PageInfo
from mineru.types import ModelJson, PageInfo
PAGE_INFO_LIST_ADAPTER = TypeAdapter(list[PageInfo])
def _validate_page_index_map(
page_count: int,
page_index_map: list[int] | None,
) -> list[int]:
"""校验实际页号映射,禁止 zip 截断、重复页号或逆序页号。"""
if page_index_map is None:
return list(range(page_count))
if len(page_index_map) != page_count:
raise ValueError(f"page_index_map length mismatch: pages={page_count}, mapping={len(page_index_map)}")
if any(isinstance(page_idx, bool) or not isinstance(page_idx, int) or page_idx < 0 for page_idx in page_index_map):
raise ValueError("page_index_map values must be non-negative integers")
if len(page_index_map) != len(set(page_index_map)):
raise ValueError("page_index_map values must be unique")
if any(current <= previous for previous, current in zip(page_index_map, page_index_map[1:])):
raise ValueError("page_index_map values must preserve strictly increasing order")
return list(page_index_map)
def _document_uses_bbox(model_list: list[list[dict[str, Any]]]) -> bool:
"""按整份文档是否出现 bbox 判定 PDF/Office,避免空白首页误判。"""
return any(
@@ -87,13 +69,10 @@ def blocks_to_page_info(
return PageInfo.model_validate(raw_page)
def model_list_to_pages(
model_list: list[list[dict[str, Any]]],
page_index_map: list[int] | None = None,
) -> list[PageInfo]:
"""在 raw 后处理结束后,一次性构造严格且可递归序列化的 PageInfo。"""
page_indices = _validate_page_index_map(len(model_list), page_index_map)
copied_model_list = deepcopy(model_list)
def model_json_to_pages(model_json: ModelJson) -> list[PageInfo]:
"""从严格 ModelJson 无副作用地构造可递归序列化的 PageInfo。"""
page_indices = model_json.resolved_page_indices
copied_model_list = deepcopy(model_json.pages)
use_bbox = _document_uses_bbox(copied_model_list)
if not use_bbox:
fix_office_paragraph_titles(copied_model_list)
+1 -1
View File
@@ -315,7 +315,7 @@ class PdfVlmParser(PdfHybridParser):
"""将旧 PdfVlmParser 构造参数归一到 Hybrid extra_high,避免继续暴露独立 VLM backend。"""
backend = kwargs.pop("backend", "vlm-engine")
kwargs.pop("effort", None)
resolved_backend, resolved_effort = resolve_backend_and_effort(backend, MAX_HYBRID_EFFORT)
resolved_=backend, resolved_effort = resolve_backend_and_effort(backend, MAX_HYBRID_EFFORT)
super().__init__(backend=resolved_backend, effort=resolved_effort, **kwargs)
+40 -1
View File
@@ -304,7 +304,7 @@ def _remove_block_fields(value: Any, excluded_fields: set[str]) -> Any:
class _StrictMiddleModel(BaseModel):
"""Middle JSON 严格模型基类,提供无副作用的统一序列化入口。"""
"""Model/Middle JSON 严格模型基类,提供无副作用的统一序列化入口。"""
model_config = ConfigDict(extra="forbid", strict=True, validate_assignment=True)
@@ -659,6 +659,44 @@ def _iter_child_blocks(block: BlockBase) -> list[BlockBase]:
return [child for child in content if isinstance(child, BlockBase)]
class ModelJson(_StrictMiddleModel):
"""Analyze 返回的完整严格 Model JSON 对象。"""
pages: list[list[dict[str, Any]]]
page_index_map: list[int]
file_suffix: Literal["pdf", "docx", "pptx", "xlsx"]
effort: Literal["flash", "low", "medium", "high", "xhigh"]
parse_mode: Literal["txt", "ocr"]
mineru_version: str = Field(min_length=1)
@model_validator(mode="after")
def _validate_page_index_map(self) -> ModelJson:
"""校验显式抽页映射,并保留空列表作为整本解析标记。"""
if not self.page_index_map:
return self
if len(self.page_index_map) != len(self.pages):
raise ValueError(f"page_index_map length mismatch: pages={len(self.pages)}, mapping={len(self.page_index_map)}")
if any(page_idx < 0 for page_idx in self.page_index_map):
raise ValueError("page_index_map values must be non-negative integers")
if len(self.page_index_map) != len(set(self.page_index_map)):
raise ValueError("page_index_map values must be unique")
if any(current <= previous for previous, current in zip(self.page_index_map, self.page_index_map[1:])):
raise ValueError("page_index_map values must preserve strictly increasing order")
return self
@property
def is_full_document(self) -> bool:
"""返回当前 Model JSON 是否表示整本文档解析。"""
return not self.page_index_map
@property
def resolved_page_indices(self) -> list[int]:
"""返回显式抽页映射或整本文档的顺序页号副本。"""
if self.is_full_document:
return list(range(len(self.pages)))
return list(self.page_index_map)
class PageInfo(_StrictMiddleModel):
"""一页的严格 Middle JSON 内容。"""
@@ -691,6 +729,7 @@ class MiddleJson(_StrictMiddleModel):
"""Analyze 返回的完整严格 Middle JSON 对象。"""
pages: list[PageInfo]
is_full_document: bool
file_suffix: Literal["pdf", "docx", "pptx", "xlsx"]
effort: Literal["flash", "low", "medium", "high", "xhigh"]
parse_mode: Literal["txt", "ocr"]
@@ -1,23 +1,39 @@
from mineru.backend.postprocess.pages import model_list_to_pages
from mineru.types import BlockType, ImageBlock
from typing import Any
from mineru.backend.postprocess.pages import model_json_to_pages
from mineru.types import BlockType, ImageBlock, ModelJson
def _model_json(pages: list[list[dict[str, Any]]]) -> ModelJson:
"""为 raw block 对象化回归构造最小严格 ModelJson。"""
return ModelJson(
pages=pages,
page_index_map=[],
file_suffix="pdf",
effort="flash",
parse_mode="txt",
mineru_version="test",
)
def test_postprocess_groups_pdf_visual_blocks_without_dict_access() -> None:
"""验证 PDF raw 视觉块会在严格对象化边界完成分组。"""
page = model_list_to_pages(
[
page = model_json_to_pages(
_model_json(
[
{
"type": BlockType.IMAGE_CAPTION,
"bbox": [0.05, 0.05, 0.6, 0.12],
"content": "Figure 1",
},
{
"type": BlockType.IMAGE,
"bbox": [0.05, 0.15, 0.6, 0.45],
},
[
{
"type": BlockType.IMAGE_CAPTION,
"bbox": [0.05, 0.05, 0.6, 0.12],
"content": "Figure 1",
},
{
"type": BlockType.IMAGE,
"bbox": [0.05, 0.15, 0.6, 0.45],
},
]
]
]
)
)[0]
assert len(page.blocks) == 1
@@ -30,13 +46,15 @@ def test_postprocess_groups_pdf_visual_blocks_without_dict_access() -> None:
def test_postprocess_groups_raw_vision_footnote_as_image_footnote() -> None:
"""验证通用 raw footnote 会归入相邻图片而不是页脚注。"""
page = model_list_to_pages(
[
page = model_json_to_pages(
_model_json(
[
{"type": BlockType.IMAGE, "bbox": [0.1, 0.1, 0.6, 0.45]},
{"type": "footnote", "bbox": [0.12, 0.47, 0.6, 0.6], "content": "Figure note"},
[
{"type": BlockType.IMAGE, "bbox": [0.1, 0.1, 0.6, 0.45]},
{"type": "footnote", "bbox": [0.12, 0.47, 0.6, 0.6], "content": "Figure note"},
]
]
]
)
)[0]
assert len(page.blocks) == 1
+1
View File
@@ -38,6 +38,7 @@ def _middle(*pages: PageInfo) -> MiddleJson:
"""构造最小严格 MiddleJson 测试对象。"""
return MiddleJson(
pages=list(pages),
is_full_document=True,
file_suffix="docx",
effort="flash",
parse_mode="txt",
+22 -5
View File
@@ -10,7 +10,7 @@ from typing import Any
from pypdf import PdfReader
from mineru.backend.postprocess.page_blocks import process_page_blocks
from mineru.backend.postprocess.pages import model_list_to_pages
from mineru.backend.postprocess.pages import model_json_to_pages
from mineru.model.flash import PdfModel
from mineru.model.flash.native_pdf import (
formulas,
@@ -26,7 +26,7 @@ from mineru.render._internal.common.inline import (
inline_plain_text,
parse_inline_content,
)
from mineru.types import MiddleJson
from mineru.types import MiddleJson, ModelJson
from mineru.utils.pdf_document import PDFDocument, get_lines_from_chars
@@ -38,6 +38,22 @@ _CJK_SYNTHETIC_PDF_NAME = "native_cjk_layout_synthetic.pdf"
_SAFE_WATERMARK_TEXT = "MINERU TEST WATERMARK"
def _model_json(
pages: list[list[dict[str, Any]]],
*,
page_index_map: list[int] | None = None,
) -> ModelJson:
"""为 Flash 样例后处理构造最小严格 ModelJson。"""
return ModelJson(
pages=pages,
page_index_map=page_index_map or [],
file_suffix="pdf",
effort="flash",
parse_mode="txt",
mineru_version="test",
)
# demo4 第 9 至 11 页中实际跨物理行显示的 URL,用于确保纯正则拼接覆盖全部形态。
_DEMO4_WRAPPED_URLS = (
"https://doi.org/10.1007/s00259-025-07388-8",
@@ -434,7 +450,7 @@ def test_demo1_keeps_five_real_tables_without_formula_false_positive() -> None:
assert sum(block["type"] == "equation" for page in model_list for block in page) == 7
assert sum(block["type"] == "caption" for page in model_list for block in page) == 10
assert sum(block["type"] == "footnote" for page in model_list for block in page) == 5
middle_pages = model_list_to_pages(model_list)
middle_pages = model_json_to_pages(_model_json(model_list))
page3_continuation = next(
block
for block in middle_pages[2].blocks
@@ -1053,7 +1069,8 @@ def test_demo3_pages6_7_and10_fix_caption_inline_titles_and_reference_tail() ->
]
page7_markdown = render_markdown(
MiddleJson(
pages=model_list_to_pages([page7], page_index_map=[6]),
pages=model_json_to_pages(_model_json([page7], page_index_map=[6])),
is_full_document=False,
file_suffix="pdf",
effort="flash",
parse_mode="txt",
@@ -1943,7 +1960,7 @@ def test_frozen_soil_reference_tails_remain_single_text_blocks() -> None:
"""验证中文论文双语图注独立标记,正文图引用和参考文献保持 text。"""
model_list = _auto_model_list("中文论文.pdf")
middle_pages = model_list_to_pages(model_list)
middle_pages = model_json_to_pages(_model_json(model_list))
page2_continuation = next(
block
for block in middle_pages[1].blocks
+1
View File
@@ -37,6 +37,7 @@ def _middle(*pages: PageInfo) -> MiddleJson:
"""构造无需 PDF bbox 的严格 Office MiddleJson。"""
return MiddleJson(
pages=list(pages),
is_full_document=True,
file_suffix="docx",
effort="flash",
parse_mode="txt",
+45 -16
View File
@@ -209,6 +209,18 @@ def _merged_row_texts(middle_json: MiddleJson) -> list[list[str]]:
return [[cell.get_text() for cell in row.find_all(["td", "th"])] for row in soup.find_all("tr")]
def _middle_json(pages: list[PageInfo], *, is_full_document: bool = True) -> MiddleJson:
"""为 LLM 后处理单元测试构造最小严格 MiddleJson。"""
return MiddleJson(
pages=pages,
is_full_document=is_full_document,
file_suffix="docx",
effort="flash",
parse_mode="txt",
mineru_version="test",
)
def test_llm_client_injects_optional_thinking_parameter() -> None:
"""验证显式 enable_thinking 通过 extra_body 发送且返回经过结构校验。"""
completions = _FakeCompletions(['</think>{"0": 2}'])
@@ -520,6 +532,7 @@ def test_mixed_cell_merge_flows_through_strict_middle_json_and_renderer() -> Non
asyncio.run(apply_llm_cross_page_cell_merge(pages, client)) # type: ignore[arg-type]
middle_json = MiddleJson(
pages=pages,
is_full_document=True,
file_suffix="pdf",
effort="high",
parse_mode="txt",
@@ -555,9 +568,10 @@ def test_postprocess_reuses_one_client_for_both_features() -> None:
table_pages[1],
]
client = _QueuedValidatedClient([{"0": 3}, [0, 1]])
middle_json = _middle_json(pages)
apply_llm_aided_postprocess(
pages,
middle_json,
_llm_config(title=True, table=True),
client=client, # type: ignore[arg-type]
)
@@ -606,9 +620,10 @@ def test_title_groups_and_table_candidates_run_concurrently() -> None:
PageInfo(page_idx=3, blocks=second_pair[1].blocks),
]
client = _ConcurrentValidatedClient(expected_calls=4)
middle_json = _middle_json(pages)
apply_llm_aided_postprocess(
pages,
middle_json,
_llm_config(title=True, table=True),
client=client, # type: ignore[arg-type]
)
@@ -627,7 +642,7 @@ def test_disabled_postprocess_does_not_construct_client(monkeypatch: pytest.Monk
monkeypatch.setattr(llm_aided, "LLMAidedClient", lambda _config: pytest.fail("client must not be created"))
apply_llm_aided_postprocess([], LLMAidedConfig())
apply_llm_aided_postprocess(_middle_json([]), LLMAidedConfig())
def test_partial_input_skips_title_without_constructing_client(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -636,20 +651,26 @@ def test_partial_input_skips_title_without_constructing_client(monkeypatch: pyte
monkeypatch.setattr(llm_aided, "LLMAidedClient", lambda _config: pytest.fail("client must not be created"))
apply_llm_aided_postprocess([], _llm_config(title=True), page_index_map=[])
apply_llm_aided_postprocess(_middle_json([], is_full_document=False), _llm_config(title=True))
def test_partial_input_skips_title_but_keeps_table_cell_merge() -> None:
"""验证抽页输入不会触发标题请求,但仍会执行跨页单元格分析。"""
pages = _cross_page_table_pages()
pages[0].blocks[0] = pages[0].blocks[0].model_copy( # type: ignore[union-attr]
update={
"index": 1,
"content": [pages[0].blocks[0].content[0].model_copy(update={"index": 1})], # type: ignore[union-attr]
}
)
title = ParagraphTitleBlock(type=BlockType.PARAGRAPH_TITLE, index=0, content="Section", level=2)
pages[0].blocks.insert(0, title)
client = _QueuedValidatedClient([[0, 1]])
middle_json = _middle_json(pages, is_full_document=False)
apply_llm_aided_postprocess(
pages,
middle_json,
_llm_config(title=True, table=True),
page_index_map=[10, 11],
client=client, # type: ignore[arg-type]
)
@@ -665,9 +686,14 @@ def test_front_vlm_cell_merge_detection_remains_disabled() -> None:
assert "enable_cross_page_table_merge=False" in runtime_path.read_text(encoding="utf-8")
def test_doc_analyze_runs_llm_after_deterministic_table_detection(monkeypatch: pytest.MonkeyPatch) -> None:
"""验证 PDF 主链路先产生 continues_prev,再执行后置 LLM 并构造 MiddleJson。"""
@pytest.mark.parametrize("page_index_map", [None, []])
def test_doc_analyze_runs_llm_after_deterministic_table_detection(
monkeypatch: pytest.MonkeyPatch,
page_index_map: list[int] | None,
) -> None:
"""验证默认或空映射的整本 PDF 均先检测续表,再执行后置 LLM。"""
from mineru.backend import analyze
from mineru.backend.postprocess import document
previous_html = "<table><tr><th>H1</th><th>H2</th></tr><tr><td>A</td><td>X</td></tr></table>"
current_html = "<table><tr><th>H1</th><th>H2</th></tr><tr><td>B</td><td>Y</td></tr></table>"
@@ -683,14 +709,12 @@ def test_doc_analyze_runs_llm_after_deterministic_table_detection(monkeypatch: p
calls: list[str] = []
def fake_llm_postprocess(
pages: list[PageInfo],
middle_json: MiddleJson,
_config: LLMAidedConfig,
*,
page_index_map: list[int] | None,
) -> None:
"""断言确定性续表标记已存在,并模拟新增 cell_merge。"""
assert page_index_map is None
current_table = pages[1].blocks[0]
assert middle_json.is_full_document is True
current_table = middle_json.pages[1].blocks[0]
assert isinstance(current_table, TableBlock)
assert current_table.continues_prev is True
assert current_table.cell_merge is None
@@ -698,19 +722,23 @@ def test_doc_analyze_runs_llm_after_deterministic_table_detection(monkeypatch: p
calls.append("llm")
monkeypatch.setattr(analyze, "analyze_pdf", lambda *_args, **_kwargs: result)
monkeypatch.setattr(analyze, "apply_llm_aided_postprocess", fake_llm_postprocess)
monkeypatch.setattr(document, "apply_llm_aided_postprocess", fake_llm_postprocess)
middle_json, _ = analyze.doc_analyze(b"pdf")
middle_json, model_json = analyze.doc_analyze(b"pdf", page_index_map=page_index_map)
current_table = middle_json.pages[1].blocks[0]
assert isinstance(current_table, TableBlock)
assert current_table.cell_merge == [1, 0]
assert model_json.page_index_map == []
assert model_json.is_full_document is True
assert middle_json.is_full_document is True
assert calls == ["llm"]
def test_doc_analyze_does_not_run_llm_for_office(monkeypatch: pytest.MonkeyPatch) -> None:
"""验证 Office 统一入口不会进入仅限 PDF 的 LLM 后处理。"""
from mineru.backend import analyze
from mineru.backend.postprocess import document
result = AnalysisResult(
model_list=[[{"type": BlockType.PARAGRAPH_TITLE, "content": "Slide", "level": 2}]],
@@ -720,9 +748,10 @@ def test_doc_analyze_does_not_run_llm_for_office(monkeypatch: pytest.MonkeyPatch
)
calls: list[str] = []
monkeypatch.setattr(analyze, "analyze_office", lambda *_args, **_kwargs: result)
monkeypatch.setattr(analyze, "apply_llm_aided_postprocess", lambda *_args, **_kwargs: calls.append("llm"))
monkeypatch.setattr(document, "apply_llm_aided_postprocess", lambda *_args, **_kwargs: calls.append("llm"))
middle_json, _ = analyze.doc_analyze(b"office", file_suffix="pptx")
assert middle_json.file_suffix == "pptx"
assert middle_json.is_full_document is True
assert calls == []
+1
View File
@@ -33,6 +33,7 @@ def _middle(*pages: PageInfo, file_suffix: str = "docx") -> MiddleJson:
"""构造最小严格 MiddleJson 测试对象。"""
return MiddleJson(
pages=list(pages),
is_full_document=True,
file_suffix=file_suffix,
effort="flash",
parse_mode="txt",
@@ -7,7 +7,7 @@ import pytest
from pydantic import ValidationError
import mineru.types as types_module
from mineru.backend.postprocess.pages import model_list_to_pages
from mineru.backend.postprocess.pages import model_json_to_pages
from mineru.types import (
BLOCK_ADAPTER,
BLOCK_TYPES,
@@ -24,6 +24,7 @@ from mineru.types import (
ImageAnnotationBlock,
ListBlock,
MiddleJson,
ModelJson,
PageAuxTextBlock,
PageBlockTypes,
PageInfo,
@@ -184,8 +185,15 @@ def test_formula_number_is_rejected_by_middle_json_boundary_and_schema() -> None
with pytest.raises(ValidationError):
parse_block({"type": RAW_FORMULA_NUMBER, "content": "(1)"})
with pytest.raises(ValidationError):
model_list_to_pages(
[[{"type": RAW_FORMULA_NUMBER, "bbox": [0.7, 0.3, 0.8, 0.4], "content": "(1)"}]]
model_json_to_pages(
ModelJson(
pages=[[{"type": RAW_FORMULA_NUMBER, "bbox": [0.7, 0.3, 0.8, 0.4], "content": "(1)"}]],
page_index_map=[],
file_suffix="pdf",
effort="flash",
parse_mode="txt",
mineru_version="test",
)
)
@@ -438,6 +446,7 @@ def test_middle_json_pdf_requires_top_level_bbox_and_round_trips() -> None:
with pytest.raises(ValidationError, match="requires bbox"):
MiddleJson(
pages=[PageInfo(page_idx=0, blocks=[TextBlock(type="text", index=0, content="x")])],
is_full_document=True,
file_suffix="pdf",
effort="flash",
parse_mode="txt",
@@ -450,6 +459,7 @@ def test_middle_json_pdf_requires_top_level_bbox_and_round_trips() -> None:
blocks=[TextBlock(type="text", index=0, bbox=(0.1, 0.1, 0.9, 0.9), content="x")],
)
],
is_full_document=True,
file_suffix="pdf",
effort="flash",
parse_mode="txt",
@@ -20,6 +20,7 @@ def _middle_json_with_table(body: TableBodyBlock) -> MiddleJson:
table = TableBlock(type="table", index=body.index, content=[body])
return MiddleJson(
pages=[PageInfo(page_idx=3, blocks=[table])],
is_full_document=True,
file_suffix="docx",
effort="flash",
parse_mode="txt",
@@ -53,6 +54,7 @@ def test_equation_export_uses_canonical_sidecar_name(tmp_path: Path) -> None:
)
middle_json = MiddleJson(
pages=[PageInfo(page_idx=3, blocks=[equation])],
is_full_document=True,
file_suffix="docx",
effort="flash",
parse_mode="txt",
+194
View File
@@ -0,0 +1,194 @@
from __future__ import annotations
import pytest
from pydantic import ValidationError
from mineru import ModelJson as PublicModelJson
from mineru.backend.postprocess import document
from mineru.backend.postprocess.document import model_json_to_middle_json
from mineru.config import LLMAidedConfig
from mineru.types import MiddleJson, ModelJson
def _model_json(
*,
pages: list[list[dict[str, object]]] | None = None,
page_index_map: list[int] | None = None,
) -> ModelJson:
"""构造包含全部必填元数据的严格 ModelJson 测试对象。"""
return ModelJson(
pages=pages if pages is not None else [[{"type": "text", "content": "正文"}]],
page_index_map=page_index_map if page_index_map is not None else [],
file_suffix="docx",
effort="flash",
parse_mode="txt",
mineru_version="3.4.0",
)
def test_model_json_is_public_and_serializes_exact_envelope() -> None:
"""验证公开 ModelJson 固定输出六个顶层字段且空映射不会被省略。"""
model_json = _model_json()
payload = model_json.to_dict()
assert PublicModelJson is ModelJson
assert list(payload) == [
"pages",
"page_index_map",
"file_suffix",
"effort",
"parse_mode",
"mineru_version",
]
assert payload == {
"pages": [[{"type": "text", "content": "正文"}]],
"page_index_map": [],
"file_suffix": "docx",
"effort": "flash",
"parse_mode": "txt",
"mineru_version": "3.4.0",
}
assert model_json.is_full_document is True
assert model_json.resolved_page_indices == [0]
assert "is_full_document" not in payload
assert "resolved_page_indices" not in payload
assert ModelJson.model_validate_json(model_json.to_json()) == model_json
def test_non_empty_page_index_map_represents_partial_input() -> None:
"""验证非空页号映射表示显式抽页并与 raw pages 一一对应。"""
model_json = _model_json(pages=[[], []], page_index_map=[3, 5])
assert model_json.page_index_map == [3, 5]
assert model_json.is_full_document is False
assert model_json.resolved_page_indices == [3, 5]
def test_empty_model_json_resolves_to_no_page_indices() -> None:
"""验证空文档仍属于整本解析且不会产生虚构页号。"""
model_json = _model_json(pages=[])
assert model_json.is_full_document is True
assert model_json.resolved_page_indices == []
@pytest.mark.parametrize(
("page_index_map", "message"),
[
([0], "length mismatch"),
([0, 0], "unique"),
([1, 0], "increasing"),
([0, -1], "non-negative"),
],
)
def test_model_json_rejects_invalid_page_index_map(page_index_map: list[int], message: str) -> None:
"""验证显式抽页映射不允许截断、重复、逆序或负数。"""
with pytest.raises(ValidationError, match=message):
_model_json(pages=[[], []], page_index_map=page_index_map)
@pytest.mark.parametrize(
"pages",
["[]", [{}], [[[]]]],
)
def test_model_json_rejects_invalid_page_structure(pages: object) -> None:
"""验证 pages 必须保持页列表、块列表和块字典三层结构。"""
with pytest.raises(ValidationError):
ModelJson(
pages=pages, # type: ignore[arg-type]
page_index_map=[],
file_suffix="pdf",
effort="flash",
parse_mode="txt",
mineru_version="3.4.0",
)
def test_model_json_requires_page_index_map_and_forbids_extra_fields() -> None:
"""验证页映射不可省略且 ModelJson 顶层不接受未声明字段。"""
payload = {
"pages": [],
"file_suffix": "pdf",
"effort": "flash",
"parse_mode": "txt",
"mineru_version": "3.4.0",
}
with pytest.raises(ValidationError, match="page_index_map"):
ModelJson.model_validate(payload)
with pytest.raises(ValidationError, match="extra_forbidden"):
ModelJson.model_validate({**payload, "page_index_map": [], "unexpected": True})
def test_middle_json_requires_and_serializes_full_document_flag() -> None:
"""验证 MiddleJson 将整本语义作为必填字段稳定序列化并往返。"""
middle_json = MiddleJson(
pages=[],
is_full_document=False,
file_suffix="pdf",
effort="flash",
parse_mode="txt",
mineru_version="3.4.0",
)
payload = middle_json.to_dict()
assert list(payload) == [
"pages",
"is_full_document",
"file_suffix",
"effort",
"parse_mode",
"mineru_version",
]
assert payload["is_full_document"] is False
assert MiddleJson.model_validate_json(middle_json.to_json()) == middle_json
def test_model_json_to_middle_json_builds_strict_document_before_pdf_llm(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""验证文档编排完整继承元数据,并让 PDF LLM 直接消费同一 MiddleJson。"""
model_json = ModelJson(
pages=[[]],
page_index_map=[3],
file_suffix="pdf",
effort="xhigh",
parse_mode="ocr",
mineru_version="3.4.0",
)
observed: list[MiddleJson] = []
def fake_llm_postprocess(middle_json: MiddleJson, _config: LLMAidedConfig) -> None:
"""记录 LLM 收到的严格 MiddleJson,并校验抽页语义已写入。"""
assert middle_json.is_full_document is False
assert [page.page_idx for page in middle_json.pages] == [3]
observed.append(middle_json)
monkeypatch.setattr(document, "apply_llm_aided_postprocess", fake_llm_postprocess)
middle_json = model_json_to_middle_json(model_json, llm_aided_config=LLMAidedConfig())
assert observed == [middle_json]
assert middle_json.file_suffix == "pdf"
assert middle_json.effort == "xhigh"
assert middle_json.parse_mode == "ocr"
assert middle_json.mineru_version == "3.4.0"
@pytest.mark.parametrize("invalid_value", [None, 0, 1, "true"])
def test_middle_json_rejects_missing_or_non_boolean_full_document_flag(invalid_value: object) -> None:
"""验证 MiddleJson 不为整本语义提供缺省值或宽松布尔转换。"""
payload = {
"pages": [],
"file_suffix": "pdf",
"effort": "flash",
"parse_mode": "txt",
"mineru_version": "3.4.0",
}
if invalid_value is not None:
payload["is_full_document"] = invalid_value
with pytest.raises(ValidationError, match="is_full_document"):
MiddleJson.model_validate(payload)
@@ -24,7 +24,7 @@ from mineru.render._internal.common.inline import (
inline_plain_text,
parse_inline_content,
)
from mineru.types import BlockType, ContentType, MiddleJson
from mineru.types import BlockType, ContentType, MiddleJson, ModelJson
from mineru.utils.pdf_document import PDFDocument
@@ -215,21 +215,36 @@ def test_doc_analyze_converts_vlm_results_before_downstream_processing(
monkeypatch.setattr(window, "_apply_seal_ocr", MagicMock())
monkeypatch.setattr(window, "_supplement_missing_image_block_containers", MagicMock())
monkeypatch.setattr(window, "_attach_visual_block_images", MagicMock())
monkeypatch.setattr(analyze, "model_list_to_pages", MagicMock(return_value=[]))
expected_middle_json = MiddleJson(
pages=[],
is_full_document=True,
file_suffix="pdf",
effort=effort, # type: ignore[arg-type]
parse_mode=parse_mode, # type: ignore[arg-type]
mineru_version="test",
)
monkeypatch.setattr(analyze, "model_json_to_middle_json", MagicMock(return_value=expected_middle_json))
monkeypatch.setattr(pipeline, "clean_memory", MagicMock())
middle_json, model_list = analyze.doc_analyze(
middle_json, model_json = analyze.doc_analyze(
b"fake-pdf",
effort=effort, # type: ignore[arg-type]
parse_mode=parse_mode, # type: ignore[arg-type]
)
assert type(model_list) is list
assert type(model_list[0]) is list
assert type(model_list[0][0]) is dict
assert model_list == [[{"type": BlockType.PAGE_NUMBER, "bbox": [0.45, 0.9, 0.55, 0.95], "content": "1"}]]
assert isinstance(model_json, ModelJson)
assert type(model_json.pages) is list
assert type(model_json.pages[0]) is list
assert type(model_json.pages[0][0]) is dict
assert model_json.pages == [[{"type": BlockType.PAGE_NUMBER, "bbox": [0.45, 0.9, 0.55, 0.95], "content": "1"}]]
assert model_json.page_index_map == []
assert model_json.file_suffix == "pdf"
assert model_json.effort == effort
assert model_json.parse_mode == parse_mode
assert isinstance(middle_json, MiddleJson)
assert middle_json is expected_middle_json
assert middle_json.pages == []
assert middle_json.is_full_document is True
assert middle_json.effort == effort
assert middle_json.parse_mode == parse_mode
assert type(source_page) is ExtractResult
@@ -16,7 +16,8 @@ from mineru.backend import analyze
from mineru.backend.analysis import office
from mineru.backend.analysis.pdf import constants, formulas, normalization, pipeline, tables, visuals, window
from mineru.types import RAW_ALGORITHM, RAW_CAPTION, RAW_FOOTNOTE, RAW_FORMULA_NUMBER, RAW_PHONETIC
from mineru.types import BlockType, MiddleJson
from mineru.types import BlockType, MiddleJson, ModelJson
from mineru.version import __version__ as mineru_version
JPEG_DATA_URI_PREFIX = "data:image/jpeg;base64,"
@@ -1201,9 +1202,9 @@ def test_aio_doc_analyze_runs_sync_entrypoint_in_thread_and_forwards_arguments(
"""验证异步门面在线程中执行同步入口,并完整转发非默认参数和返回值。"""
caller_thread_id = threading.get_ident()
observed: dict[str, object] = {}
expected_result = (MagicMock(spec=MiddleJson), [[{"type": BlockType.TEXT, "content": "async"}]])
expected_result = (MagicMock(spec=MiddleJson), MagicMock(spec=ModelJson))
def fake_doc_analyze(**kwargs: object) -> tuple[MiddleJson, list[list[dict[str, object]]]]:
def fake_doc_analyze(**kwargs: object) -> tuple[MiddleJson, ModelJson]:
"""记录工作线程及异步门面传入的全部关键字参数。"""
observed["thread_id"] = threading.get_ident()
observed["kwargs"] = kwargs
@@ -1238,7 +1239,8 @@ def test_aio_doc_analyze_propagates_sync_entrypoint_error(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""验证同步入口异常会穿过工作线程并由异步调用方原样收到。"""
def fake_doc_analyze(**_kwargs: object) -> tuple[MiddleJson, list[list[dict[str, object]]]]:
def fake_doc_analyze(**_kwargs: object) -> tuple[MiddleJson, ModelJson]:
"""模拟同步文档分析阶段抛出参数异常。"""
raise ValueError("async analyze failed")
@@ -1249,11 +1251,11 @@ def test_aio_doc_analyze_propagates_sync_entrypoint_error(
@pytest.mark.parametrize("file_suffix", ["docx", "pptx", "xlsx"])
def test_doc_analyze_office_returns_model_list_without_pdf_processing(
def test_doc_analyze_office_returns_model_json_without_pdf_processing(
monkeypatch: pytest.MonkeyPatch,
file_suffix: str,
) -> None:
"""验证三类 Office 文件直接返回模型结果,且不进入任何 PDF 处理阶段。"""
"""验证三类 Office 文件返回严格 ModelJson,且不进入任何 PDF 处理阶段。"""
source_model_list = [[{"type": BlockType.TEXT, "content": "原始 \\(office\\) 内容"}]]
events: list[str] = []
model_factories: dict[str, MagicMock] = {}
@@ -1294,7 +1296,7 @@ def test_doc_analyze_office_returns_model_list_without_pdf_processing(
monkeypatch.setattr(pipeline, "_normalize_pdf_model_list", model_list_normalizer)
monkeypatch.setattr(office.time, "perf_counter", fake_perf_counter)
middle_json, model_list = analyze.doc_analyze(
middle_json, model_json = analyze.doc_analyze(
b"office-bytes",
effort="xhigh",
parse_mode="ocr",
@@ -1306,8 +1308,16 @@ def test_doc_analyze_office_returns_model_list_without_pdf_processing(
assert middle_json.file_suffix == file_suffix
assert middle_json.effort == "flash"
assert middle_json.parse_mode == "txt"
assert model_list is source_model_list
assert model_list[0][0]["content"] == "原始 \\(office\\) 内容"
assert middle_json.is_full_document is True
assert isinstance(model_json, ModelJson)
assert model_json.pages == source_model_list
assert model_json.page_index_map == []
assert model_json.is_full_document is True
assert model_json.file_suffix == file_suffix
assert model_json.effort == "flash"
assert model_json.parse_mode == "txt"
assert model_json.mineru_version == mineru_version
assert model_json.pages[0][0]["content"] == "原始 \\(office\\) 内容"
for suffix, model_factory in model_factories.items():
assert model_factory.call_count == (1 if suffix == file_suffix else 0)
file_stream = selected_model.predict.call_args.args[0]
@@ -1472,7 +1482,7 @@ def test_pdf_infer_timer_excludes_hybrid_vlm_initialization_and_cleanup(
monkeypatch.setattr(pipeline, "_normalize_pdf_model_list", fake_normalize_model_list)
monkeypatch.setattr(pipeline, "clean_memory", fake_clean_memory)
middle_json, model_list = analyze.doc_analyze(
middle_json, model_json = analyze.doc_analyze(
b"empty-pdf",
effort="high",
parse_mode="txt",
@@ -1480,7 +1490,11 @@ def test_pdf_infer_timer_excludes_hybrid_vlm_initialization_and_cleanup(
assert isinstance(middle_json, MiddleJson)
assert middle_json.pages == []
assert model_list == []
assert middle_json.is_full_document is True
assert isinstance(model_json, ModelJson)
assert model_json.pages == []
assert model_json.page_index_map == []
assert model_json.is_full_document is True
assert events == [
"hybrid_init",
"vlm_init",
@@ -1566,7 +1580,7 @@ def test_doc_analyze_office_real_samples(file_suffix: str, expected_page_count:
"""验证统一入口可直接分析三类真实 Office 样例并返回完整分页结果。"""
sample_path = _OFFICE_SAMPLE_DIR / f"{file_suffix}_01.{file_suffix}"
middle_json, model_list = analyze.doc_analyze(
middle_json, model_json = analyze.doc_analyze(
sample_path.read_bytes(),
effort="high",
parse_mode="ocr",
@@ -1574,13 +1588,22 @@ def test_doc_analyze_office_real_samples(file_suffix: str, expected_page_count:
)
assert isinstance(middle_json, MiddleJson)
assert isinstance(model_json, ModelJson)
assert middle_json.is_full_document is True
assert len(middle_json.pages) == expected_page_count
assert all(page.page_idx == page_idx for page_idx, page in enumerate(middle_json.pages))
assert len(model_list) == expected_page_count
assert all(isinstance(page, list) for page in model_list)
assert len(model_json.pages) == expected_page_count
assert all(isinstance(page, list) for page in model_json.pages)
assert model_json.page_index_map == []
assert model_json.file_suffix == file_suffix
assert model_json.effort == "flash"
assert model_json.parse_mode == "txt"
if file_suffix == "docx":
model_equations = [
block for page_model_list in model_list for block in page_model_list if block.get("type") == BlockType.EQUATION
block
for page_model_list in model_json.pages
for block in page_model_list
if block.get("type") == BlockType.EQUATION
]
middle_equations = [block for page in middle_json.pages for block in page.blocks if block.type == BlockType.EQUATION]
assert model_equations
@@ -1592,7 +1615,7 @@ def test_doc_analyze_flash_real_pdf_returns_typed_middle_json() -> None:
"""验证一页真实 PDF 经 Flash Analyze 后返回严格对象且 raw 结果无废弃字段。"""
sample_path = _PROJECT_ROOT / "demo" / "pdfs" / "2407.00079v4_origi-10.pdf"
middle_json, model_list = analyze.doc_analyze(
middle_json, model_json = analyze.doc_analyze(
sample_path.read_bytes(),
effort="flash",
parse_mode="txt",
@@ -1600,17 +1623,25 @@ def test_doc_analyze_flash_real_pdf_returns_typed_middle_json() -> None:
)
assert isinstance(middle_json, MiddleJson)
assert len(middle_json.pages) == len(model_list) == 1
assert isinstance(model_json, ModelJson)
assert middle_json.is_full_document is True
assert len(middle_json.pages) == len(model_json.pages) == 1
assert all(block.bbox is not None for block in middle_json.pages[0].blocks)
assert all("merge_prev" not in block for page in model_list for block in page)
assert all("merge_prev" not in block for page in model_json.pages for block in page)
assert model_json.page_index_map == []
assert model_json.file_suffix == "pdf"
assert model_json.effort == "flash"
assert model_json.parse_mode == "txt"
assert model_json.mineru_version == mineru_version
assert MiddleJson.model_validate_json(middle_json.to_json()) == middle_json
assert ModelJson.model_validate_json(model_json.to_json()) == model_json
def test_doc_analyze_flash_demo1_uses_canonical_equation_type() -> None:
"""验证真实 demo1.pdf 的两层 Flash 输出统一使用 equation 与 LaTeX tag。"""
sample_path = _PROJECT_ROOT / "demo" / "pdfs" / "demo1.pdf"
middle_json, model_list = analyze.doc_analyze(
middle_json, model_json = analyze.doc_analyze(
sample_path.read_bytes(),
effort="flash",
parse_mode="txt",
@@ -1618,32 +1649,27 @@ def test_doc_analyze_flash_demo1_uses_canonical_equation_type() -> None:
)
model_equations = [
block for page_model_list in model_list for block in page_model_list if block.get("type") == BlockType.EQUATION
block for page_model_list in model_json.pages for block in page_model_list if block.get("type") == BlockType.EQUATION
]
middle_equations = [block for page in middle_json.pages for block in page.blocks if block.type == BlockType.EQUATION]
assert middle_json.is_full_document is True
assert model_equations
assert len(middle_equations) == len(model_equations)
assert [block.content for block in middle_equations] == [
block["content"]
for block in model_equations
]
assert [block.content for block in middle_equations] == [block["content"] for block in model_equations]
for formula_number in range(1, 8):
marker = rf"\tag{{{formula_number}}}"
assert sum(marker in block["content"] for block in model_equations) == 1
assert not [
block
for block in model_equations
if any(
block["content"].rstrip().endswith(f"({formula_number})")
for formula_number in range(1, 8)
)
if any(block["content"].rstrip().endswith(f"({formula_number})") for formula_number in range(1, 8))
]
assert "interline_equation" not in middle_json.to_json()
def test_doc_analyze_flash_returns_complete_model_list_and_typed_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
"""验证 Flash 多窗口补充完整 raw model list,并返回严格 MiddleJson。"""
def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
"""验证 Flash 多窗口补充完整 raw pages,并返回严格 ModelJson 与 MiddleJson。"""
from mineru.model import flash as flash_model
events: list[str] = []
@@ -1740,7 +1766,7 @@ def test_doc_analyze_flash_returns_complete_model_list_and_typed_middle_json(mon
monkeypatch.setattr(pipeline.time, "perf_counter", fake_perf_counter)
monkeypatch.setattr(flash_model, "PdfModel", MagicMock(return_value=fake_pdf_model))
middle_json, model_list = analyze.doc_analyze(
middle_json, model_json = analyze.doc_analyze(
b"fake-pdf",
effort="flash",
parse_mode="txt",
@@ -1748,16 +1774,20 @@ def test_doc_analyze_flash_returns_complete_model_list_and_typed_middle_json(mon
)
assert isinstance(middle_json, MiddleJson)
assert isinstance(model_json, ModelJson)
assert middle_json.is_full_document is False
assert [page.page_idx for page in middle_json.pages] == [7, 8, 9]
assert model_list is source_model_list
assert model_json.pages == source_model_list
assert model_json.page_index_map == [7, 8, 9]
assert model_json.is_full_document is False
assert requested_ranges == [(0, 1), (2, 2)]
assert model_list[0][0]["content"] == "第一页 <eq>x+y</eq>"
assert model_list[2][0]["content"] == "第三页 <eq>z</eq>"
assert model_list[2][0]["type"] == BlockType.EQUATION
assert model_json.pages[0][0]["content"] == "第一页 <eq>x+y</eq>"
assert model_json.pages[2][0]["content"] == "第三页 <eq>z</eq>"
assert model_json.pages[2][0]["type"] == BlockType.EQUATION
assert middle_json.pages[2].blocks[0].type == BlockType.EQUATION
assert "interline_equation" not in middle_json.to_json()
assert "image_base64" not in model_list[0][0]
for block in (model_list[1][0], model_list[2][0]):
assert "image_base64" not in model_json.pages[0][0]
for block in (model_json.pages[1][0], model_json.pages[2][0]):
crop_image = _decode_jpeg_data_uri(block["image_base64"])
crop_image.close()
for image in rendered_images:
+20 -3
View File
@@ -4,6 +4,7 @@ import math
from io import BytesIO
from pathlib import Path
from types import SimpleNamespace
from typing import Any
from unittest.mock import MagicMock
from zipfile import ZipFile
@@ -13,7 +14,7 @@ from mineru.backend.analysis.pdf import window
from mineru.backend.analysis.pdf.text import content as text_content
from mineru.backend.analysis.pdf.text.models import _AnalyzeLine, _AnalyzeSpan
from mineru.backend.analysis.pdf.text.native import txt_spans_extract
from mineru.backend.postprocess.pages import model_list_to_pages
from mineru.backend.postprocess.pages import model_json_to_pages
from mineru.model.flash import PdfModel
from mineru.render import render_docx, render_html, render_markdown, render_structured_content
from mineru.types import (
@@ -22,6 +23,7 @@ from mineru.types import (
BlockType,
ContentType,
MiddleJson,
ModelJson,
PageInfo,
)
from mineru.utils.pdf_document import PDFDocument, PDFLinkAnnotation
@@ -39,6 +41,18 @@ from mineru.utils.pdf_text_styles import (
)
def _model_json(pages: list[list[dict[str, Any]]]) -> ModelJson:
"""为 PDF 文本样式渲染测试构造最小严格 ModelJson。"""
return ModelJson(
pages=pages,
page_index_map=[],
file_suffix="pdf",
effort="flash",
parse_mode="txt",
mineru_version="test",
)
def _build_native_text_style_pdf() -> bytes:
"""构造包含字体、删除线、下划线及其反例的单页原生 PDF。"""
@@ -1761,7 +1775,8 @@ def test_flash_native_pdf_styles_reach_model_middle_and_renderers() -> None:
assert '<text style="bold">bold italic sample</text>' in joined_model_content
middle = MiddleJson(
pages=model_list_to_pages(model_list),
pages=model_json_to_pages(_model_json(model_list)),
is_full_document=True,
file_suffix="pdf",
effort="flash",
parse_mode="txt",
@@ -1806,7 +1821,8 @@ def test_demo1_pdf_link_reaches_model_middle_and_all_renderers() -> None:
)
middle = MiddleJson(
pages=model_list_to_pages(model_list),
pages=model_json_to_pages(_model_json(model_list)),
is_full_document=True,
file_suffix="pdf",
effort="flash",
parse_mode="txt",
@@ -1819,6 +1835,7 @@ def test_demo1_pdf_link_reaches_model_middle_and_all_renderers() -> None:
)
link_middle = MiddleJson(
pages=[PageInfo(page_idx=0, blocks=[link_block])],
is_full_document=True,
file_suffix="pdf",
effort="flash",
parse_mode="txt",
+67 -48
View File
@@ -1,15 +1,16 @@
from copy import deepcopy
from typing import Any
import pytest
from mineru.backend.postprocess import pages as pages_module
from mineru.backend.postprocess.lists import fix_office_list_blocks
from mineru.backend.postprocess.pages import model_list_to_pages
from mineru.backend.postprocess.pages import model_json_to_pages
from mineru.types import (
ChartBlock,
DocTitleBlock,
EquationBlock,
ImageBlock,
ListBlock,
ModelJson,
PageInfo,
ParagraphTitleBlock,
RefTextBlock,
@@ -18,7 +19,29 @@ from mineru.types import (
)
def test_model_list_to_pages_keeps_equation_type_without_mutating_input() -> None:
def _model_json(
pages: list[list[dict[str, Any]]],
*,
page_index_map: list[int] | None = None,
) -> ModelJson:
"""为 PageInfo 后处理测试构造最小严格 ModelJson。"""
return ModelJson(
pages=pages,
page_index_map=page_index_map or [],
file_suffix="pdf",
effort="flash",
parse_mode="txt",
mineru_version="test",
)
def test_model_json_to_pages_replaces_raw_model_list_entrypoint() -> None:
"""验证严格 ModelJson 是唯一文档级 PageInfo 转换入口。"""
assert callable(pages_module.model_json_to_pages)
assert not hasattr(pages_module, "model_list_to_pages")
def test_model_json_to_pages_keeps_equation_type_without_mutating_input() -> None:
"""验证 equation 经单页后处理和严格对象化后保持类型、内容与图片载荷。"""
model_list = [
[
@@ -32,7 +55,7 @@ def test_model_list_to_pages_keeps_equation_type_without_mutating_input() -> Non
]
original = deepcopy(model_list)
page = model_list_to_pages(model_list)[0]
page = model_json_to_pages(_model_json(model_list))[0]
assert model_list == original
assert isinstance(page.blocks[0], EquationBlock)
@@ -41,7 +64,7 @@ def test_model_list_to_pages_keeps_equation_type_without_mutating_input() -> Non
assert page.blocks[0].image_base64 == "data:image/jpeg;base64,/9j/2Q=="
def test_model_list_to_pages_returns_typed_pdf_tree_without_mutating_input() -> None:
def test_model_json_to_pages_returns_typed_pdf_tree_without_mutating_input() -> None:
"""验证 PDF raw dict 只在副本上分组,并返回具体 PageInfo/Block 对象。"""
model_list = [
[
@@ -58,7 +81,7 @@ def test_model_list_to_pages_returns_typed_pdf_tree_without_mutating_input() ->
]
original = deepcopy(model_list)
pages = model_list_to_pages(model_list, page_index_map=[7])
pages = model_json_to_pages(_model_json(model_list, page_index_map=[7]))
assert model_list == original
assert isinstance(pages[0], PageInfo)
@@ -72,7 +95,7 @@ def test_model_list_to_pages_returns_typed_pdf_tree_without_mutating_input() ->
assert isinstance(pages[0].blocks[1].content[0], TextBlock)
def test_model_list_to_pages_preserves_recursive_office_list_and_index() -> None:
def test_model_json_to_pages_preserves_recursive_office_list_and_index() -> None:
"""验证 Office list/index 任意深度递归对象化,并清理未匹配目录 anchor。"""
model_list = [
[
@@ -106,7 +129,7 @@ def test_model_list_to_pages_preserves_recursive_office_list_and_index() -> None
]
]
page = model_list_to_pages(model_list)[0]
page = model_json_to_pages(_model_json(model_list))[0]
list_block = page.blocks[0]
index_block = page.blocks[1]
@@ -216,7 +239,7 @@ def test_fix_office_list_blocks_uses_local_ordered_markers_at_each_depth() -> No
)
def test_model_list_to_pages_maps_index_anchor_to_document_title_type_and_level() -> None:
def test_model_json_to_pages_maps_index_anchor_to_document_title_type_and_level() -> None:
"""验证 Office 目录叶子按跨页目标 anchor 继承真实标题类型与层级。"""
model_list = [
[
@@ -245,7 +268,7 @@ def test_model_list_to_pages_maps_index_anchor_to_document_title_type_and_level(
],
]
pages = model_list_to_pages(model_list)
pages = model_json_to_pages(_model_json(model_list))
index = pages[0].blocks[0]
doc_leaf = index.content[0]
section_leaf = index.content[1].content[0]
@@ -263,7 +286,7 @@ def test_model_list_to_pages_maps_index_anchor_to_document_title_type_and_level(
assert "anchor" not in missing_leaf.model_fields_set
def test_model_list_to_pages_uses_first_title_for_duplicate_anchor() -> None:
def test_model_json_to_pages_uses_first_title_for_duplicate_anchor() -> None:
"""验证重复 anchor 按文档顺序使用首个标题目标。"""
model_list = [
[
@@ -276,7 +299,7 @@ def test_model_list_to_pages_uses_first_title_for_duplicate_anchor() -> None:
]
]
page = model_list_to_pages(model_list)[0]
page = model_json_to_pages(_model_json(model_list))[0]
index = page.blocks[2]
assert isinstance(index.content[0], ParagraphTitleBlock)
@@ -298,7 +321,7 @@ def test_office_paragraph_numbering_is_document_wide_and_copy_only() -> None:
]
original = deepcopy(model_list)
pages = model_list_to_pages(model_list)
pages = model_json_to_pages(_model_json(model_list))
titles = [block for page in pages for block in page.blocks]
assert all(isinstance(block, ParagraphTitleBlock) for block in titles)
@@ -327,7 +350,7 @@ def test_office_paragraph_numbering_clears_deeper_levels() -> None:
]
]
titles = model_list_to_pages(model_list)[0].blocks
titles = model_json_to_pages(_model_json(model_list))[0].blocks
assert [title.content for title in titles] == [
"1 A",
@@ -342,7 +365,7 @@ def test_office_paragraph_numbering_clears_deeper_levels() -> None:
def test_visual_body_drops_empty_parent_subtype() -> None:
"""验证 raw visual 的空 subtype 只用于父块判断,不会泄漏到严格 body 模型。"""
page = model_list_to_pages([[{"type": "image", "content": None, "sub_type": None}]])[0]
page = model_json_to_pages(_model_json([[{"type": "image", "content": None, "sub_type": None}]]))[0]
assert isinstance(page.blocks[0], ImageBlock)
assert page.blocks[0].sub_type is None
@@ -352,7 +375,7 @@ def test_visual_body_drops_empty_parent_subtype() -> None:
def test_chart_none_content_is_normalized_to_empty_string() -> None:
"""验证 raw chart 的 null content 在严格对象化前规范为空字符串。"""
page = model_list_to_pages([[{"type": "chart", "content": None}]])[0]
page = model_json_to_pages(_model_json([[{"type": "chart", "content": None}]]))[0]
assert isinstance(page.blocks[0], ChartBlock)
assert page.blocks[0].content[0].type == "chart_body"
@@ -361,14 +384,16 @@ def test_chart_none_content_is_normalized_to_empty_string() -> None:
def test_raw_title_levels_are_normalized_to_global_hierarchy() -> None:
"""验证 raw 标题在严格对象化前归一为全局一至六级层级。"""
page = model_list_to_pages(
[
page = model_json_to_pages(
_model_json(
[
{"type": "doc_title", "content": "Document"},
{"type": "paragraph_title", "content": "Section", "level": 1},
{"type": "paragraph_title", "content": "Deep", "level": 9},
[
{"type": "doc_title", "content": "Document"},
{"type": "paragraph_title", "content": "Section", "level": 1},
{"type": "paragraph_title", "content": "Deep", "level": 9},
]
]
]
)
)[0]
assert [block.level for block in page.blocks] == [1, 2, 6]
@@ -376,8 +401,8 @@ def test_raw_title_levels_are_normalized_to_global_hierarchy() -> None:
def test_pdf_raw_paragraph_title_level_is_clamped_to_six() -> None:
"""验证带 bbox 的 PDF raw 深层标题在严格对象化前归一为六级。"""
page = model_list_to_pages(
[[{"type": "paragraph_title", "content": "Deep", "level": 9, "bbox": [0.1, 0.1, 0.9, 0.2]}]]
page = model_json_to_pages(
_model_json([[{"type": "paragraph_title", "content": "Deep", "level": 9, "bbox": [0.1, 0.1, 0.9, 0.2]}]])
)[0]
assert isinstance(page.blocks[0], ParagraphTitleBlock)
@@ -402,7 +427,7 @@ def test_office_title_level_is_clamped_before_numbering_and_index_mapping() -> N
]
]
page = model_list_to_pages(model_list)[0]
page = model_json_to_pages(_model_json(model_list))[0]
title = page.blocks[0]
index = page.blocks[1]
@@ -432,7 +457,7 @@ def test_pdf_continuation_is_typed_and_line_metadata_is_removed() -> None:
]
]
page = model_list_to_pages(model_list)[0]
page = model_json_to_pages(_model_json(model_list))[0]
assert all(isinstance(block, TextBlock) for block in page.blocks)
assert page.blocks[0].continues_prev is None
@@ -459,7 +484,7 @@ def test_pdf_ref_text_continuation_is_typed_and_line_metadata_is_removed() -> No
]
]
page = model_list_to_pages(model_list)[0]
page = model_json_to_pages(_model_json(model_list))[0]
assert all(isinstance(block, RefTextBlock) for block in page.blocks)
assert page.blocks[0].continues_prev is None
@@ -469,11 +494,13 @@ def test_pdf_ref_text_continuation_is_typed_and_line_metadata_is_removed() -> No
def test_pdf_detection_scans_past_empty_first_page() -> None:
"""验证整份文档 bbox 判定不会把 PDF 空白首页误认为 Office。"""
pages = model_list_to_pages(
[
[],
[{"type": "text", "bbox": [0.1, 0.1, 0.9, 0.3], "content": "later", "lines": []}],
]
pages = model_json_to_pages(
_model_json(
[
[],
[{"type": "text", "bbox": [0.1, 0.1, 0.9, 0.3], "content": "later", "lines": []}],
]
)
)
assert pages[0].blocks == []
@@ -484,23 +511,15 @@ def test_cross_page_table_continuation_remains_raw_postprocess() -> None:
"""验证连续页表格仍在对象化前写入 continues_prev。"""
html_a = "<table><tr><td>A</td><td>B</td></tr></table>"
html_b = "<table><tr><td>C</td><td>D</td></tr></table>"
pages = model_list_to_pages(
[
[{"type": "table", "bbox": [0.1, 0.1, 0.9, 0.85], "content": html_a}],
[{"type": "table", "bbox": [0.1, 0.1, 0.9, 0.85], "content": html_b}],
]
pages = model_json_to_pages(
_model_json(
[
[{"type": "table", "bbox": [0.1, 0.1, 0.9, 0.85], "content": html_a}],
[{"type": "table", "bbox": [0.1, 0.1, 0.9, 0.85], "content": html_b}],
]
)
)
assert isinstance(pages[0].blocks[0], TableBlock)
assert pages[0].blocks[0].continues_prev is None
assert pages[1].blocks[0].continues_prev is True
@pytest.mark.parametrize(
("page_index_map", "message"),
[([0], "length mismatch"), ([0, 0], "unique"), ([1, 0], "increasing"), ([0, -1], "non-negative")],
)
def test_page_index_map_is_strict(page_index_map: list[int], message: str) -> None:
"""验证页号映射不允许截断、重复、逆序或负数。"""
with pytest.raises(ValueError, match=message):
model_list_to_pages([[], []], page_index_map=page_index_map)
+1
View File
@@ -26,6 +26,7 @@ def _middle(*pages: PageInfo) -> MiddleJson:
"""构造无需 PDF bbox 的最小严格 MiddleJson。"""
return MiddleJson(
pages=list(pages),
is_full_document=True,
file_suffix="docx",
effort="flash",
parse_mode="txt",
@@ -31,6 +31,7 @@ def _middle(*pages: PageInfo, file_suffix: str = "docx") -> MiddleJson:
"""构造最小严格 MiddleJson 测试对象。"""
return MiddleJson(
pages=list(pages),
is_full_document=True,
file_suffix=file_suffix,
effort="flash",
parse_mode="txt",
+1 -1
View File
@@ -81,7 +81,7 @@ def _table(
def _page(page_idx: int, blocks: list[dict[str, Any]]) -> dict[str, Any]:
"""构造 model_list_to_pages 输出契约的页面。"""
"""构造 ModelJson 后处理阶段使用的 raw 页面。"""
return {"page_idx": page_idx, "blocks": blocks}