mirror of
https://github.com/opendatalab/MinerU.git
synced 2026-10-28 02:36:45 +08:00
feat: update ModelJson handling and enhance document processing with new properties
This commit is contained in:
@@ -130,7 +130,7 @@ llm_aided:
|
||||
cross_page_table_cell_merge: false
|
||||
```
|
||||
|
||||
- `title_leveling` groups paragraph titles into levels 2 through 6 by document-title boundaries and runs only for whole-PDF input where `page_index_map` is `None`. It is skipped for page-selected input.
|
||||
- `title_leveling` groups paragraph titles into levels 2 through 6 by document-title boundaries and runs only when `MiddleJson.is_full_document` is `true`. Page-selected input is persisted as `false` and skips title leveling.
|
||||
- `cross_page_table_cell_merge` asks the LLM whether each pair of boundary-row cells continues after the existing rules identify a cross-page table.
|
||||
- Table cell merge does not require whole-document input. Both features are disabled by default and share one asynchronous client, one connection configuration, and the `max_concurrency` request limit, which defaults to 16.
|
||||
- `max_concurrency` must be an integer of at least 1 and can be overridden with `MINERU_LLM_AIDED_MAX_CONCURRENCY`.
|
||||
|
||||
+2
-2
@@ -53,8 +53,8 @@ llm_aided:
|
||||
```
|
||||
|
||||
两个功能默认关闭并共享同一套 OpenAI-compatible 连接参数。`title_leveling` 在严格
|
||||
`PageInfo.blocks` 上优化标题层级,但只在 `page_index_map` 为 `None` 的整本 PDF 输入中运行;
|
||||
抽页输入会跳过标题优化。`cross_page_table_cell_merge` 不受整本输入限制,只消费确定性后处理
|
||||
`MiddleJson.pages` 上优化标题层级,但只在 `MiddleJson.is_full_document=true` 的整本 PDF 输入中运行;
|
||||
抽页结果会跳过标题优化。`cross_page_table_cell_merge` 不受整本输入限制,只消费确定性后处理
|
||||
已经写入的 `continues_prev`,再把逐单元格的 `0/1` 结果写入后表根块 `cell_merge`。两个功能
|
||||
通过同一个异步客户端并发请求,`max_concurrency` 是不小于 1 的共享请求上限,默认值为 16。
|
||||
标题分级结果允许 1 到 6,段落标题的 1 会归一为 2,最终严格标题层级范围为 2 到 6。前置
|
||||
|
||||
@@ -4,12 +4,13 @@
|
||||
|
||||
读者: backend 开发者、SDK 开发者、输出开发者
|
||||
|
||||
范围: `doc_analyze()` 第一返回值及 `mineru/types.py` 中的公开 Middle JSON 对象
|
||||
范围: `doc_analyze()` 返回值及 `mineru/types.py` 中的公开 Model/Middle JSON 对象
|
||||
|
||||
## 标准来源
|
||||
|
||||
当前事实标准由以下 Pydantic v2 模型定义:
|
||||
|
||||
- `ModelJson`
|
||||
- `MiddleJson`
|
||||
- `PageInfo`
|
||||
- `Block` discriminated union
|
||||
@@ -19,21 +20,44 @@
|
||||
`model_validate()` / `model_validate_json()`,序列化入口是 `to_dict()` /
|
||||
`to_json()`。不再提供旧 `Block(...)` 通用构造器或 dict-like 兼容接口。
|
||||
|
||||
本轮未迁移的 `ParseResult`、renderer、LLM 标题修正和 Structured Content
|
||||
不属于本文档描述的对象链路。
|
||||
本轮未迁移的 `ParseResult`、renderer 和 Structured Content 不属于本文档描述的对象链路。
|
||||
|
||||
`doc_analyze()` 与 `aio_doc_analyze()` 固定返回
|
||||
`tuple[MiddleJson, ModelJson]`,不再把第二返回值暴露为裸 model list。
|
||||
|
||||
## `ModelJson`
|
||||
|
||||
| 字段 | 类型 | 要求 |
|
||||
|------|------|------|
|
||||
| `pages` | `list[list[dict]]` | 必填,保存 Analyze 产生的 raw blocks |
|
||||
| `page_index_map` | `list[int]` | 必填;空列表表示整本默认顺序,非空时与 pages 等长且唯一递增 |
|
||||
| `file_suffix` | `pdf/docx/pptx/xlsx` | 必填 |
|
||||
| `effort` | `flash/low/medium/high/xhigh` | 必填,使用分析后实际值 |
|
||||
| `parse_mode` | `txt/ocr` | 必填,使用分析后实际值 |
|
||||
| `mineru_version` | 非空字符串 | 必填 |
|
||||
|
||||
`ModelJson.is_full_document` 是由 `page_index_map` 是否为空计算的运行时属性,
|
||||
不进入 JSON;`resolved_page_indices` 同样不序列化,并把空映射展开为顺序页号。
|
||||
任意非空映射均表示显式抽页,不根据映射值猜测是否覆盖整本。
|
||||
|
||||
文档级转换只接受完整 ModelJson:`model_json_to_pages(model_json)` 负责严格页面
|
||||
对象化,`model_json_to_middle_json(model_json, llm_aided_config=...)` 负责构造
|
||||
MiddleJson 并执行适用的 PDF 后处理。不再提供裸 model list 加独立页映射的转换入口。
|
||||
|
||||
## `MiddleJson`
|
||||
|
||||
| 字段 | 类型 | 要求 |
|
||||
|------|------|------|
|
||||
| `pages` | `list[PageInfo]` | 必填,`page_idx` 唯一且严格递增 |
|
||||
| `is_full_document` | `bool` | 必填,保存整本或抽页语义,不提供默认值 |
|
||||
| `file_suffix` | `pdf/docx/pptx/xlsx` | 必填 |
|
||||
| `effort` | `flash/low/medium/high/xhigh` | 必填 |
|
||||
| `parse_mode` | `txt/ocr` | 必填 |
|
||||
| `mineru_version` | 非空字符串 | 必填 |
|
||||
|
||||
当 `file_suffix="pdf"` 时,每个页面顶层 block 必须具有有效 bbox。Office
|
||||
顶层 block 可以没有 bbox。
|
||||
顶层 block 可以没有 bbox。PDF 标题分级直接读取 `is_full_document`;缺少该字段的
|
||||
旧 Middle JSON 无法通过严格解析,需要从 ModelJson 重新生成。
|
||||
|
||||
## `PageInfo`
|
||||
|
||||
|
||||
@@ -130,7 +130,7 @@ llm_aided:
|
||||
cross_page_table_cell_merge: false
|
||||
```
|
||||
|
||||
- `title_leveling`:仅在 `page_index_map` 为 `None` 的整本 PDF 输入中,以文档标题为边界分组优化 2~6 级段落标题;抽页输入会跳过该功能。
|
||||
- `title_leveling`:仅在 `MiddleJson.is_full_document=true` 的整本 PDF 输入中,以文档标题为边界分组优化 2~6 级段落标题;抽页结果持久化为 `false` 并跳过该功能。
|
||||
- `cross_page_table_cell_merge`:在现有规则确认跨页续表后,通过 LLM 判断边界行中各组相邻单元格是否续接。
|
||||
- table cell merge 不要求整本输入;两个功能默认关闭,并通过同一个异步客户端共享连接参数和 `max_concurrency` 请求上限,默认值为 16。
|
||||
- `max_concurrency` 必须是不小于 1 的整数,可通过 `MINERU_LLM_AIDED_MAX_CONCURRENCY` 覆盖。
|
||||
|
||||
+3
-3
@@ -1,6 +1,6 @@
|
||||
# Copyright (c) Opendatalab. All rights reserved.
|
||||
"""MinerU Analyze 与严格 Middle JSON 的最小公开入口。"""
|
||||
"""MinerU Analyze 与严格 Model/Middle JSON 的最小公开入口。"""
|
||||
|
||||
from .types import Block, MiddleJson, PageInfo
|
||||
from .types import Block, MiddleJson, ModelJson, PageInfo
|
||||
|
||||
__all__ = ["Block", "MiddleJson", "PageInfo"]
|
||||
__all__ = ["Block", "MiddleJson", "ModelJson", "PageInfo"]
|
||||
|
||||
+14
-18
@@ -4,17 +4,16 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import Any, Literal, cast
|
||||
from typing import Literal, cast
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from mineru.backend.analysis.contracts import AnalyzeEffort, OfficeSuffix, ParseMode
|
||||
from mineru.backend.analysis.office import analyze_office
|
||||
from mineru.backend.analysis.pdf.pipeline import analyze_pdf
|
||||
from mineru.backend.postprocess.llm_aided import apply_llm_aided_postprocess
|
||||
from mineru.backend.postprocess.pages import model_list_to_pages
|
||||
from mineru.backend.postprocess.document import model_json_to_middle_json
|
||||
from mineru.config import config
|
||||
from mineru.types import MiddleJson
|
||||
from mineru.types import MiddleJson, ModelJson
|
||||
from mineru.version import __version__ as mineru_version
|
||||
|
||||
_SUPPORTED_FILE_SUFFIXES = {"pdf", "docx", "pptx", "xlsx"}
|
||||
@@ -36,8 +35,8 @@ def doc_analyze(
|
||||
image_analysis: bool = True,
|
||||
page_index_map: list[int] | None = None,
|
||||
file_suffix: Literal["pdf", "docx", "pptx", "xlsx"] = "pdf",
|
||||
) -> tuple[MiddleJson, list[list[dict[str, Any]]]]:
|
||||
"""生产 model-list,并在统一边界构造严格 MiddleJson。"""
|
||||
) -> tuple[MiddleJson, ModelJson]:
|
||||
"""生产严格 ModelJson,并在统一边界构造严格 MiddleJson。"""
|
||||
if file_suffix not in _SUPPORTED_FILE_SUFFIXES:
|
||||
raise ValueError(f"Unsupported file suffix: {file_suffix!r}")
|
||||
|
||||
@@ -52,22 +51,19 @@ def doc_analyze(
|
||||
result = analyze_office(file_bytes, cast(OfficeSuffix, file_suffix))
|
||||
|
||||
_log_infer_performance(file_suffix, len(result.model_list), result.elapsed)
|
||||
pages = model_list_to_pages(result.model_list, page_index_map)
|
||||
if file_suffix == "pdf":
|
||||
apply_llm_aided_postprocess(
|
||||
pages,
|
||||
config.llm_aided,
|
||||
page_index_map=page_index_map,
|
||||
)
|
||||
|
||||
middle_json = MiddleJson(
|
||||
pages=pages,
|
||||
model_json = ModelJson(
|
||||
pages=result.model_list,
|
||||
page_index_map=page_index_map or [],
|
||||
file_suffix=file_suffix,
|
||||
effort=result.effort,
|
||||
parse_mode=result.parse_mode,
|
||||
mineru_version=mineru_version,
|
||||
)
|
||||
return middle_json, result.model_list
|
||||
middle_json = model_json_to_middle_json(
|
||||
model_json,
|
||||
llm_aided_config=config.llm_aided,
|
||||
)
|
||||
return middle_json, model_json
|
||||
|
||||
|
||||
async def aio_doc_analyze(
|
||||
@@ -77,7 +73,7 @@ async def aio_doc_analyze(
|
||||
image_analysis: bool = True,
|
||||
page_index_map: list[int] | None = None,
|
||||
file_suffix: Literal["pdf", "docx", "pptx", "xlsx"] = "pdf",
|
||||
) -> tuple[MiddleJson, list[list[dict[str, Any]]]]:
|
||||
) -> tuple[MiddleJson, ModelJson]:
|
||||
"""在线程中执行统一文档分析,避免阻塞调用方事件循环。"""
|
||||
return await asyncio.to_thread(
|
||||
doc_analyze,
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
# Copyright (c) Opendatalab. All rights reserved.
|
||||
"""将 raw model-list 转换为严格 Middle JSON 页对象的后处理层。"""
|
||||
"""将严格 ModelJson 转换并增强为 MiddleJson 的后处理层。"""
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
# Copyright (c) Opendatalab. All rights reserved.
|
||||
"""严格 ModelJson 到 MiddleJson 的文档级后处理编排。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from mineru.config import LLMAidedConfig
|
||||
from mineru.types import MiddleJson, ModelJson
|
||||
|
||||
from .llm_aided import apply_llm_aided_postprocess
|
||||
from .pages import model_json_to_pages
|
||||
|
||||
|
||||
def model_json_to_middle_json(
|
||||
model_json: ModelJson,
|
||||
*,
|
||||
llm_aided_config: LLMAidedConfig,
|
||||
) -> MiddleJson:
|
||||
"""从严格 ModelJson 构造 MiddleJson,并执行适用的文档级增强。"""
|
||||
middle_json = MiddleJson(
|
||||
pages=model_json_to_pages(model_json),
|
||||
is_full_document=model_json.is_full_document,
|
||||
file_suffix=model_json.file_suffix,
|
||||
effort=model_json.effort,
|
||||
parse_mode=model_json.parse_mode,
|
||||
mineru_version=model_json.mineru_version,
|
||||
)
|
||||
if model_json.file_suffix == "pdf":
|
||||
apply_llm_aided_postprocess(middle_json, llm_aided_config)
|
||||
return middle_json
|
||||
@@ -8,7 +8,7 @@ import asyncio
|
||||
from loguru import logger
|
||||
|
||||
from mineru.config import LLMAidedConfig
|
||||
from mineru.types import PageInfo
|
||||
from mineru.types import MiddleJson
|
||||
|
||||
from .llm_client import LLMAidedClient
|
||||
from .table_merge.llm_cell_merge import apply_llm_cross_page_cell_merge
|
||||
@@ -17,18 +17,18 @@ from .title_leveling import apply_llm_title_leveling
|
||||
|
||||
def _resolve_enabled_features(
|
||||
config: LLMAidedConfig,
|
||||
page_index_map: list[int] | None,
|
||||
middle_json: MiddleJson,
|
||||
) -> tuple[bool, bool]:
|
||||
"""根据整本输入约束解析本次实际启用的标题与表格功能。"""
|
||||
title_enabled = config.features.title_leveling and page_index_map is None
|
||||
title_enabled = config.features.title_leveling and middle_json.is_full_document
|
||||
table_enabled = config.features.cross_page_table_cell_merge
|
||||
if config.features.title_leveling and page_index_map is not None:
|
||||
logger.info("Skipping LLM title leveling because page_index_map is not None")
|
||||
if config.features.title_leveling and not middle_json.is_full_document:
|
||||
logger.info("Skipping LLM title leveling because the input is not a full document")
|
||||
return title_enabled, table_enabled
|
||||
|
||||
|
||||
async def _apply_llm_aided_postprocess(
|
||||
pages: list[PageInfo],
|
||||
middle_json: MiddleJson,
|
||||
config: LLMAidedConfig,
|
||||
*,
|
||||
title_enabled: bool,
|
||||
@@ -40,9 +40,9 @@ async def _apply_llm_aided_postprocess(
|
||||
try:
|
||||
tasks = []
|
||||
if title_enabled:
|
||||
tasks.append(apply_llm_title_leveling(pages, resolved_client))
|
||||
tasks.append(apply_llm_title_leveling(middle_json.pages, resolved_client))
|
||||
if table_enabled:
|
||||
tasks.append(apply_llm_cross_page_cell_merge(pages, resolved_client))
|
||||
tasks.append(apply_llm_cross_page_cell_merge(middle_json.pages, resolved_client))
|
||||
await asyncio.gather(*tasks)
|
||||
finally:
|
||||
if client is None:
|
||||
@@ -50,20 +50,19 @@ async def _apply_llm_aided_postprocess(
|
||||
|
||||
|
||||
def apply_llm_aided_postprocess(
|
||||
pages: list[PageInfo],
|
||||
middle_json: MiddleJson,
|
||||
config: LLMAidedConfig,
|
||||
*,
|
||||
page_index_map: list[int] | None = None,
|
||||
client: LLMAidedClient | None = None,
|
||||
) -> None:
|
||||
"""按配置和整本输入约束同步桥接异步 LLM 后处理。"""
|
||||
title_enabled, table_enabled = _resolve_enabled_features(config, page_index_map)
|
||||
title_enabled, table_enabled = _resolve_enabled_features(config, middle_json)
|
||||
if not title_enabled and not table_enabled:
|
||||
return
|
||||
|
||||
asyncio.run(
|
||||
_apply_llm_aided_postprocess(
|
||||
pages,
|
||||
middle_json,
|
||||
config,
|
||||
title_enabled=title_enabled,
|
||||
table_enabled=table_enabled,
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# Copyright (c) Opendatalab. All rights reserved.
|
||||
"""raw model-list 到严格 PageInfo 列表的唯一转换边界。"""
|
||||
"""严格 ModelJson 到 PageInfo 列表的唯一转换边界。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -12,29 +12,11 @@ from mineru.backend.postprocess.lists import fix_office_index_title_blocks, fix_
|
||||
from mineru.backend.postprocess.page_blocks import process_page_blocks
|
||||
from mineru.backend.postprocess.paragraphs import merge_para_text_blocks
|
||||
from mineru.backend.postprocess.table_merge import merge_table
|
||||
from mineru.types import PageInfo
|
||||
from mineru.types import ModelJson, PageInfo
|
||||
|
||||
PAGE_INFO_LIST_ADAPTER = TypeAdapter(list[PageInfo])
|
||||
|
||||
|
||||
def _validate_page_index_map(
|
||||
page_count: int,
|
||||
page_index_map: list[int] | None,
|
||||
) -> list[int]:
|
||||
"""校验实际页号映射,禁止 zip 截断、重复页号或逆序页号。"""
|
||||
if page_index_map is None:
|
||||
return list(range(page_count))
|
||||
if len(page_index_map) != page_count:
|
||||
raise ValueError(f"page_index_map length mismatch: pages={page_count}, mapping={len(page_index_map)}")
|
||||
if any(isinstance(page_idx, bool) or not isinstance(page_idx, int) or page_idx < 0 for page_idx in page_index_map):
|
||||
raise ValueError("page_index_map values must be non-negative integers")
|
||||
if len(page_index_map) != len(set(page_index_map)):
|
||||
raise ValueError("page_index_map values must be unique")
|
||||
if any(current <= previous for previous, current in zip(page_index_map, page_index_map[1:])):
|
||||
raise ValueError("page_index_map values must preserve strictly increasing order")
|
||||
return list(page_index_map)
|
||||
|
||||
|
||||
def _document_uses_bbox(model_list: list[list[dict[str, Any]]]) -> bool:
|
||||
"""按整份文档是否出现 bbox 判定 PDF/Office,避免空白首页误判。"""
|
||||
return any(
|
||||
@@ -87,13 +69,10 @@ def blocks_to_page_info(
|
||||
return PageInfo.model_validate(raw_page)
|
||||
|
||||
|
||||
def model_list_to_pages(
|
||||
model_list: list[list[dict[str, Any]]],
|
||||
page_index_map: list[int] | None = None,
|
||||
) -> list[PageInfo]:
|
||||
"""在 raw 后处理结束后,一次性构造严格且可递归序列化的 PageInfo。"""
|
||||
page_indices = _validate_page_index_map(len(model_list), page_index_map)
|
||||
copied_model_list = deepcopy(model_list)
|
||||
def model_json_to_pages(model_json: ModelJson) -> list[PageInfo]:
|
||||
"""从严格 ModelJson 无副作用地构造可递归序列化的 PageInfo。"""
|
||||
page_indices = model_json.resolved_page_indices
|
||||
copied_model_list = deepcopy(model_json.pages)
|
||||
use_bbox = _document_uses_bbox(copied_model_list)
|
||||
if not use_bbox:
|
||||
fix_office_paragraph_titles(copied_model_list)
|
||||
|
||||
@@ -315,7 +315,7 @@ class PdfVlmParser(PdfHybridParser):
|
||||
"""将旧 PdfVlmParser 构造参数归一到 Hybrid extra_high,避免继续暴露独立 VLM backend。"""
|
||||
backend = kwargs.pop("backend", "vlm-engine")
|
||||
kwargs.pop("effort", None)
|
||||
resolved_backend, resolved_effort = resolve_backend_and_effort(backend, MAX_HYBRID_EFFORT)
|
||||
resolved_=backend, resolved_effort = resolve_backend_and_effort(backend, MAX_HYBRID_EFFORT)
|
||||
super().__init__(backend=resolved_backend, effort=resolved_effort, **kwargs)
|
||||
|
||||
|
||||
|
||||
+40
-1
@@ -304,7 +304,7 @@ def _remove_block_fields(value: Any, excluded_fields: set[str]) -> Any:
|
||||
|
||||
|
||||
class _StrictMiddleModel(BaseModel):
|
||||
"""Middle JSON 严格模型基类,提供无副作用的统一序列化入口。"""
|
||||
"""Model/Middle JSON 严格模型基类,提供无副作用的统一序列化入口。"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid", strict=True, validate_assignment=True)
|
||||
|
||||
@@ -659,6 +659,44 @@ def _iter_child_blocks(block: BlockBase) -> list[BlockBase]:
|
||||
return [child for child in content if isinstance(child, BlockBase)]
|
||||
|
||||
|
||||
class ModelJson(_StrictMiddleModel):
|
||||
"""Analyze 返回的完整严格 Model JSON 对象。"""
|
||||
|
||||
pages: list[list[dict[str, Any]]]
|
||||
page_index_map: list[int]
|
||||
file_suffix: Literal["pdf", "docx", "pptx", "xlsx"]
|
||||
effort: Literal["flash", "low", "medium", "high", "xhigh"]
|
||||
parse_mode: Literal["txt", "ocr"]
|
||||
mineru_version: str = Field(min_length=1)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_page_index_map(self) -> ModelJson:
|
||||
"""校验显式抽页映射,并保留空列表作为整本解析标记。"""
|
||||
if not self.page_index_map:
|
||||
return self
|
||||
if len(self.page_index_map) != len(self.pages):
|
||||
raise ValueError(f"page_index_map length mismatch: pages={len(self.pages)}, mapping={len(self.page_index_map)}")
|
||||
if any(page_idx < 0 for page_idx in self.page_index_map):
|
||||
raise ValueError("page_index_map values must be non-negative integers")
|
||||
if len(self.page_index_map) != len(set(self.page_index_map)):
|
||||
raise ValueError("page_index_map values must be unique")
|
||||
if any(current <= previous for previous, current in zip(self.page_index_map, self.page_index_map[1:])):
|
||||
raise ValueError("page_index_map values must preserve strictly increasing order")
|
||||
return self
|
||||
|
||||
@property
|
||||
def is_full_document(self) -> bool:
|
||||
"""返回当前 Model JSON 是否表示整本文档解析。"""
|
||||
return not self.page_index_map
|
||||
|
||||
@property
|
||||
def resolved_page_indices(self) -> list[int]:
|
||||
"""返回显式抽页映射或整本文档的顺序页号副本。"""
|
||||
if self.is_full_document:
|
||||
return list(range(len(self.pages)))
|
||||
return list(self.page_index_map)
|
||||
|
||||
|
||||
class PageInfo(_StrictMiddleModel):
|
||||
"""一页的严格 Middle JSON 内容。"""
|
||||
|
||||
@@ -691,6 +729,7 @@ class MiddleJson(_StrictMiddleModel):
|
||||
"""Analyze 返回的完整严格 Middle JSON 对象。"""
|
||||
|
||||
pages: list[PageInfo]
|
||||
is_full_document: bool
|
||||
file_suffix: Literal["pdf", "docx", "pptx", "xlsx"]
|
||||
effort: Literal["flash", "low", "medium", "high", "xhigh"]
|
||||
parse_mode: Literal["txt", "ocr"]
|
||||
|
||||
@@ -1,23 +1,39 @@
|
||||
from mineru.backend.postprocess.pages import model_list_to_pages
|
||||
from mineru.types import BlockType, ImageBlock
|
||||
from typing import Any
|
||||
|
||||
from mineru.backend.postprocess.pages import model_json_to_pages
|
||||
from mineru.types import BlockType, ImageBlock, ModelJson
|
||||
|
||||
|
||||
def _model_json(pages: list[list[dict[str, Any]]]) -> ModelJson:
|
||||
"""为 raw block 对象化回归构造最小严格 ModelJson。"""
|
||||
return ModelJson(
|
||||
pages=pages,
|
||||
page_index_map=[],
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="test",
|
||||
)
|
||||
|
||||
|
||||
def test_postprocess_groups_pdf_visual_blocks_without_dict_access() -> None:
|
||||
"""验证 PDF raw 视觉块会在严格对象化边界完成分组。"""
|
||||
page = model_list_to_pages(
|
||||
[
|
||||
page = model_json_to_pages(
|
||||
_model_json(
|
||||
[
|
||||
{
|
||||
"type": BlockType.IMAGE_CAPTION,
|
||||
"bbox": [0.05, 0.05, 0.6, 0.12],
|
||||
"content": "Figure 1",
|
||||
},
|
||||
{
|
||||
"type": BlockType.IMAGE,
|
||||
"bbox": [0.05, 0.15, 0.6, 0.45],
|
||||
},
|
||||
[
|
||||
{
|
||||
"type": BlockType.IMAGE_CAPTION,
|
||||
"bbox": [0.05, 0.05, 0.6, 0.12],
|
||||
"content": "Figure 1",
|
||||
},
|
||||
{
|
||||
"type": BlockType.IMAGE,
|
||||
"bbox": [0.05, 0.15, 0.6, 0.45],
|
||||
},
|
||||
]
|
||||
]
|
||||
]
|
||||
)
|
||||
)[0]
|
||||
|
||||
assert len(page.blocks) == 1
|
||||
@@ -30,13 +46,15 @@ def test_postprocess_groups_pdf_visual_blocks_without_dict_access() -> None:
|
||||
|
||||
def test_postprocess_groups_raw_vision_footnote_as_image_footnote() -> None:
|
||||
"""验证通用 raw footnote 会归入相邻图片而不是页脚注。"""
|
||||
page = model_list_to_pages(
|
||||
[
|
||||
page = model_json_to_pages(
|
||||
_model_json(
|
||||
[
|
||||
{"type": BlockType.IMAGE, "bbox": [0.1, 0.1, 0.6, 0.45]},
|
||||
{"type": "footnote", "bbox": [0.12, 0.47, 0.6, 0.6], "content": "Figure note"},
|
||||
[
|
||||
{"type": BlockType.IMAGE, "bbox": [0.1, 0.1, 0.6, 0.45]},
|
||||
{"type": "footnote", "bbox": [0.12, 0.47, 0.6, 0.6], "content": "Figure note"},
|
||||
]
|
||||
]
|
||||
]
|
||||
)
|
||||
)[0]
|
||||
|
||||
assert len(page.blocks) == 1
|
||||
|
||||
@@ -38,6 +38,7 @@ def _middle(*pages: PageInfo) -> MiddleJson:
|
||||
"""构造最小严格 MiddleJson 测试对象。"""
|
||||
return MiddleJson(
|
||||
pages=list(pages),
|
||||
is_full_document=True,
|
||||
file_suffix="docx",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
|
||||
@@ -10,7 +10,7 @@ from typing import Any
|
||||
from pypdf import PdfReader
|
||||
|
||||
from mineru.backend.postprocess.page_blocks import process_page_blocks
|
||||
from mineru.backend.postprocess.pages import model_list_to_pages
|
||||
from mineru.backend.postprocess.pages import model_json_to_pages
|
||||
from mineru.model.flash import PdfModel
|
||||
from mineru.model.flash.native_pdf import (
|
||||
formulas,
|
||||
@@ -26,7 +26,7 @@ from mineru.render._internal.common.inline import (
|
||||
inline_plain_text,
|
||||
parse_inline_content,
|
||||
)
|
||||
from mineru.types import MiddleJson
|
||||
from mineru.types import MiddleJson, ModelJson
|
||||
from mineru.utils.pdf_document import PDFDocument, get_lines_from_chars
|
||||
|
||||
|
||||
@@ -38,6 +38,22 @@ _CJK_SYNTHETIC_PDF_NAME = "native_cjk_layout_synthetic.pdf"
|
||||
_SAFE_WATERMARK_TEXT = "MINERU TEST WATERMARK"
|
||||
|
||||
|
||||
def _model_json(
|
||||
pages: list[list[dict[str, Any]]],
|
||||
*,
|
||||
page_index_map: list[int] | None = None,
|
||||
) -> ModelJson:
|
||||
"""为 Flash 样例后处理构造最小严格 ModelJson。"""
|
||||
return ModelJson(
|
||||
pages=pages,
|
||||
page_index_map=page_index_map or [],
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="test",
|
||||
)
|
||||
|
||||
|
||||
# demo4 第 9 至 11 页中实际跨物理行显示的 URL,用于确保纯正则拼接覆盖全部形态。
|
||||
_DEMO4_WRAPPED_URLS = (
|
||||
"https://doi.org/10.1007/s00259-025-07388-8",
|
||||
@@ -434,7 +450,7 @@ def test_demo1_keeps_five_real_tables_without_formula_false_positive() -> None:
|
||||
assert sum(block["type"] == "equation" for page in model_list for block in page) == 7
|
||||
assert sum(block["type"] == "caption" for page in model_list for block in page) == 10
|
||||
assert sum(block["type"] == "footnote" for page in model_list for block in page) == 5
|
||||
middle_pages = model_list_to_pages(model_list)
|
||||
middle_pages = model_json_to_pages(_model_json(model_list))
|
||||
page3_continuation = next(
|
||||
block
|
||||
for block in middle_pages[2].blocks
|
||||
@@ -1053,7 +1069,8 @@ def test_demo3_pages6_7_and10_fix_caption_inline_titles_and_reference_tail() ->
|
||||
]
|
||||
page7_markdown = render_markdown(
|
||||
MiddleJson(
|
||||
pages=model_list_to_pages([page7], page_index_map=[6]),
|
||||
pages=model_json_to_pages(_model_json([page7], page_index_map=[6])),
|
||||
is_full_document=False,
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
@@ -1943,7 +1960,7 @@ def test_frozen_soil_reference_tails_remain_single_text_blocks() -> None:
|
||||
"""验证中文论文双语图注独立标记,正文图引用和参考文献保持 text。"""
|
||||
|
||||
model_list = _auto_model_list("中文论文.pdf")
|
||||
middle_pages = model_list_to_pages(model_list)
|
||||
middle_pages = model_json_to_pages(_model_json(model_list))
|
||||
page2_continuation = next(
|
||||
block
|
||||
for block in middle_pages[1].blocks
|
||||
|
||||
@@ -37,6 +37,7 @@ def _middle(*pages: PageInfo) -> MiddleJson:
|
||||
"""构造无需 PDF bbox 的严格 Office MiddleJson。"""
|
||||
return MiddleJson(
|
||||
pages=list(pages),
|
||||
is_full_document=True,
|
||||
file_suffix="docx",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
|
||||
@@ -209,6 +209,18 @@ def _merged_row_texts(middle_json: MiddleJson) -> list[list[str]]:
|
||||
return [[cell.get_text() for cell in row.find_all(["td", "th"])] for row in soup.find_all("tr")]
|
||||
|
||||
|
||||
def _middle_json(pages: list[PageInfo], *, is_full_document: bool = True) -> MiddleJson:
|
||||
"""为 LLM 后处理单元测试构造最小严格 MiddleJson。"""
|
||||
return MiddleJson(
|
||||
pages=pages,
|
||||
is_full_document=is_full_document,
|
||||
file_suffix="docx",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="test",
|
||||
)
|
||||
|
||||
|
||||
def test_llm_client_injects_optional_thinking_parameter() -> None:
|
||||
"""验证显式 enable_thinking 通过 extra_body 发送且返回经过结构校验。"""
|
||||
completions = _FakeCompletions(['</think>{"0": 2}'])
|
||||
@@ -520,6 +532,7 @@ def test_mixed_cell_merge_flows_through_strict_middle_json_and_renderer() -> Non
|
||||
asyncio.run(apply_llm_cross_page_cell_merge(pages, client)) # type: ignore[arg-type]
|
||||
middle_json = MiddleJson(
|
||||
pages=pages,
|
||||
is_full_document=True,
|
||||
file_suffix="pdf",
|
||||
effort="high",
|
||||
parse_mode="txt",
|
||||
@@ -555,9 +568,10 @@ def test_postprocess_reuses_one_client_for_both_features() -> None:
|
||||
table_pages[1],
|
||||
]
|
||||
client = _QueuedValidatedClient([{"0": 3}, [0, 1]])
|
||||
middle_json = _middle_json(pages)
|
||||
|
||||
apply_llm_aided_postprocess(
|
||||
pages,
|
||||
middle_json,
|
||||
_llm_config(title=True, table=True),
|
||||
client=client, # type: ignore[arg-type]
|
||||
)
|
||||
@@ -606,9 +620,10 @@ def test_title_groups_and_table_candidates_run_concurrently() -> None:
|
||||
PageInfo(page_idx=3, blocks=second_pair[1].blocks),
|
||||
]
|
||||
client = _ConcurrentValidatedClient(expected_calls=4)
|
||||
middle_json = _middle_json(pages)
|
||||
|
||||
apply_llm_aided_postprocess(
|
||||
pages,
|
||||
middle_json,
|
||||
_llm_config(title=True, table=True),
|
||||
client=client, # type: ignore[arg-type]
|
||||
)
|
||||
@@ -627,7 +642,7 @@ def test_disabled_postprocess_does_not_construct_client(monkeypatch: pytest.Monk
|
||||
|
||||
monkeypatch.setattr(llm_aided, "LLMAidedClient", lambda _config: pytest.fail("client must not be created"))
|
||||
|
||||
apply_llm_aided_postprocess([], LLMAidedConfig())
|
||||
apply_llm_aided_postprocess(_middle_json([]), LLMAidedConfig())
|
||||
|
||||
|
||||
def test_partial_input_skips_title_without_constructing_client(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
@@ -636,20 +651,26 @@ def test_partial_input_skips_title_without_constructing_client(monkeypatch: pyte
|
||||
|
||||
monkeypatch.setattr(llm_aided, "LLMAidedClient", lambda _config: pytest.fail("client must not be created"))
|
||||
|
||||
apply_llm_aided_postprocess([], _llm_config(title=True), page_index_map=[])
|
||||
apply_llm_aided_postprocess(_middle_json([], is_full_document=False), _llm_config(title=True))
|
||||
|
||||
|
||||
def test_partial_input_skips_title_but_keeps_table_cell_merge() -> None:
|
||||
"""验证抽页输入不会触发标题请求,但仍会执行跨页单元格分析。"""
|
||||
pages = _cross_page_table_pages()
|
||||
pages[0].blocks[0] = pages[0].blocks[0].model_copy( # type: ignore[union-attr]
|
||||
update={
|
||||
"index": 1,
|
||||
"content": [pages[0].blocks[0].content[0].model_copy(update={"index": 1})], # type: ignore[union-attr]
|
||||
}
|
||||
)
|
||||
title = ParagraphTitleBlock(type=BlockType.PARAGRAPH_TITLE, index=0, content="Section", level=2)
|
||||
pages[0].blocks.insert(0, title)
|
||||
client = _QueuedValidatedClient([[0, 1]])
|
||||
middle_json = _middle_json(pages, is_full_document=False)
|
||||
|
||||
apply_llm_aided_postprocess(
|
||||
pages,
|
||||
middle_json,
|
||||
_llm_config(title=True, table=True),
|
||||
page_index_map=[10, 11],
|
||||
client=client, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
@@ -665,9 +686,14 @@ def test_front_vlm_cell_merge_detection_remains_disabled() -> None:
|
||||
assert "enable_cross_page_table_merge=False" in runtime_path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_doc_analyze_runs_llm_after_deterministic_table_detection(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""验证 PDF 主链路先产生 continues_prev,再执行后置 LLM 并构造 MiddleJson。"""
|
||||
@pytest.mark.parametrize("page_index_map", [None, []])
|
||||
def test_doc_analyze_runs_llm_after_deterministic_table_detection(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
page_index_map: list[int] | None,
|
||||
) -> None:
|
||||
"""验证默认或空映射的整本 PDF 均先检测续表,再执行后置 LLM。"""
|
||||
from mineru.backend import analyze
|
||||
from mineru.backend.postprocess import document
|
||||
|
||||
previous_html = "<table><tr><th>H1</th><th>H2</th></tr><tr><td>A</td><td>X</td></tr></table>"
|
||||
current_html = "<table><tr><th>H1</th><th>H2</th></tr><tr><td>B</td><td>Y</td></tr></table>"
|
||||
@@ -683,14 +709,12 @@ def test_doc_analyze_runs_llm_after_deterministic_table_detection(monkeypatch: p
|
||||
calls: list[str] = []
|
||||
|
||||
def fake_llm_postprocess(
|
||||
pages: list[PageInfo],
|
||||
middle_json: MiddleJson,
|
||||
_config: LLMAidedConfig,
|
||||
*,
|
||||
page_index_map: list[int] | None,
|
||||
) -> None:
|
||||
"""断言确定性续表标记已存在,并模拟新增 cell_merge。"""
|
||||
assert page_index_map is None
|
||||
current_table = pages[1].blocks[0]
|
||||
assert middle_json.is_full_document is True
|
||||
current_table = middle_json.pages[1].blocks[0]
|
||||
assert isinstance(current_table, TableBlock)
|
||||
assert current_table.continues_prev is True
|
||||
assert current_table.cell_merge is None
|
||||
@@ -698,19 +722,23 @@ def test_doc_analyze_runs_llm_after_deterministic_table_detection(monkeypatch: p
|
||||
calls.append("llm")
|
||||
|
||||
monkeypatch.setattr(analyze, "analyze_pdf", lambda *_args, **_kwargs: result)
|
||||
monkeypatch.setattr(analyze, "apply_llm_aided_postprocess", fake_llm_postprocess)
|
||||
monkeypatch.setattr(document, "apply_llm_aided_postprocess", fake_llm_postprocess)
|
||||
|
||||
middle_json, _ = analyze.doc_analyze(b"pdf")
|
||||
middle_json, model_json = analyze.doc_analyze(b"pdf", page_index_map=page_index_map)
|
||||
|
||||
current_table = middle_json.pages[1].blocks[0]
|
||||
assert isinstance(current_table, TableBlock)
|
||||
assert current_table.cell_merge == [1, 0]
|
||||
assert model_json.page_index_map == []
|
||||
assert model_json.is_full_document is True
|
||||
assert middle_json.is_full_document is True
|
||||
assert calls == ["llm"]
|
||||
|
||||
|
||||
def test_doc_analyze_does_not_run_llm_for_office(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""验证 Office 统一入口不会进入仅限 PDF 的 LLM 后处理。"""
|
||||
from mineru.backend import analyze
|
||||
from mineru.backend.postprocess import document
|
||||
|
||||
result = AnalysisResult(
|
||||
model_list=[[{"type": BlockType.PARAGRAPH_TITLE, "content": "Slide", "level": 2}]],
|
||||
@@ -720,9 +748,10 @@ def test_doc_analyze_does_not_run_llm_for_office(monkeypatch: pytest.MonkeyPatch
|
||||
)
|
||||
calls: list[str] = []
|
||||
monkeypatch.setattr(analyze, "analyze_office", lambda *_args, **_kwargs: result)
|
||||
monkeypatch.setattr(analyze, "apply_llm_aided_postprocess", lambda *_args, **_kwargs: calls.append("llm"))
|
||||
monkeypatch.setattr(document, "apply_llm_aided_postprocess", lambda *_args, **_kwargs: calls.append("llm"))
|
||||
|
||||
middle_json, _ = analyze.doc_analyze(b"office", file_suffix="pptx")
|
||||
|
||||
assert middle_json.file_suffix == "pptx"
|
||||
assert middle_json.is_full_document is True
|
||||
assert calls == []
|
||||
|
||||
@@ -33,6 +33,7 @@ def _middle(*pages: PageInfo, file_suffix: str = "docx") -> MiddleJson:
|
||||
"""构造最小严格 MiddleJson 测试对象。"""
|
||||
return MiddleJson(
|
||||
pages=list(pages),
|
||||
is_full_document=True,
|
||||
file_suffix=file_suffix,
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
|
||||
@@ -7,7 +7,7 @@ import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
import mineru.types as types_module
|
||||
from mineru.backend.postprocess.pages import model_list_to_pages
|
||||
from mineru.backend.postprocess.pages import model_json_to_pages
|
||||
from mineru.types import (
|
||||
BLOCK_ADAPTER,
|
||||
BLOCK_TYPES,
|
||||
@@ -24,6 +24,7 @@ from mineru.types import (
|
||||
ImageAnnotationBlock,
|
||||
ListBlock,
|
||||
MiddleJson,
|
||||
ModelJson,
|
||||
PageAuxTextBlock,
|
||||
PageBlockTypes,
|
||||
PageInfo,
|
||||
@@ -184,8 +185,15 @@ def test_formula_number_is_rejected_by_middle_json_boundary_and_schema() -> None
|
||||
with pytest.raises(ValidationError):
|
||||
parse_block({"type": RAW_FORMULA_NUMBER, "content": "(1)"})
|
||||
with pytest.raises(ValidationError):
|
||||
model_list_to_pages(
|
||||
[[{"type": RAW_FORMULA_NUMBER, "bbox": [0.7, 0.3, 0.8, 0.4], "content": "(1)"}]]
|
||||
model_json_to_pages(
|
||||
ModelJson(
|
||||
pages=[[{"type": RAW_FORMULA_NUMBER, "bbox": [0.7, 0.3, 0.8, 0.4], "content": "(1)"}]],
|
||||
page_index_map=[],
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="test",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -438,6 +446,7 @@ def test_middle_json_pdf_requires_top_level_bbox_and_round_trips() -> None:
|
||||
with pytest.raises(ValidationError, match="requires bbox"):
|
||||
MiddleJson(
|
||||
pages=[PageInfo(page_idx=0, blocks=[TextBlock(type="text", index=0, content="x")])],
|
||||
is_full_document=True,
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
@@ -450,6 +459,7 @@ def test_middle_json_pdf_requires_top_level_bbox_and_round_trips() -> None:
|
||||
blocks=[TextBlock(type="text", index=0, bbox=(0.1, 0.1, 0.9, 0.9), content="x")],
|
||||
)
|
||||
],
|
||||
is_full_document=True,
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
|
||||
@@ -20,6 +20,7 @@ def _middle_json_with_table(body: TableBodyBlock) -> MiddleJson:
|
||||
table = TableBlock(type="table", index=body.index, content=[body])
|
||||
return MiddleJson(
|
||||
pages=[PageInfo(page_idx=3, blocks=[table])],
|
||||
is_full_document=True,
|
||||
file_suffix="docx",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
@@ -53,6 +54,7 @@ def test_equation_export_uses_canonical_sidecar_name(tmp_path: Path) -> None:
|
||||
)
|
||||
middle_json = MiddleJson(
|
||||
pages=[PageInfo(page_idx=3, blocks=[equation])],
|
||||
is_full_document=True,
|
||||
file_suffix="docx",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from mineru import ModelJson as PublicModelJson
|
||||
from mineru.backend.postprocess import document
|
||||
from mineru.backend.postprocess.document import model_json_to_middle_json
|
||||
from mineru.config import LLMAidedConfig
|
||||
from mineru.types import MiddleJson, ModelJson
|
||||
|
||||
|
||||
def _model_json(
|
||||
*,
|
||||
pages: list[list[dict[str, object]]] | None = None,
|
||||
page_index_map: list[int] | None = None,
|
||||
) -> ModelJson:
|
||||
"""构造包含全部必填元数据的严格 ModelJson 测试对象。"""
|
||||
return ModelJson(
|
||||
pages=pages if pages is not None else [[{"type": "text", "content": "正文"}]],
|
||||
page_index_map=page_index_map if page_index_map is not None else [],
|
||||
file_suffix="docx",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="3.4.0",
|
||||
)
|
||||
|
||||
|
||||
def test_model_json_is_public_and_serializes_exact_envelope() -> None:
|
||||
"""验证公开 ModelJson 固定输出六个顶层字段且空映射不会被省略。"""
|
||||
model_json = _model_json()
|
||||
|
||||
payload = model_json.to_dict()
|
||||
|
||||
assert PublicModelJson is ModelJson
|
||||
assert list(payload) == [
|
||||
"pages",
|
||||
"page_index_map",
|
||||
"file_suffix",
|
||||
"effort",
|
||||
"parse_mode",
|
||||
"mineru_version",
|
||||
]
|
||||
assert payload == {
|
||||
"pages": [[{"type": "text", "content": "正文"}]],
|
||||
"page_index_map": [],
|
||||
"file_suffix": "docx",
|
||||
"effort": "flash",
|
||||
"parse_mode": "txt",
|
||||
"mineru_version": "3.4.0",
|
||||
}
|
||||
assert model_json.is_full_document is True
|
||||
assert model_json.resolved_page_indices == [0]
|
||||
assert "is_full_document" not in payload
|
||||
assert "resolved_page_indices" not in payload
|
||||
assert ModelJson.model_validate_json(model_json.to_json()) == model_json
|
||||
|
||||
|
||||
def test_non_empty_page_index_map_represents_partial_input() -> None:
|
||||
"""验证非空页号映射表示显式抽页并与 raw pages 一一对应。"""
|
||||
model_json = _model_json(pages=[[], []], page_index_map=[3, 5])
|
||||
|
||||
assert model_json.page_index_map == [3, 5]
|
||||
assert model_json.is_full_document is False
|
||||
assert model_json.resolved_page_indices == [3, 5]
|
||||
|
||||
|
||||
def test_empty_model_json_resolves_to_no_page_indices() -> None:
|
||||
"""验证空文档仍属于整本解析且不会产生虚构页号。"""
|
||||
model_json = _model_json(pages=[])
|
||||
|
||||
assert model_json.is_full_document is True
|
||||
assert model_json.resolved_page_indices == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("page_index_map", "message"),
|
||||
[
|
||||
([0], "length mismatch"),
|
||||
([0, 0], "unique"),
|
||||
([1, 0], "increasing"),
|
||||
([0, -1], "non-negative"),
|
||||
],
|
||||
)
|
||||
def test_model_json_rejects_invalid_page_index_map(page_index_map: list[int], message: str) -> None:
|
||||
"""验证显式抽页映射不允许截断、重复、逆序或负数。"""
|
||||
with pytest.raises(ValidationError, match=message):
|
||||
_model_json(pages=[[], []], page_index_map=page_index_map)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"pages",
|
||||
["[]", [{}], [[[]]]],
|
||||
)
|
||||
def test_model_json_rejects_invalid_page_structure(pages: object) -> None:
|
||||
"""验证 pages 必须保持页列表、块列表和块字典三层结构。"""
|
||||
with pytest.raises(ValidationError):
|
||||
ModelJson(
|
||||
pages=pages, # type: ignore[arg-type]
|
||||
page_index_map=[],
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="3.4.0",
|
||||
)
|
||||
|
||||
|
||||
def test_model_json_requires_page_index_map_and_forbids_extra_fields() -> None:
|
||||
"""验证页映射不可省略且 ModelJson 顶层不接受未声明字段。"""
|
||||
payload = {
|
||||
"pages": [],
|
||||
"file_suffix": "pdf",
|
||||
"effort": "flash",
|
||||
"parse_mode": "txt",
|
||||
"mineru_version": "3.4.0",
|
||||
}
|
||||
with pytest.raises(ValidationError, match="page_index_map"):
|
||||
ModelJson.model_validate(payload)
|
||||
|
||||
with pytest.raises(ValidationError, match="extra_forbidden"):
|
||||
ModelJson.model_validate({**payload, "page_index_map": [], "unexpected": True})
|
||||
|
||||
|
||||
def test_middle_json_requires_and_serializes_full_document_flag() -> None:
|
||||
"""验证 MiddleJson 将整本语义作为必填字段稳定序列化并往返。"""
|
||||
middle_json = MiddleJson(
|
||||
pages=[],
|
||||
is_full_document=False,
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="3.4.0",
|
||||
)
|
||||
|
||||
payload = middle_json.to_dict()
|
||||
|
||||
assert list(payload) == [
|
||||
"pages",
|
||||
"is_full_document",
|
||||
"file_suffix",
|
||||
"effort",
|
||||
"parse_mode",
|
||||
"mineru_version",
|
||||
]
|
||||
assert payload["is_full_document"] is False
|
||||
assert MiddleJson.model_validate_json(middle_json.to_json()) == middle_json
|
||||
|
||||
|
||||
def test_model_json_to_middle_json_builds_strict_document_before_pdf_llm(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""验证文档编排完整继承元数据,并让 PDF LLM 直接消费同一 MiddleJson。"""
|
||||
model_json = ModelJson(
|
||||
pages=[[]],
|
||||
page_index_map=[3],
|
||||
file_suffix="pdf",
|
||||
effort="xhigh",
|
||||
parse_mode="ocr",
|
||||
mineru_version="3.4.0",
|
||||
)
|
||||
observed: list[MiddleJson] = []
|
||||
|
||||
def fake_llm_postprocess(middle_json: MiddleJson, _config: LLMAidedConfig) -> None:
|
||||
"""记录 LLM 收到的严格 MiddleJson,并校验抽页语义已写入。"""
|
||||
assert middle_json.is_full_document is False
|
||||
assert [page.page_idx for page in middle_json.pages] == [3]
|
||||
observed.append(middle_json)
|
||||
|
||||
monkeypatch.setattr(document, "apply_llm_aided_postprocess", fake_llm_postprocess)
|
||||
|
||||
middle_json = model_json_to_middle_json(model_json, llm_aided_config=LLMAidedConfig())
|
||||
|
||||
assert observed == [middle_json]
|
||||
assert middle_json.file_suffix == "pdf"
|
||||
assert middle_json.effort == "xhigh"
|
||||
assert middle_json.parse_mode == "ocr"
|
||||
assert middle_json.mineru_version == "3.4.0"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("invalid_value", [None, 0, 1, "true"])
|
||||
def test_middle_json_rejects_missing_or_non_boolean_full_document_flag(invalid_value: object) -> None:
|
||||
"""验证 MiddleJson 不为整本语义提供缺省值或宽松布尔转换。"""
|
||||
payload = {
|
||||
"pages": [],
|
||||
"file_suffix": "pdf",
|
||||
"effort": "flash",
|
||||
"parse_mode": "txt",
|
||||
"mineru_version": "3.4.0",
|
||||
}
|
||||
if invalid_value is not None:
|
||||
payload["is_full_document"] = invalid_value
|
||||
|
||||
with pytest.raises(ValidationError, match="is_full_document"):
|
||||
MiddleJson.model_validate(payload)
|
||||
@@ -24,7 +24,7 @@ from mineru.render._internal.common.inline import (
|
||||
inline_plain_text,
|
||||
parse_inline_content,
|
||||
)
|
||||
from mineru.types import BlockType, ContentType, MiddleJson
|
||||
from mineru.types import BlockType, ContentType, MiddleJson, ModelJson
|
||||
from mineru.utils.pdf_document import PDFDocument
|
||||
|
||||
|
||||
@@ -215,21 +215,36 @@ def test_doc_analyze_converts_vlm_results_before_downstream_processing(
|
||||
monkeypatch.setattr(window, "_apply_seal_ocr", MagicMock())
|
||||
monkeypatch.setattr(window, "_supplement_missing_image_block_containers", MagicMock())
|
||||
monkeypatch.setattr(window, "_attach_visual_block_images", MagicMock())
|
||||
monkeypatch.setattr(analyze, "model_list_to_pages", MagicMock(return_value=[]))
|
||||
expected_middle_json = MiddleJson(
|
||||
pages=[],
|
||||
is_full_document=True,
|
||||
file_suffix="pdf",
|
||||
effort=effort, # type: ignore[arg-type]
|
||||
parse_mode=parse_mode, # type: ignore[arg-type]
|
||||
mineru_version="test",
|
||||
)
|
||||
monkeypatch.setattr(analyze, "model_json_to_middle_json", MagicMock(return_value=expected_middle_json))
|
||||
monkeypatch.setattr(pipeline, "clean_memory", MagicMock())
|
||||
|
||||
middle_json, model_list = analyze.doc_analyze(
|
||||
middle_json, model_json = analyze.doc_analyze(
|
||||
b"fake-pdf",
|
||||
effort=effort, # type: ignore[arg-type]
|
||||
parse_mode=parse_mode, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
assert type(model_list) is list
|
||||
assert type(model_list[0]) is list
|
||||
assert type(model_list[0][0]) is dict
|
||||
assert model_list == [[{"type": BlockType.PAGE_NUMBER, "bbox": [0.45, 0.9, 0.55, 0.95], "content": "1"}]]
|
||||
assert isinstance(model_json, ModelJson)
|
||||
assert type(model_json.pages) is list
|
||||
assert type(model_json.pages[0]) is list
|
||||
assert type(model_json.pages[0][0]) is dict
|
||||
assert model_json.pages == [[{"type": BlockType.PAGE_NUMBER, "bbox": [0.45, 0.9, 0.55, 0.95], "content": "1"}]]
|
||||
assert model_json.page_index_map == []
|
||||
assert model_json.file_suffix == "pdf"
|
||||
assert model_json.effort == effort
|
||||
assert model_json.parse_mode == parse_mode
|
||||
assert isinstance(middle_json, MiddleJson)
|
||||
assert middle_json is expected_middle_json
|
||||
assert middle_json.pages == []
|
||||
assert middle_json.is_full_document is True
|
||||
assert middle_json.effort == effort
|
||||
assert middle_json.parse_mode == parse_mode
|
||||
assert type(source_page) is ExtractResult
|
||||
|
||||
@@ -16,7 +16,8 @@ from mineru.backend import analyze
|
||||
from mineru.backend.analysis import office
|
||||
from mineru.backend.analysis.pdf import constants, formulas, normalization, pipeline, tables, visuals, window
|
||||
from mineru.types import RAW_ALGORITHM, RAW_CAPTION, RAW_FOOTNOTE, RAW_FORMULA_NUMBER, RAW_PHONETIC
|
||||
from mineru.types import BlockType, MiddleJson
|
||||
from mineru.types import BlockType, MiddleJson, ModelJson
|
||||
from mineru.version import __version__ as mineru_version
|
||||
|
||||
|
||||
JPEG_DATA_URI_PREFIX = "data:image/jpeg;base64,"
|
||||
@@ -1201,9 +1202,9 @@ def test_aio_doc_analyze_runs_sync_entrypoint_in_thread_and_forwards_arguments(
|
||||
"""验证异步门面在线程中执行同步入口,并完整转发非默认参数和返回值。"""
|
||||
caller_thread_id = threading.get_ident()
|
||||
observed: dict[str, object] = {}
|
||||
expected_result = (MagicMock(spec=MiddleJson), [[{"type": BlockType.TEXT, "content": "async"}]])
|
||||
expected_result = (MagicMock(spec=MiddleJson), MagicMock(spec=ModelJson))
|
||||
|
||||
def fake_doc_analyze(**kwargs: object) -> tuple[MiddleJson, list[list[dict[str, object]]]]:
|
||||
def fake_doc_analyze(**kwargs: object) -> tuple[MiddleJson, ModelJson]:
|
||||
"""记录工作线程及异步门面传入的全部关键字参数。"""
|
||||
observed["thread_id"] = threading.get_ident()
|
||||
observed["kwargs"] = kwargs
|
||||
@@ -1238,7 +1239,8 @@ def test_aio_doc_analyze_propagates_sync_entrypoint_error(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""验证同步入口异常会穿过工作线程并由异步调用方原样收到。"""
|
||||
def fake_doc_analyze(**_kwargs: object) -> tuple[MiddleJson, list[list[dict[str, object]]]]:
|
||||
|
||||
def fake_doc_analyze(**_kwargs: object) -> tuple[MiddleJson, ModelJson]:
|
||||
"""模拟同步文档分析阶段抛出参数异常。"""
|
||||
raise ValueError("async analyze failed")
|
||||
|
||||
@@ -1249,11 +1251,11 @@ def test_aio_doc_analyze_propagates_sync_entrypoint_error(
|
||||
|
||||
|
||||
@pytest.mark.parametrize("file_suffix", ["docx", "pptx", "xlsx"])
|
||||
def test_doc_analyze_office_returns_model_list_without_pdf_processing(
|
||||
def test_doc_analyze_office_returns_model_json_without_pdf_processing(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
file_suffix: str,
|
||||
) -> None:
|
||||
"""验证三类 Office 文件直接返回模型结果,且不进入任何 PDF 处理阶段。"""
|
||||
"""验证三类 Office 文件返回严格 ModelJson,且不进入任何 PDF 处理阶段。"""
|
||||
source_model_list = [[{"type": BlockType.TEXT, "content": "原始 \\(office\\) 内容"}]]
|
||||
events: list[str] = []
|
||||
model_factories: dict[str, MagicMock] = {}
|
||||
@@ -1294,7 +1296,7 @@ def test_doc_analyze_office_returns_model_list_without_pdf_processing(
|
||||
monkeypatch.setattr(pipeline, "_normalize_pdf_model_list", model_list_normalizer)
|
||||
monkeypatch.setattr(office.time, "perf_counter", fake_perf_counter)
|
||||
|
||||
middle_json, model_list = analyze.doc_analyze(
|
||||
middle_json, model_json = analyze.doc_analyze(
|
||||
b"office-bytes",
|
||||
effort="xhigh",
|
||||
parse_mode="ocr",
|
||||
@@ -1306,8 +1308,16 @@ def test_doc_analyze_office_returns_model_list_without_pdf_processing(
|
||||
assert middle_json.file_suffix == file_suffix
|
||||
assert middle_json.effort == "flash"
|
||||
assert middle_json.parse_mode == "txt"
|
||||
assert model_list is source_model_list
|
||||
assert model_list[0][0]["content"] == "原始 \\(office\\) 内容"
|
||||
assert middle_json.is_full_document is True
|
||||
assert isinstance(model_json, ModelJson)
|
||||
assert model_json.pages == source_model_list
|
||||
assert model_json.page_index_map == []
|
||||
assert model_json.is_full_document is True
|
||||
assert model_json.file_suffix == file_suffix
|
||||
assert model_json.effort == "flash"
|
||||
assert model_json.parse_mode == "txt"
|
||||
assert model_json.mineru_version == mineru_version
|
||||
assert model_json.pages[0][0]["content"] == "原始 \\(office\\) 内容"
|
||||
for suffix, model_factory in model_factories.items():
|
||||
assert model_factory.call_count == (1 if suffix == file_suffix else 0)
|
||||
file_stream = selected_model.predict.call_args.args[0]
|
||||
@@ -1472,7 +1482,7 @@ def test_pdf_infer_timer_excludes_hybrid_vlm_initialization_and_cleanup(
|
||||
monkeypatch.setattr(pipeline, "_normalize_pdf_model_list", fake_normalize_model_list)
|
||||
monkeypatch.setattr(pipeline, "clean_memory", fake_clean_memory)
|
||||
|
||||
middle_json, model_list = analyze.doc_analyze(
|
||||
middle_json, model_json = analyze.doc_analyze(
|
||||
b"empty-pdf",
|
||||
effort="high",
|
||||
parse_mode="txt",
|
||||
@@ -1480,7 +1490,11 @@ def test_pdf_infer_timer_excludes_hybrid_vlm_initialization_and_cleanup(
|
||||
|
||||
assert isinstance(middle_json, MiddleJson)
|
||||
assert middle_json.pages == []
|
||||
assert model_list == []
|
||||
assert middle_json.is_full_document is True
|
||||
assert isinstance(model_json, ModelJson)
|
||||
assert model_json.pages == []
|
||||
assert model_json.page_index_map == []
|
||||
assert model_json.is_full_document is True
|
||||
assert events == [
|
||||
"hybrid_init",
|
||||
"vlm_init",
|
||||
@@ -1566,7 +1580,7 @@ def test_doc_analyze_office_real_samples(file_suffix: str, expected_page_count:
|
||||
"""验证统一入口可直接分析三类真实 Office 样例并返回完整分页结果。"""
|
||||
sample_path = _OFFICE_SAMPLE_DIR / f"{file_suffix}_01.{file_suffix}"
|
||||
|
||||
middle_json, model_list = analyze.doc_analyze(
|
||||
middle_json, model_json = analyze.doc_analyze(
|
||||
sample_path.read_bytes(),
|
||||
effort="high",
|
||||
parse_mode="ocr",
|
||||
@@ -1574,13 +1588,22 @@ def test_doc_analyze_office_real_samples(file_suffix: str, expected_page_count:
|
||||
)
|
||||
|
||||
assert isinstance(middle_json, MiddleJson)
|
||||
assert isinstance(model_json, ModelJson)
|
||||
assert middle_json.is_full_document is True
|
||||
assert len(middle_json.pages) == expected_page_count
|
||||
assert all(page.page_idx == page_idx for page_idx, page in enumerate(middle_json.pages))
|
||||
assert len(model_list) == expected_page_count
|
||||
assert all(isinstance(page, list) for page in model_list)
|
||||
assert len(model_json.pages) == expected_page_count
|
||||
assert all(isinstance(page, list) for page in model_json.pages)
|
||||
assert model_json.page_index_map == []
|
||||
assert model_json.file_suffix == file_suffix
|
||||
assert model_json.effort == "flash"
|
||||
assert model_json.parse_mode == "txt"
|
||||
if file_suffix == "docx":
|
||||
model_equations = [
|
||||
block for page_model_list in model_list for block in page_model_list if block.get("type") == BlockType.EQUATION
|
||||
block
|
||||
for page_model_list in model_json.pages
|
||||
for block in page_model_list
|
||||
if block.get("type") == BlockType.EQUATION
|
||||
]
|
||||
middle_equations = [block for page in middle_json.pages for block in page.blocks if block.type == BlockType.EQUATION]
|
||||
assert model_equations
|
||||
@@ -1592,7 +1615,7 @@ def test_doc_analyze_flash_real_pdf_returns_typed_middle_json() -> None:
|
||||
"""验证一页真实 PDF 经 Flash Analyze 后返回严格对象且 raw 结果无废弃字段。"""
|
||||
sample_path = _PROJECT_ROOT / "demo" / "pdfs" / "2407.00079v4_origi-10.pdf"
|
||||
|
||||
middle_json, model_list = analyze.doc_analyze(
|
||||
middle_json, model_json = analyze.doc_analyze(
|
||||
sample_path.read_bytes(),
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
@@ -1600,17 +1623,25 @@ def test_doc_analyze_flash_real_pdf_returns_typed_middle_json() -> None:
|
||||
)
|
||||
|
||||
assert isinstance(middle_json, MiddleJson)
|
||||
assert len(middle_json.pages) == len(model_list) == 1
|
||||
assert isinstance(model_json, ModelJson)
|
||||
assert middle_json.is_full_document is True
|
||||
assert len(middle_json.pages) == len(model_json.pages) == 1
|
||||
assert all(block.bbox is not None for block in middle_json.pages[0].blocks)
|
||||
assert all("merge_prev" not in block for page in model_list for block in page)
|
||||
assert all("merge_prev" not in block for page in model_json.pages for block in page)
|
||||
assert model_json.page_index_map == []
|
||||
assert model_json.file_suffix == "pdf"
|
||||
assert model_json.effort == "flash"
|
||||
assert model_json.parse_mode == "txt"
|
||||
assert model_json.mineru_version == mineru_version
|
||||
assert MiddleJson.model_validate_json(middle_json.to_json()) == middle_json
|
||||
assert ModelJson.model_validate_json(model_json.to_json()) == model_json
|
||||
|
||||
|
||||
def test_doc_analyze_flash_demo1_uses_canonical_equation_type() -> None:
|
||||
"""验证真实 demo1.pdf 的两层 Flash 输出统一使用 equation 与 LaTeX tag。"""
|
||||
sample_path = _PROJECT_ROOT / "demo" / "pdfs" / "demo1.pdf"
|
||||
|
||||
middle_json, model_list = analyze.doc_analyze(
|
||||
middle_json, model_json = analyze.doc_analyze(
|
||||
sample_path.read_bytes(),
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
@@ -1618,32 +1649,27 @@ def test_doc_analyze_flash_demo1_uses_canonical_equation_type() -> None:
|
||||
)
|
||||
|
||||
model_equations = [
|
||||
block for page_model_list in model_list for block in page_model_list if block.get("type") == BlockType.EQUATION
|
||||
block for page_model_list in model_json.pages for block in page_model_list if block.get("type") == BlockType.EQUATION
|
||||
]
|
||||
middle_equations = [block for page in middle_json.pages for block in page.blocks if block.type == BlockType.EQUATION]
|
||||
|
||||
assert middle_json.is_full_document is True
|
||||
assert model_equations
|
||||
assert len(middle_equations) == len(model_equations)
|
||||
assert [block.content for block in middle_equations] == [
|
||||
block["content"]
|
||||
for block in model_equations
|
||||
]
|
||||
assert [block.content for block in middle_equations] == [block["content"] for block in model_equations]
|
||||
for formula_number in range(1, 8):
|
||||
marker = rf"\tag{{{formula_number}}}"
|
||||
assert sum(marker in block["content"] for block in model_equations) == 1
|
||||
assert not [
|
||||
block
|
||||
for block in model_equations
|
||||
if any(
|
||||
block["content"].rstrip().endswith(f"({formula_number})")
|
||||
for formula_number in range(1, 8)
|
||||
)
|
||||
if any(block["content"].rstrip().endswith(f"({formula_number})") for formula_number in range(1, 8))
|
||||
]
|
||||
assert "interline_equation" not in middle_json.to_json()
|
||||
|
||||
|
||||
def test_doc_analyze_flash_returns_complete_model_list_and_typed_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""验证 Flash 多窗口补充完整 raw model list,并返回严格 MiddleJson。"""
|
||||
def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""验证 Flash 多窗口补充完整 raw pages,并返回严格 ModelJson 与 MiddleJson。"""
|
||||
from mineru.model import flash as flash_model
|
||||
|
||||
events: list[str] = []
|
||||
@@ -1740,7 +1766,7 @@ def test_doc_analyze_flash_returns_complete_model_list_and_typed_middle_json(mon
|
||||
monkeypatch.setattr(pipeline.time, "perf_counter", fake_perf_counter)
|
||||
monkeypatch.setattr(flash_model, "PdfModel", MagicMock(return_value=fake_pdf_model))
|
||||
|
||||
middle_json, model_list = analyze.doc_analyze(
|
||||
middle_json, model_json = analyze.doc_analyze(
|
||||
b"fake-pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
@@ -1748,16 +1774,20 @@ def test_doc_analyze_flash_returns_complete_model_list_and_typed_middle_json(mon
|
||||
)
|
||||
|
||||
assert isinstance(middle_json, MiddleJson)
|
||||
assert isinstance(model_json, ModelJson)
|
||||
assert middle_json.is_full_document is False
|
||||
assert [page.page_idx for page in middle_json.pages] == [7, 8, 9]
|
||||
assert model_list is source_model_list
|
||||
assert model_json.pages == source_model_list
|
||||
assert model_json.page_index_map == [7, 8, 9]
|
||||
assert model_json.is_full_document is False
|
||||
assert requested_ranges == [(0, 1), (2, 2)]
|
||||
assert model_list[0][0]["content"] == "第一页 <eq>x+y</eq>"
|
||||
assert model_list[2][0]["content"] == "第三页 <eq>z</eq>"
|
||||
assert model_list[2][0]["type"] == BlockType.EQUATION
|
||||
assert model_json.pages[0][0]["content"] == "第一页 <eq>x+y</eq>"
|
||||
assert model_json.pages[2][0]["content"] == "第三页 <eq>z</eq>"
|
||||
assert model_json.pages[2][0]["type"] == BlockType.EQUATION
|
||||
assert middle_json.pages[2].blocks[0].type == BlockType.EQUATION
|
||||
assert "interline_equation" not in middle_json.to_json()
|
||||
assert "image_base64" not in model_list[0][0]
|
||||
for block in (model_list[1][0], model_list[2][0]):
|
||||
assert "image_base64" not in model_json.pages[0][0]
|
||||
for block in (model_json.pages[1][0], model_json.pages[2][0]):
|
||||
crop_image = _decode_jpeg_data_uri(block["image_base64"])
|
||||
crop_image.close()
|
||||
for image in rendered_images:
|
||||
|
||||
@@ -4,6 +4,7 @@ import math
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
from zipfile import ZipFile
|
||||
|
||||
@@ -13,7 +14,7 @@ from mineru.backend.analysis.pdf import window
|
||||
from mineru.backend.analysis.pdf.text import content as text_content
|
||||
from mineru.backend.analysis.pdf.text.models import _AnalyzeLine, _AnalyzeSpan
|
||||
from mineru.backend.analysis.pdf.text.native import txt_spans_extract
|
||||
from mineru.backend.postprocess.pages import model_list_to_pages
|
||||
from mineru.backend.postprocess.pages import model_json_to_pages
|
||||
from mineru.model.flash import PdfModel
|
||||
from mineru.render import render_docx, render_html, render_markdown, render_structured_content
|
||||
from mineru.types import (
|
||||
@@ -22,6 +23,7 @@ from mineru.types import (
|
||||
BlockType,
|
||||
ContentType,
|
||||
MiddleJson,
|
||||
ModelJson,
|
||||
PageInfo,
|
||||
)
|
||||
from mineru.utils.pdf_document import PDFDocument, PDFLinkAnnotation
|
||||
@@ -39,6 +41,18 @@ from mineru.utils.pdf_text_styles import (
|
||||
)
|
||||
|
||||
|
||||
def _model_json(pages: list[list[dict[str, Any]]]) -> ModelJson:
|
||||
"""为 PDF 文本样式渲染测试构造最小严格 ModelJson。"""
|
||||
return ModelJson(
|
||||
pages=pages,
|
||||
page_index_map=[],
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="test",
|
||||
)
|
||||
|
||||
|
||||
def _build_native_text_style_pdf() -> bytes:
|
||||
"""构造包含字体、删除线、下划线及其反例的单页原生 PDF。"""
|
||||
|
||||
@@ -1761,7 +1775,8 @@ def test_flash_native_pdf_styles_reach_model_middle_and_renderers() -> None:
|
||||
assert '<text style="bold">bold italic sample</text>' in joined_model_content
|
||||
|
||||
middle = MiddleJson(
|
||||
pages=model_list_to_pages(model_list),
|
||||
pages=model_json_to_pages(_model_json(model_list)),
|
||||
is_full_document=True,
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
@@ -1806,7 +1821,8 @@ def test_demo1_pdf_link_reaches_model_middle_and_all_renderers() -> None:
|
||||
)
|
||||
|
||||
middle = MiddleJson(
|
||||
pages=model_list_to_pages(model_list),
|
||||
pages=model_json_to_pages(_model_json(model_list)),
|
||||
is_full_document=True,
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
@@ -1819,6 +1835,7 @@ def test_demo1_pdf_link_reaches_model_middle_and_all_renderers() -> None:
|
||||
)
|
||||
link_middle = MiddleJson(
|
||||
pages=[PageInfo(page_idx=0, blocks=[link_block])],
|
||||
is_full_document=True,
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
|
||||
@@ -1,15 +1,16 @@
|
||||
from copy import deepcopy
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from mineru.backend.postprocess import pages as pages_module
|
||||
from mineru.backend.postprocess.lists import fix_office_list_blocks
|
||||
from mineru.backend.postprocess.pages import model_list_to_pages
|
||||
from mineru.backend.postprocess.pages import model_json_to_pages
|
||||
from mineru.types import (
|
||||
ChartBlock,
|
||||
DocTitleBlock,
|
||||
EquationBlock,
|
||||
ImageBlock,
|
||||
ListBlock,
|
||||
ModelJson,
|
||||
PageInfo,
|
||||
ParagraphTitleBlock,
|
||||
RefTextBlock,
|
||||
@@ -18,7 +19,29 @@ from mineru.types import (
|
||||
)
|
||||
|
||||
|
||||
def test_model_list_to_pages_keeps_equation_type_without_mutating_input() -> None:
|
||||
def _model_json(
|
||||
pages: list[list[dict[str, Any]]],
|
||||
*,
|
||||
page_index_map: list[int] | None = None,
|
||||
) -> ModelJson:
|
||||
"""为 PageInfo 后处理测试构造最小严格 ModelJson。"""
|
||||
return ModelJson(
|
||||
pages=pages,
|
||||
page_index_map=page_index_map or [],
|
||||
file_suffix="pdf",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
mineru_version="test",
|
||||
)
|
||||
|
||||
|
||||
def test_model_json_to_pages_replaces_raw_model_list_entrypoint() -> None:
|
||||
"""验证严格 ModelJson 是唯一文档级 PageInfo 转换入口。"""
|
||||
assert callable(pages_module.model_json_to_pages)
|
||||
assert not hasattr(pages_module, "model_list_to_pages")
|
||||
|
||||
|
||||
def test_model_json_to_pages_keeps_equation_type_without_mutating_input() -> None:
|
||||
"""验证 equation 经单页后处理和严格对象化后保持类型、内容与图片载荷。"""
|
||||
model_list = [
|
||||
[
|
||||
@@ -32,7 +55,7 @@ def test_model_list_to_pages_keeps_equation_type_without_mutating_input() -> Non
|
||||
]
|
||||
original = deepcopy(model_list)
|
||||
|
||||
page = model_list_to_pages(model_list)[0]
|
||||
page = model_json_to_pages(_model_json(model_list))[0]
|
||||
|
||||
assert model_list == original
|
||||
assert isinstance(page.blocks[0], EquationBlock)
|
||||
@@ -41,7 +64,7 @@ def test_model_list_to_pages_keeps_equation_type_without_mutating_input() -> Non
|
||||
assert page.blocks[0].image_base64 == "data:image/jpeg;base64,/9j/2Q=="
|
||||
|
||||
|
||||
def test_model_list_to_pages_returns_typed_pdf_tree_without_mutating_input() -> None:
|
||||
def test_model_json_to_pages_returns_typed_pdf_tree_without_mutating_input() -> None:
|
||||
"""验证 PDF raw dict 只在副本上分组,并返回具体 PageInfo/Block 对象。"""
|
||||
model_list = [
|
||||
[
|
||||
@@ -58,7 +81,7 @@ def test_model_list_to_pages_returns_typed_pdf_tree_without_mutating_input() ->
|
||||
]
|
||||
original = deepcopy(model_list)
|
||||
|
||||
pages = model_list_to_pages(model_list, page_index_map=[7])
|
||||
pages = model_json_to_pages(_model_json(model_list, page_index_map=[7]))
|
||||
|
||||
assert model_list == original
|
||||
assert isinstance(pages[0], PageInfo)
|
||||
@@ -72,7 +95,7 @@ def test_model_list_to_pages_returns_typed_pdf_tree_without_mutating_input() ->
|
||||
assert isinstance(pages[0].blocks[1].content[0], TextBlock)
|
||||
|
||||
|
||||
def test_model_list_to_pages_preserves_recursive_office_list_and_index() -> None:
|
||||
def test_model_json_to_pages_preserves_recursive_office_list_and_index() -> None:
|
||||
"""验证 Office list/index 任意深度递归对象化,并清理未匹配目录 anchor。"""
|
||||
model_list = [
|
||||
[
|
||||
@@ -106,7 +129,7 @@ def test_model_list_to_pages_preserves_recursive_office_list_and_index() -> None
|
||||
]
|
||||
]
|
||||
|
||||
page = model_list_to_pages(model_list)[0]
|
||||
page = model_json_to_pages(_model_json(model_list))[0]
|
||||
list_block = page.blocks[0]
|
||||
index_block = page.blocks[1]
|
||||
|
||||
@@ -216,7 +239,7 @@ def test_fix_office_list_blocks_uses_local_ordered_markers_at_each_depth() -> No
|
||||
)
|
||||
|
||||
|
||||
def test_model_list_to_pages_maps_index_anchor_to_document_title_type_and_level() -> None:
|
||||
def test_model_json_to_pages_maps_index_anchor_to_document_title_type_and_level() -> None:
|
||||
"""验证 Office 目录叶子按跨页目标 anchor 继承真实标题类型与层级。"""
|
||||
model_list = [
|
||||
[
|
||||
@@ -245,7 +268,7 @@ def test_model_list_to_pages_maps_index_anchor_to_document_title_type_and_level(
|
||||
],
|
||||
]
|
||||
|
||||
pages = model_list_to_pages(model_list)
|
||||
pages = model_json_to_pages(_model_json(model_list))
|
||||
index = pages[0].blocks[0]
|
||||
doc_leaf = index.content[0]
|
||||
section_leaf = index.content[1].content[0]
|
||||
@@ -263,7 +286,7 @@ def test_model_list_to_pages_maps_index_anchor_to_document_title_type_and_level(
|
||||
assert "anchor" not in missing_leaf.model_fields_set
|
||||
|
||||
|
||||
def test_model_list_to_pages_uses_first_title_for_duplicate_anchor() -> None:
|
||||
def test_model_json_to_pages_uses_first_title_for_duplicate_anchor() -> None:
|
||||
"""验证重复 anchor 按文档顺序使用首个标题目标。"""
|
||||
model_list = [
|
||||
[
|
||||
@@ -276,7 +299,7 @@ def test_model_list_to_pages_uses_first_title_for_duplicate_anchor() -> None:
|
||||
]
|
||||
]
|
||||
|
||||
page = model_list_to_pages(model_list)[0]
|
||||
page = model_json_to_pages(_model_json(model_list))[0]
|
||||
index = page.blocks[2]
|
||||
|
||||
assert isinstance(index.content[0], ParagraphTitleBlock)
|
||||
@@ -298,7 +321,7 @@ def test_office_paragraph_numbering_is_document_wide_and_copy_only() -> None:
|
||||
]
|
||||
original = deepcopy(model_list)
|
||||
|
||||
pages = model_list_to_pages(model_list)
|
||||
pages = model_json_to_pages(_model_json(model_list))
|
||||
titles = [block for page in pages for block in page.blocks]
|
||||
|
||||
assert all(isinstance(block, ParagraphTitleBlock) for block in titles)
|
||||
@@ -327,7 +350,7 @@ def test_office_paragraph_numbering_clears_deeper_levels() -> None:
|
||||
]
|
||||
]
|
||||
|
||||
titles = model_list_to_pages(model_list)[0].blocks
|
||||
titles = model_json_to_pages(_model_json(model_list))[0].blocks
|
||||
|
||||
assert [title.content for title in titles] == [
|
||||
"1 A",
|
||||
@@ -342,7 +365,7 @@ def test_office_paragraph_numbering_clears_deeper_levels() -> None:
|
||||
|
||||
def test_visual_body_drops_empty_parent_subtype() -> None:
|
||||
"""验证 raw visual 的空 subtype 只用于父块判断,不会泄漏到严格 body 模型。"""
|
||||
page = model_list_to_pages([[{"type": "image", "content": None, "sub_type": None}]])[0]
|
||||
page = model_json_to_pages(_model_json([[{"type": "image", "content": None, "sub_type": None}]]))[0]
|
||||
|
||||
assert isinstance(page.blocks[0], ImageBlock)
|
||||
assert page.blocks[0].sub_type is None
|
||||
@@ -352,7 +375,7 @@ def test_visual_body_drops_empty_parent_subtype() -> None:
|
||||
|
||||
def test_chart_none_content_is_normalized_to_empty_string() -> None:
|
||||
"""验证 raw chart 的 null content 在严格对象化前规范为空字符串。"""
|
||||
page = model_list_to_pages([[{"type": "chart", "content": None}]])[0]
|
||||
page = model_json_to_pages(_model_json([[{"type": "chart", "content": None}]]))[0]
|
||||
|
||||
assert isinstance(page.blocks[0], ChartBlock)
|
||||
assert page.blocks[0].content[0].type == "chart_body"
|
||||
@@ -361,14 +384,16 @@ def test_chart_none_content_is_normalized_to_empty_string() -> None:
|
||||
|
||||
def test_raw_title_levels_are_normalized_to_global_hierarchy() -> None:
|
||||
"""验证 raw 标题在严格对象化前归一为全局一至六级层级。"""
|
||||
page = model_list_to_pages(
|
||||
[
|
||||
page = model_json_to_pages(
|
||||
_model_json(
|
||||
[
|
||||
{"type": "doc_title", "content": "Document"},
|
||||
{"type": "paragraph_title", "content": "Section", "level": 1},
|
||||
{"type": "paragraph_title", "content": "Deep", "level": 9},
|
||||
[
|
||||
{"type": "doc_title", "content": "Document"},
|
||||
{"type": "paragraph_title", "content": "Section", "level": 1},
|
||||
{"type": "paragraph_title", "content": "Deep", "level": 9},
|
||||
]
|
||||
]
|
||||
]
|
||||
)
|
||||
)[0]
|
||||
|
||||
assert [block.level for block in page.blocks] == [1, 2, 6]
|
||||
@@ -376,8 +401,8 @@ def test_raw_title_levels_are_normalized_to_global_hierarchy() -> None:
|
||||
|
||||
def test_pdf_raw_paragraph_title_level_is_clamped_to_six() -> None:
|
||||
"""验证带 bbox 的 PDF raw 深层标题在严格对象化前归一为六级。"""
|
||||
page = model_list_to_pages(
|
||||
[[{"type": "paragraph_title", "content": "Deep", "level": 9, "bbox": [0.1, 0.1, 0.9, 0.2]}]]
|
||||
page = model_json_to_pages(
|
||||
_model_json([[{"type": "paragraph_title", "content": "Deep", "level": 9, "bbox": [0.1, 0.1, 0.9, 0.2]}]])
|
||||
)[0]
|
||||
|
||||
assert isinstance(page.blocks[0], ParagraphTitleBlock)
|
||||
@@ -402,7 +427,7 @@ def test_office_title_level_is_clamped_before_numbering_and_index_mapping() -> N
|
||||
]
|
||||
]
|
||||
|
||||
page = model_list_to_pages(model_list)[0]
|
||||
page = model_json_to_pages(_model_json(model_list))[0]
|
||||
title = page.blocks[0]
|
||||
index = page.blocks[1]
|
||||
|
||||
@@ -432,7 +457,7 @@ def test_pdf_continuation_is_typed_and_line_metadata_is_removed() -> None:
|
||||
]
|
||||
]
|
||||
|
||||
page = model_list_to_pages(model_list)[0]
|
||||
page = model_json_to_pages(_model_json(model_list))[0]
|
||||
|
||||
assert all(isinstance(block, TextBlock) for block in page.blocks)
|
||||
assert page.blocks[0].continues_prev is None
|
||||
@@ -459,7 +484,7 @@ def test_pdf_ref_text_continuation_is_typed_and_line_metadata_is_removed() -> No
|
||||
]
|
||||
]
|
||||
|
||||
page = model_list_to_pages(model_list)[0]
|
||||
page = model_json_to_pages(_model_json(model_list))[0]
|
||||
|
||||
assert all(isinstance(block, RefTextBlock) for block in page.blocks)
|
||||
assert page.blocks[0].continues_prev is None
|
||||
@@ -469,11 +494,13 @@ def test_pdf_ref_text_continuation_is_typed_and_line_metadata_is_removed() -> No
|
||||
|
||||
def test_pdf_detection_scans_past_empty_first_page() -> None:
|
||||
"""验证整份文档 bbox 判定不会把 PDF 空白首页误认为 Office。"""
|
||||
pages = model_list_to_pages(
|
||||
[
|
||||
[],
|
||||
[{"type": "text", "bbox": [0.1, 0.1, 0.9, 0.3], "content": "later", "lines": []}],
|
||||
]
|
||||
pages = model_json_to_pages(
|
||||
_model_json(
|
||||
[
|
||||
[],
|
||||
[{"type": "text", "bbox": [0.1, 0.1, 0.9, 0.3], "content": "later", "lines": []}],
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
assert pages[0].blocks == []
|
||||
@@ -484,23 +511,15 @@ def test_cross_page_table_continuation_remains_raw_postprocess() -> None:
|
||||
"""验证连续页表格仍在对象化前写入 continues_prev。"""
|
||||
html_a = "<table><tr><td>A</td><td>B</td></tr></table>"
|
||||
html_b = "<table><tr><td>C</td><td>D</td></tr></table>"
|
||||
pages = model_list_to_pages(
|
||||
[
|
||||
[{"type": "table", "bbox": [0.1, 0.1, 0.9, 0.85], "content": html_a}],
|
||||
[{"type": "table", "bbox": [0.1, 0.1, 0.9, 0.85], "content": html_b}],
|
||||
]
|
||||
pages = model_json_to_pages(
|
||||
_model_json(
|
||||
[
|
||||
[{"type": "table", "bbox": [0.1, 0.1, 0.9, 0.85], "content": html_a}],
|
||||
[{"type": "table", "bbox": [0.1, 0.1, 0.9, 0.85], "content": html_b}],
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
assert isinstance(pages[0].blocks[0], TableBlock)
|
||||
assert pages[0].blocks[0].continues_prev is None
|
||||
assert pages[1].blocks[0].continues_prev is True
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("page_index_map", "message"),
|
||||
[([0], "length mismatch"), ([0, 0], "unique"), ([1, 0], "increasing"), ([0, -1], "non-negative")],
|
||||
)
|
||||
def test_page_index_map_is_strict(page_index_map: list[int], message: str) -> None:
|
||||
"""验证页号映射不允许截断、重复、逆序或负数。"""
|
||||
with pytest.raises(ValueError, match=message):
|
||||
model_list_to_pages([[], []], page_index_map=page_index_map)
|
||||
|
||||
@@ -26,6 +26,7 @@ def _middle(*pages: PageInfo) -> MiddleJson:
|
||||
"""构造无需 PDF bbox 的最小严格 MiddleJson。"""
|
||||
return MiddleJson(
|
||||
pages=list(pages),
|
||||
is_full_document=True,
|
||||
file_suffix="docx",
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
|
||||
@@ -31,6 +31,7 @@ def _middle(*pages: PageInfo, file_suffix: str = "docx") -> MiddleJson:
|
||||
"""构造最小严格 MiddleJson 测试对象。"""
|
||||
return MiddleJson(
|
||||
pages=list(pages),
|
||||
is_full_document=True,
|
||||
file_suffix=file_suffix,
|
||||
effort="flash",
|
||||
parse_mode="txt",
|
||||
|
||||
@@ -81,7 +81,7 @@ def _table(
|
||||
|
||||
|
||||
def _page(page_idx: int, blocks: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""构造 model_list_to_pages 输出契约的页面。"""
|
||||
"""构造 ModelJson 后处理阶段使用的 raw 页面。"""
|
||||
return {"page_idx": page_idx, "blocks": blocks}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user