opensource/MinerU
3
0
Fork 0
mirror of https://github.com/opendatalab/MinerU.git synced 2026-09-01 04:26:21 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat: enhance output_files.md with new content structure and file generation details

Browse Source
This commit is contained in:
myhloli
2026-03-29 00:44:52 +08:00
parent 5593088ff7
commit eac9f2f894
2 changed files with 230 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/en/reference/output_files.md
+110 -95
View File
@@ -4,6 +4,8 @@
After executing the `mineru` command, in addition to the main markdown file output, multiple auxiliary files are generated for debugging, quality inspection, and further processing. These files include:
The exact set of generated files depends on the backend and the input document type.
- **Visual debugging files**: Help users intuitively understand the document parsing process and results
- **Structured data files**: Contain detailed parsing data for secondary development
@@ -60,109 +62,45 @@ The following sections provide detailed descriptions of each file's purpose and
**File naming format**: `{original_filename}_model.json`
##### Data Structure Definition
```python
from pydantic import BaseModel, Field
from enum import IntEnum
class CategoryType(IntEnum):
"""Content category enumeration"""
title = 0 # Title
plain_text = 1 # Text
abandon = 2 # Including headers, footers, page numbers, and page annotations
figure = 3 # Image
figure_caption = 4 # Image caption
table = 5 # Table
table_caption = 6 # Table caption
table_footnote = 7 # Table footnote
isolate_formula = 8 # Interline formula
formula_caption = 9 # Interline formula number
embedding = 13 # Inline formula
isolated = 14 # Interline formula
text = 15 # OCR recognition result
class PageInfo(BaseModel):
"""Page information"""
page_no: int = Field(description="Page number, first page is 0", ge=0)
height: int = Field(description="Page height", gt=0)
width: int = Field(description="Page width", ge=0)
class ObjectInferenceResult(BaseModel):
"""Object recognition result"""
category_id: CategoryType = Field(description="Category", ge=0)
poly: list[float] = Field(description="Quadrilateral coordinates, format: [x0,y0,x1,y1,x2,y2,x3,y3]")
score: float = Field(description="Confidence score of inference result")
latex: str | None = Field(description="LaTeX parsing result", default=None)
html: str | None = Field(description="HTML parsing result", default=None)
class PageInferenceResults(BaseModel):
"""Page inference results"""
layout_dets: list[ObjectInferenceResult] = Field(description="Page recognition results")
page_info: PageInfo = Field(description="Page metadata")
# Complete inference results
inference_result: list[PageInferenceResults] = []
```
##### Coordinate System Description
`poly` coordinate format: `[x0, y0, x1, y1, x2, y2, x3, y3]`
- Represents coordinates of top-left, top-right, bottom-right, bottom-left points respectively
- Coordinate origin is at the top-left corner of the page
![poly coordinate diagram](../images/poly.png)
##### Sample Data
```json
[
{
"layout_dets": [
{
"category_id": 2,
"poly": [
99.1906967163086,
100.3119125366211,
730.3707885742188,
100.3119125366211,
730.3707885742188,
245.81326293945312,
99.1906967163086,
245.81326293945312
],
"score": 0.9999997615814209
}
"cls_id": 12,
"label": "header",
"score": 0.93,
"bbox": [
1217,
104,
1516,
134
],
"page_info": {
"page_no": 0,
"height": 2339,
"width": 1654
}
"index": 2
},
{
"layout_dets": [
{
"category_id": 5,
"poly": [
99.13092803955078,
2210.680419921875,
497.3183898925781,
2210.680419921875,
497.3183898925781,
2264.78076171875,
99.13092803955078,
2264.78076171875
],
"score": 0.9999997019767761
}
"cls_id": 6,
"label": "doc_title",
"score": 0.9751,
"bbox": [
275,
181,
1512,
292
],
"page_info": {
"page_no": 1,
"height": 2339,
"width": 1654
}
"index": 3
},
{
"cls_id": 22,
"label": "text",
"score": 0.9217,
"bbox": [
275,
330,
524,
370
],
"index": 4
}
]
```
@@ -176,7 +114,7 @@ inference_result: list[PageInferenceResults] = []
| Field Name | Type | Description |
|------------|------|-------------|
| `pdf_info` | `list[dict]` | Array of parsing results for each page |
| `_backend` | `string` | Parsing mode: `pipeline` or `vlm` |
| `_backend` | `string` | Parsing mode: `pipeline`, `vlm`, or `office` |
| `_version_name` | `string` | MinerU version number |
##### Page Information Structure (pdf_info)
@@ -361,8 +299,13 @@ This is a simplified version of `middle.json` that stores all readable content b
|------|-------------|
| `image` | Image |
| `table` | Table |
| `chart` | Chart |
| `text` | Text/Title |
| `equation` | Interline formula |
| `seal` | Seal |
| `code` | Code block / algorithm block |
| `list` | List / reference list |
| `header` / `footer` / `page_number` / `aside_text` / `page_footnote` | Page auxiliary blocks |
##### Text Level Identification
@@ -377,6 +320,8 @@ Text levels are distinguished through the `text_level` field:
- All content blocks include a `page_idx` field indicating the page number (starting from 0).
- All content blocks include a `bbox` field representing the bounding box coordinates of the content block `[x0, y0, x1, y1]`, mapped to a range of 0-1000.
- `code` entries use `sub_type` to distinguish `code` and `algorithm`, and may include fields such as `code_body`, `code_caption`, and `code_footnote`.
- `list` entries may use `sub_type` to distinguish ordinary lists from reference-style lists.
##### Sample Data
@@ -443,6 +388,74 @@ Text levels are distinguished through the `text_level` field:
]
```
### Common Content List V2 (content_list_v2.json)
**File naming format**: `{original_filename}_content_list_v2.json`
##### Functionality
`content_list_v2.json` is the new structured output added in 3.0. All backends now emit it in addition to the legacy `content_list.json`:
- The top level is grouped by page for page-oriented consumption
- Each item uses a unified `type + content` structure for easier programmatic processing
- The exact supported `type` values depend on the backend and input type
##### Common Fields
| Field | Type | Description |
|------|------|-------------|
| `type` | `string` | Content type |
| `content` | `dict` | Structured payload for the given `type` |
| `bbox` | `list[int]` | Optional bounding box mapped into the 0-1000 coordinate range |
| `anchor` | `string` | Optional anchor; some `DOCX` titles or index items may include it |
##### Common Types
| Type | Description |
|------|-------------|
| `title` | Title block with `title_content` and `level` |
| `paragraph` | Paragraph block with `paragraph_content` |
| `equation_interline` | Interline formula with `math_content` and `math_type` |
| `image` / `table` / `chart` / `seal` | Visual blocks with image paths, captions, and related structured fields |
| `code` | Code block with `code_content`, `code_caption`, `code_footnote`, and `code_language` |
| `algorithm` | Algorithm block with `algorithm_content`, `algorithm_caption`, and `algorithm_footnote` |
| `list` / `index` | List and index blocks with `list_items` |
| `page_header` / `page_footer` / `page_number` / `page_aside_text` / `page_footnote` | Page auxiliary blocks |
##### Sample Data
```json
[
[
{
"type": "title",
"content": {
"title_content": [
{
"type": "text",
"content": "1 Introduction"
}
],
"level": 1
},
"bbox": [83, 121, 917, 156]
},
{
"type": "page_footnote",
"content": {
"page_footnote_content": [
{
"type": "text",
"content": "* Corresponding author"
}
]
},
"bbox": [71, 815, 915, 841]
}
]
]
```
### VLM Backend Output Results
#### Model Inference Results (model.json)
@@ -644,6 +657,7 @@ Based on the pipeline format, with these VLM-specific extensions:
- All `discarded_blocks` entries are also output (e.g., headers, footers, page numbers, margin notes, page footnotes).
- Existing types (`image`, `table`, `text`, `equation`) remain unchanged.
- `bbox` still uses the 0–1000 normalized coordinate mapping.
- Starting with 3.0, the VLM backend also emits `*_content_list_v2.json`; see the common V2 section above for the shared structure.
##### Examples
@@ -707,6 +721,7 @@ The above files constitute MinerU's complete output results. Users can choose ap
- **Content extraction**: (Use simplified files):
* *.md
* content_list.json
* content_list_v2.json
- **Secondary development**: (Use structured files):
* middle.json
docs/zh/reference/output_files.md
+120 -95
View File
@@ -4,6 +4,8 @@
`mineru` 命令执行后，除了输出主要的 markdown 文件外，还会生成多个辅助文件用于调试、质检和进一步处理。这些文件包括：
具体会生成哪些文件，取决于后端类型和输入文档类型。
- **可视化调试文件**：帮助用户直观了解文档解析过程和结果
- **结构化数据文件**：包含详细的解析数据，可用于二次开发
@@ -60,109 +62,45 @@
**文件命名格式**：`{原文件名}_model.json`
##### 数据结构定义
```python
from pydantic import BaseModel, Field
from enum import IntEnum
class CategoryType(IntEnum):
"""内容类别枚举"""
title = 0 # 标题
plain_text = 1 # 文本
abandon = 2 # 包括页眉页脚页码和页面注释
figure = 3 # 图片
figure_caption = 4 # 图片描述
table = 5 # 表格
table_caption = 6 # 表格描述
table_footnote = 7 # 表格注释
isolate_formula = 8 # 行间公式
formula_caption = 9 # 行间公式的标号
embedding = 13 # 行内公式
isolated = 14 # 行间公式
text = 15 # OCR 识别结果
class PageInfo(BaseModel):
"""页面信息"""
page_no: int = Field(description="页码序号，第一页的序号是 0", ge=0)
height: int = Field(description="页面高度", gt=0)
width: int = Field(description="页面宽度", ge=0)
class ObjectInferenceResult(BaseModel):
"""对象识别结果"""
category_id: CategoryType = Field(description="类别", ge=0)
poly: list[float] = Field(description="四边形坐标，格式为 [x0,y0,x1,y1,x2,y2,x3,y3]")
score: float = Field(description="推理结果的置信度")
latex: str | None = Field(description="LaTeX 解析结果", default=None)
html: str | None = Field(description="HTML 解析结果", default=None)
class PageInferenceResults(BaseModel):
"""页面推理结果"""
layout_dets: list[ObjectInferenceResult] = Field(description="页面识别结果")
page_info: PageInfo = Field(description="页面元信息")
# 完整的推理结果
inference_result: list[PageInferenceResults] = []
```
##### 坐标系统说明
`poly` 坐标格式：`[x0, y0, x1, y1, x2, y2, x3, y3]`
- 分别表示左上、右上、右下、左下四点的坐标
- 坐标原点在页面左上角
![poly 坐标示意图](../images/poly.png)
##### 示例数据
```json
[
{
"layout_dets": [
{
"category_id": 2,
"poly": [
99.1906967163086,
100.3119125366211,
730.3707885742188,
100.3119125366211,
730.3707885742188,
245.81326293945312,
99.1906967163086,
245.81326293945312
],
"score": 0.9999997615814209
}
"cls_id": 12,
"label": "header",
"score": 0.93,
"bbox": [
1217,
104,
1516,
134
],
"page_info": {
"page_no": 0,
"height": 2339,
"width": 1654
}
"index": 2
},
{
"layout_dets": [
{
"category_id": 5,
"poly": [
99.13092803955078,
2210.680419921875,
497.3183898925781,
2210.680419921875,
497.3183898925781,
2264.78076171875,
99.13092803955078,
2264.78076171875
],
"score": 0.9999997019767761
}
"cls_id": 6,
"label": "doc_title",
"score": 0.9751,
"bbox": [
275,
181,
1512,
292
],
"page_info": {
"page_no": 1,
"height": 2339,
"width": 1654
}
"index": 3
},
{
"cls_id": 22,
"label": "text",
"score": 0.9217,
"bbox": [
275,
330,
524,
370
],
"index": 4
}
]
```
@@ -176,7 +114,7 @@ inference_result: list[PageInferenceResults] = []
| 字段名 | 类型 | 说明 |
|--------|------|------|
| `pdf_info` | `list[dict]` | 每一页的解析结果数组 |
| `_backend` | `string` | 解析模式：`pipeline` 或 `vlm` |
| `_backend` | `string` | 解析模式：`pipeline`、`vlm` 或 `office` |
| `_version_name` | `string` | MinerU 版本号 |
##### 页面信息结构 (pdf_info)
@@ -361,8 +299,13 @@ inference_result: list[PageInferenceResults] = []
|------|------|
| `image` | 图片 |
| `table` | 表格 |
| `chart` | 图表 |
| `text` | 文本/标题 |
| `equation` | 行间公式 |
| `seal` | 印章 |
| `code` | 代码块 / 算法块 |
| `list` | 列表 / 参考文献列表 |
| `header` / `footer` / `page_number` / `aside_text` / `page_footnote` | 页面辅助块 |
##### 文本层级标识
@@ -377,6 +320,8 @@ inference_result: list[PageInferenceResults] = []
- 所有内容块都包含 `page_idx` 字段，表示所在页码（从 0 开始）。
- 所有内容块都包含 `bbox` 字段，表示内容块的边界框坐标 `[x0, y0, x1, y1]` 映射在0-1000范围内的结果。
- `code` 类型会通过 `sub_type` 区分 `code` 和 `algorithm`，并可包含 `code_body`、`code_caption`、`code_footnote` 等字段。
- `list` 类型可通过 `sub_type` 区分普通列表和参考文献列表。
##### 示例数据
@@ -443,6 +388,84 @@ inference_result: list[PageInferenceResults] = []
]
```
### 通用内容列表 V2 (content_list_v2.json)
**文件命名格式**：`{原文件名}_content_list_v2.json`
##### 功能说明
`content_list_v2.json` 是 3.0 起新增的结构化输出文件，所有后端都会在保留 `content_list.json` 的同时额外输出该文件：
- 顶层是按页分组的列表，便于按页消费结果
- 每个内容块使用统一的 `type + content` 结构，适合程序化处理
- 不同后端和输入类型支持的 `type` 会有所不同
##### 通用字段
| 字段名 | 类型 | 说明 |
|--------|------|------|
| `type` | `string` | 内容类型 |
| `content` | `dict` | 与 `type` 对应的结构化内容 |
| `bbox` | `list[int]` | 可选，0-1000 范围的边界框 |
| `anchor` | `string` | 可选，部分 `DOCX` 标题或索引项会携带锚点 |
##### 常见类型
| 类型 | 说明 |
|------|------|
| `title` | 标题块，包含 `title_content` 与 `level` |
| `paragraph` | 段落块，包含 `paragraph_content` |
| `equation_interline` | 行间公式，包含 `math_content`、`math_type` |
| `image` / `table` / `chart` / `seal` | 视觉类块，包含图片路径、说明文字等结构化字段 |
| `code` | 代码块，包含 `code_content`、`code_caption`、`code_footnote`、`code_language` |
| `algorithm` | 算法块，包含 `algorithm_content`、`algorithm_caption`、`algorithm_footnote` |
| `list` / `index` | 列表与索引，包含 `list_items` |
| `page_header` / `page_footer` / `page_number` / `page_aside_text` / `page_footnote` | 页面辅助块 |
##### 示例数据
```json
[
[
{
"type": "title",
"content": {
"title_content": [
{
"type": "text",
"content": "1 Introduction"
}
],
"level": 1
},
"bbox": [
83,
121,
917,
156
]
},
{
"type": "page_footnote",
"content": {
"page_footnote_content": [
{
"type": "text",
"content": "* Corresponding author"
}
]
},
"bbox": [
71,
815,
915,
841
]
}
]
]
```
### VLM 后端 输出结果
#### 模型推理结果 (model.json)
@@ -741,6 +764,7 @@ vlm 后端的 content_list.json 文件结构与 pipeline 后端类似，伴随
* `page_number`
* `aside_text`
* `page_footnote`
- 3.0 起，vlm 后端也会同时输出 `*_content_list_v2.json`，其通用结构见上文“通用内容列表 V2”。
##### 示例数据
- code 类型 content
@@ -822,6 +846,7 @@ vlm 后端的 content_list.json 文件结构与 pipeline 后端类似，伴随
- **内容提取**(使用简化文件):
* *.md
* content_list.json
* content_list_v2.json
- **二次开发**(使用结构化文件):
* middle.json