- Update model download instructions for versions 0.9.x and later
- Simplify demo scripts by removing unnecessary model configuration
- Add visualization function to draw bounding boxes
- Update CLI help message with new URL
- Add new layout model option: DocLayout-YOLO
- Implement model initialization and prediction for DocLayout-YOLO
- Update configuration options to include new model- Modify existing code to support both LayoutLMv3 and DocLayout-YOLO models
- Update Gradio app to support more Custom Switch
Refactor the draw bbox functions by removing unused imports and simplifying the
code logic for drawing layout and line sorting bounding boxes. Adjust the debug
configuration to enable content list dumping and disable markdown making mode.
Introduce an additional argument `draw_bbox` in the `draw_bbox_with_number` function to
enable toggling the drawing of bounding boxes on or off. When set to `False`, no bounding
box will be drawn, allowing for situations where only text
Add a new function `draw_line_sort_bbox` to visualize the sorting of lines on each page.
This includes indexing lines and handling both text and non-text elements such as tables
and images for better content organization.
Also, comment out GPU-related code for flexibility and remove overlaps in bounding box
detection, which improves the accuracy of layout splitting.
Implement a new function `draw_layout_sort_bbox` in `draw_bbox.py` to visualize the
layout sorting results using the `LayoutLMv3ForTokenClassification` model. This function
predicts the order of layout elements and draws them in the sorted sequence on the PDF pages.
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
* fix(ocr_mkcontent): improve language detection and content formatting (#458)
Optimize the language detection logic to enhance content formatting. This
change addresses issues with long word segmentation. Language detection now uses a
threshold to determine the language of a text based on the proportion of English characters.
Formatting rules for content have been updated to consider a list of languages (initially
including Chinese, Japanese, and Korean) where no space is added between content segments
for inline equations and text spans, improving the handling of Asian languages.
The impact of these changes includes improved accuracy in language detection, better
segmentation of long words, and more appropriate spacing in content formatting for multiple
languages.
* fix(self_modify): merge detection boxes for optimized text region detection (#448)
Merge adjacent and overlapping detection boxes to optimize text region detection in
the document. Post processing of text boxes is enhanced by consolidating them into
larger text lines, taking into account their vertical and horizontal alignment. This
improvement reduces fragmentation and improves the readability of detected text blocks.
* fix(pdf-extract): adjust box threshold for OCR detection (#447)
Tuned the detection box threshold parameter in the OCR model initialization to improve the
accuracy of text extraction from images. The threshold was modified from 0.6 to
0.3 to filter out smaller detection boxes, which is expected to enhance the quality of the extracted
text by reducing noise and false positives in the OCR process.
* feat: rename the file generated by command line tools (#401)
* feat: rename the file generated by command line tools
* feat: add pdf filename as prefix to {span,layout,model}.pdf
---------
Co-authored-by: icecraft <tmortred@gmail.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* fix(ocr_mkcontent): revise table caption output (#397)
* fix(ocr_mkcontent): revise table caption output
- Ensuring that
table captions are properly included in the output.
- Remove the redundant `table_caption` variable。
* Update cla.yml
* Update bug_report.yml
* feat(cli): add debug option for detailed error handling
Enable users to invoke the CLI command with a new debug flag to get detailed debugging information.
* fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified
to improve the accuracy and consistency of OCR processing. The new values are set to 25
to ensure more precise image cropping and pasting which leads to better OCR recognition
results.
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* fix(pdf-extract-kit): increase crop_paste margin for OCR processingDouble the crop_paste margin from25 to 50 to ensure better OCR accuracy and
handling of border cases. This change will help in improving the overall quality of
OCR'ed text by providing more context around the detected text areas.
* fix(common): deep copy model list before drawing model bbox
Use a deep copy of the original model list in `drow_model_bbox` to avoid potential
modifications to the source data. This ensures the integrity of the original models
is maintained while generating the model bounding boxes visualization.
---------
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* build(docker): update docker build step (#471)
* build(docker): update base image to Ubuntu 22.04 and install PaddlePaddleUpgrade the Docker base image from ubuntu:latest to ubuntu:22.04 for improved
performance and stability.
Additionally, integrate PaddlePaddle GPU version 3.0.0b1
into the Docker build for enhanced AI capabilities. The MinIO configuration file has
also been updated to the latest version.
* build(dockerfile): Updated the Dockerfile
* build(Dockerfile): update Dockerfile
* docs(docker): add instructions for quick deployment with Docker
Include Docker-based deployment instructions in the README for both English and
Chinese locales. This update provides users a quick-start guide to using Docker for
deployment, with notes on GPU VRAM requirements and default acceleration features.
* build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself.
* build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself.
* upload an introduction about chemical formula and update readme.md (#489)
* upload an introduction about chemical formula
* rename 2 files
* update readme.md at TODO in chemstery
* rename 2 files and update readme.md at TODO in chemstery
* update README_zh-CN.md at TODO in chemstery
* upload an introduction about chemical formula and update readme.md (#489)
* upload an introduction about chemical formula
* rename 2 files
* update readme.md at TODO in chemstery
* rename 2 files and update readme.md at TODO in chemstery
* update README_zh-CN.md at TODO in chemstery
* fix: remove the default value of output option in tools/cli.py and tools/cli_dev.py (#494)
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* feat: add test case (#499)
Co-authored-by: quyuan <quyuan@pjlab.org>
* Update cla.yml
* Update gpu-ci.yml
* Update cli.yml
* Delete .github/workflows/gpu-ci.yml
* fix(pdf-parse-union-core): #492 decrease span threshold for block filling (#500)
Reduce the span threshold used in fill_spans_in_blocks from 0.6 to 0.3 to
improve the accuracy of block filling based on layout analysis.
* fix(detect_all_bboxes): remove small overlapping blocks by merging (#501)
Previously, small blocks that overlapped with larger ones were merely removed. This fix
changes the approach to merge smaller blocks into the larger block instead, ensuring that
no information is lost and the larger block encompasses all the text content fully.
* feat(cli&analyze&pipeline): add start_page and end_page args for pagination (#507)
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* Feat/support rag (#510)
* Create requirements-docker.txt
* feat: update deps to support rag
* feat: add support to rag, add rag_data_reader api for rag integration
* feat: let user retrieve the filename of the processed file
* feat: add projects demo for rag integrations
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* Update Dockerfile
* feat(gradio): add app by gradio (#512)
* fix: replace \u0002, \u0003 in common text (#521)
* fix replace \u0002, \u0003 in common text
* fix(para): When an English line ends with a hyphen, do not add a space at the end.
* fix(end_page_id):Fix the issue where end_page_id is corrected to len-1 when its input is 0. (#518)
* fix(para): When an English line ends with a hyphen, do not add a space at the end. (#523)
* fix replace \u0002, \u0003 in common text
* fix(para): When an English line ends with a hyphen, do not add a space at the end.
* fix: delete hyphen at end of line
* Release: Release 0.7.1 verison, update dev (#527)
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#493)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#508)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* Update cla.yml
* Delete .github/workflows/gpu-ci.yml
* Update Huggingface and ModelScope links to organization account
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* Update cla.yml
* Delete .github/workflows/gpu-ci.yml
* Update Huggingface and ModelScope links to organization account
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: wangbinDL <wangbin_research@163.com>
---------
Co-authored-by: Kaiwen Liu <lkw_buaa@163.com>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* Hotfix readme 0.7.1 (#529)
* release: release 0.7.1 version (#526)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#493)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#508)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* Update cla.yml
* Delete .github/workflows/gpu-ci.yml
* Update Huggingface and ModelScope links to organization account
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* Update cla.yml
* Delete .github/workflows/gpu-ci.yml
* Update Huggingface and ModelScope links to organization account
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: wangbinDL <wangbin_research@163.com>
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: Kaiwen Liu <lkw_buaa@163.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* Update README.md
* Update README_zh-CN.md
* Update README_zh-CN.md
---------
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: Kaiwen Liu <lkw_buaa@163.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* Update README_zh-CN.md
delete Known issue about table recognition
* Update Dockerfile
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 (#542)
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: typo error in markdown (#536)
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* fix(gradio): remove unused imports and simplify pdf display (#534)
Removed the previously used gradio and gradio-pdf imports which were not leveraged in the code. Also,
replaced the custom `show_pdf` function with direct use of the `PDF` component from gradio for a simpler
and more integrated PDF upload and display solution, improving code maintainability and readability.
* Feat/support footnote in figure (#532)
* feat: support figure footnote
* feat: using the relative position to combine footnote, table, image
* feat: add the readme of projects
* fix: code spell in unittest
---------
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* refactor(pdf_extract_kit): implement singleton pattern for atomic models (#533)
Refactor the pdf_extract_kit module to utilize a singleton pattern when initializing
atomic models. This change ensures that atomic models are instantiated at most once,
optimizing memory usage and reducing redundant initialization steps. The AtomModelSingleton
class now manages the instantiation and retrieval of atomic models, improving the
overall structure and efficiency of the codebase.
* Update README.md
* Update README_zh-CN.md
* Update README_zh-CN.md
add HF、modelscope、colab url
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update README_zh-CN.md
* Rename README.md to README_zh-CN.md
* Create readme.md
* Rename readme.md to README.md
* Rename README.md to README_zh-CN.md
* Update README_zh-CN.md
* Create README.md
* Update README.md
* Update README.md
* Update README.md
* Update README_zh-CN.md
* Update README.md
* Update README_zh-CN.md
* Update README_zh-CN.md
* Update README.md
* Update README_zh-CN.md
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 (#573)
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* Update README_zh-CN.md
* Update README.md
* Update README.md
* Update README.md
* Update README_zh-CN.md
* add rag data api
* Update README_zh-CN.md
update rag api image
* Update README.md
docs: remove RAG related release notes
* Update README_zh-CN.md
docs: remove RAG related release notes
* Update README_zh-CN.md
update 更新记录
---------
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: icecraft <tmortred@163.com>
Co-authored-by: icecraft <tmortred@gmail.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Siyu Hao <131659128+GDDGCZ518@users.noreply.github.com>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: quyuan <quyuan@pjlab.org>
Co-authored-by: Kaiwen Liu <lkw_buaa@163.com>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: wangbinDL <wangbin_research@163.com>
Pass the `lang` parameter to `custom_model_init` in `doc_analyze` to support language-specific OCR configurations. This enhancement allows the use of language information to improve OCR accuracy when processing PDFs.
* Create requirements-docker.txt
* feat: update deps to support rag
* feat: add support to rag, add rag_data_reader api for rag integration
* feat: let user retrieve the filename of the processed file
* feat: add projects demo for rag integrations
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* fix(ocr_mkcontent): revise table caption output
- Ensuring that
table captions are properly included in the output.
- Remove the redundant `table_caption` variable。
* Update cla.yml
* Update bug_report.yml
* feat(cli): add debug option for detailed error handling
Enable users to invoke the CLI command with a new debug flag to get detailed debugging information.
* fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified
to improve the accuracy and consistency of OCR processing. The new values are set to 25
to ensure more precise image cropping and pasting which leads to better OCR recognition
results.
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* fix(pdf-extract-kit): increase crop_paste margin for OCR processingDouble the crop_paste margin from25 to 50 to ensure better OCR accuracy and
handling of border cases. This change will help in improving the overall quality of
OCR'ed text by providing more context around the detected text areas.
* fix(common): deep copy model list before drawing model bbox
Use a deep copy of the original model list in `drow_model_bbox` to avoid potential
modifications to the source data. This ensures the integrity of the original models
is maintained while generating the model bounding boxes visualization.
---------
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* feat: rename the file generated by command line tools
* feat: add pdf filename as prefix to {span,layout,model}.pdf
---------
Co-authored-by: icecraft <tmortred@gmail.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
Implement the feature to draw bounding boxes for model elements in the PDF. This includes
adding new drawing functions and modifying existing ones to accommodate the new feature.
Also, updates are made to CLI tools and common utilities to support the model bbox drawing.
xu rui