- Add LLM-aided formula and text correction functionality
- Update config reader to include LLM-aided settings
- Create new LLM-aided processing module
- Update main processing script to incorporate LLM-aided corrections
- Modify download scripts to check for new config version
- Move ligature replacement function to pdf_parse_union_core_v2.py
- Optimize ligature replacement using a more efficient approach
- Modify text extraction flags to preserve ligatures in PDF content
- Remove unnecessary function from ocr_mkcontent.py
- In Chinese, Japanese, and Korean (CJK) languages, no space is needed for line breaks within paragraphs.
- However, if an inline equation is at the end of a line, a space should be added to separate it from the following text.
- This change improves the formatting of documents containing both CJK text and inline equations.
- Extract language detection to block level instead of line level
- Improve logic for handling Chinese, Japanese, and Korean languages
- Refactor code for better readability and performance
- Optimize handling of hyphenated words at line ends
- Introduce language detection to determine line spacing based on language context
- Implement different spacing rules for Chinese/Japanese/Korean and Western texts
- Adjust span content handling based on detected language and span type
- Add empty paragraph handling for pages with no content
- Append an empty markdown object when a page has no paragraphs
- Increment page number even if no content is present
- Remove unused language detection code
- Simplify text content processing logic
- Update span sorting and text extraction in pdf_parse_union_core_v2.py
- Implement __replace_ligatures function to split ligature characters- Integrate ligature replacement into the merge_para_with_text function
- Handle common ligatures such as fi, fl, ff, ffi, and ffl
- Optimize content stripping and checking logic
- Add special case handling for single-character content
- Adjust spacing rules for different content types
- Include InlineEquation in the condition for handling text content
- Remove separate block for InlineEquation processing
- Ensures consistent handling of inline equations and text, improving content formatting
- Add check for 'image_path' in spans to avoid errors when it's missing
- Update image handling in both paragraph text and content dictionary
- Improve error handling and make the code more robust
- Add support for drawing bounding boxes of table and image sub-blocks
- Implement sorting of table blocks based on type order
- Update bounding box drawing for text and title blocks
- Refactor code to handle different block types and their sub-blocks
- Split image and table blocks into separate categories
- Add group_id to image and table blocks- Update block processing logic to handle new categories
- Modify layout splitting and span filling to accommodate new block types
- Adjust block indexing and sorting to consider new structures
- Remove unused parameters parse_type and lang from various functions
- Simplify function calls by removing unnecessary arguments
- Update related files to reflect these changes
- Remove unused functions such as split_long_words, ocr_mk_mm_markdown_with_para, etc.
- Simplify ocr_mk_markdown_with_para_core_v2 by removing unnecessary language detection and word splitting logic- Remove wordninja dependency from requirements
- Update ocr_model_init to include additional parameters for OCR model configuration
- Add detection for list and index blocks in OCR processing- Implement merging of list and index blocks across pages
- Update block types to include list and index categories
- Adjust text merging logic to handle new block types
- Modify layout drawing to distinguish list and index blocks
function signature to include the `drop_reason` parameter and handling it within the
function logic. This enhancement allows the function to convey the reason for dropping
content in the output.
The pipeline now supports passing the language parameter to parsing functions and
during markdown conversion to optimize processing based on the specified language.
This enhancement allows for more accurate parsing and markdown generation, particularly
when dealing with non-English content.
* feat: support figure footnote
* feat: using the relative position to combine footnote, table, image
* feat: add the readme of projects
* fix: code spell in unittest
---------
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* fix(ocr_mkcontent): revise table caption output
- Ensuring that
table captions are properly included in the output.
- Remove the redundant `table_caption` variable。
* Update cla.yml
* Update bug_report.yml
* feat(cli): add debug option for detailed error handling
Enable users to invoke the CLI command with a new debug flag to get detailed debugging information.
* fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified
to improve the accuracy and consistency of OCR processing. The new values are set to 25
to ensure more precise image cropping and pasting which leads to better OCR recognition
results.
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* fix(pdf-extract-kit): increase crop_paste margin for OCR processingDouble the crop_paste margin from25 to 50 to ensure better OCR accuracy and
handling of border cases. This change will help in improving the overall quality of
OCR'ed text by providing more context around the detected text areas.
* fix(common): deep copy model list before drawing model bbox
Use a deep copy of the original model list in `drow_model_bbox` to avoid potential
modifications to the source data. This ensures the integrity of the original models
is maintained while generating the model bounding boxes visualization.
---------
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Optimize the language detection logic to enhance content formatting. This
change addresses issues with long word segmentation. Language detection now uses a
threshold to determine the language of a text based on the proportion of English characters.
Formatting rules for content have been updated to consider a list of languages (initially
including Chinese, Japanese, and Korean) where no space is added between content segments
for inline equations and text spans, improving the handling of Asian languages.
The impact of these changes includes improved accuracy in language detection, better
segmentation of long words, and more appropriate spacing in content formatting for multiple
languages.
Ensure proper formatting of inline equations by adding spaces outside the equation delimitersto prevent markdown from interpreting the equation content as part of a link. This addresses
the issue where inline OCR equations appear without the correct markdown formatting.
* # add table recognition using struct-eqtable
## Changelog
31/07/20204
- Support table recognition. Table images will be converted into html.
### how to use the new feature:
set the attribute 'table-mode' to 'true' in magic-pdf.json
### caution:
it takes 200s to 500s to convert a single table image using cpu
* # add table recognition using struct-eqtable
## Changelog
31/07/20204
- Support table recognition. Table images will be converted into LaTex.
### how to use the new feature:
set the attribute 'table-mode' to 'true' in magic-pdf.json
### caution:
it takes 200s to 500s to convert a single table image using cpu
* # feat(model inference): add table recognition and convertion to LaTeX
# What's Changed
### New Features
- Add table content recognition, we use weights of [StructEqTable](https://github.com/UniModal4Reasoning/StructEqTable-Deploy) to convert table image to LaTex.
### Instruction
- pip install pypandoc struct-eqtable==0.1.0
- Download [StructEqTable weights](https://huggingface.co/wanderkid/PDF-Extract-Kit/tree/main/models/TabRec) and put it under models/ directory.
- Edit 'table-mode' value to turn on table recognition function which is turned off by default.
- If you did not download any models before, refer to [how to download models](docs/how_to_download_models_zh_cn.md)。
* add table recognition and convertion to LaTeX
* add table recognition and conversion to LaTeX
* add table recognition and conversion to LaTeX
* add table recognition and conversion to LaTeX
---------
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
myhloli