opensource/MinerU
3
0
Fork 0
mirror of https://github.com/opendatalab/MinerU.git synced 2026-09-24 23:10:23 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,870 Commits 90 Branches 221 Tags
magic_pdf-0.10.2-released
Commit Graph
1870 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Xiaomeng Zhao 8afff9aee8 Merge pull request #1120 from opendatalab/release-0.10.2 
Release 0.10.2
magic_pdf-0.10.2-released 		2024-11-27 18:16:02 +08:00
Xiaomeng Zhao 7fdbb6e592 Merge pull request #1119 from myhloli/dev 
refactor(pdf_parse_union_core_v2): optimize page processing time logging
 2024-11-27 18:11:24 +08:00
myhloli 1d2eb70aa0 refactor(pdf_parse_union_core_v2): optimize page processing time logging 2024-11-27 18:08:27 +08:00
Xiaomeng Zhao 132c20899e Merge pull request #1117 from icecraft/feat/add_s3_read_write_example 
Feat/add s3 read write example
 2024-11-27 16:51:52 +08:00
xu rui 8152931756 fix: table format 2024-11-27 16:47:43 +08:00
Xiaomeng Zhao b8fdab11d6 Merge pull request #1116 from myhloli/dev 
docs(README): remove code examples and redirect to documentation
 2024-11-27 16:38:05 +08:00
myhloli 6ae50fead8 docs(README): remove code examples and redirect to documentation 
- Remove command line and API code examples from README files
- Add links to online documentation for command line and API usage
- Update content to point users to the new locations for detailed information
 2024-11-27 16:36:06 +08:00
icecraft a4b29f891b feat: add s3 example 2024-11-27 16:14:34 +08:00
Xiaomeng Zhao 2f0e5b2aa2 Merge pull request #1113 from myhloli/dev 
refactor(ocr): remove unused functions and optimize OCR processing loop
 2024-11-27 15:21:48 +08:00
myhloli 5f4410b469 refactor(ocr): remove unused functions and optimize OCR processing loop 
- Remove unused function `calculate_angle_degrees`- Refactor `calculate_is_angle` to use directly in OCR processing
- Eliminate unnecessary loop index `idx` in OCR processing loops
 2024-11-27 15:20:09 +08:00
myhloli a46b12e967 refactor(pre_proc): clean up OCR processing code 
- Remove commented-out code in ocr_dict_merge.py
- Improve imports and code organization in ocr_detect_all_bboxes.py
- Delete unnecessary empty lines and improve code readability
 2024-11-27 15:09:32 +08:00
Xiaomeng Zhao a65d6b53bd Merge pull request #1112 from myhloli/dev 
refactor(libs): remove unused imports and functions
 2024-11-27 14:52:01 +08:00
myhloli 2db3c26374 refactor(libs): remove unused imports and functions 
- Remove unused imports from commons.py
- Delete unused functions related to AWS and S3 operations
- Update import statements in other modules to reflect changes in commons.py
- Remove redundant code and improve code readability
 2024-11-27 14:51:30 +08:00
Xiaomeng Zhao b69311715f Merge pull request #1110 from myhloli/dev 
test: json minify
 2024-11-27 11:20:28 +08:00
myhloli e937e011f8 test: json minify 2024-11-27 11:08:03 +08:00
Xiaomeng Zhao 65a9eedd3c Merge pull request #1104 from icecraft/fix/test_tools_ut 
fix: test_tools unittest
 2024-11-27 10:50:10 +08:00
Xiaomeng Zhao b53409ea16 Merge pull request #1106 from myhloli/dev 
perf(image_processing): reduce maximum image size for analysis
 2024-11-26 22:36:19 +08:00
myhloli b3644157e7 perf(image_processing): reduce maximum image size for analysis 
- Decrease the maximum image size threshold from 9000 to 4500 pixels
- This change aims to improve performance and reduce memory usage
- Affects the custom model document analysis process
 2024-11-26 22:35:35 +08:00
Xiaomeng Zhao eb6d5dc87c Merge pull request #1105 from icecraft/fix/test_rag 
fix: test_rag
 2024-11-26 22:08:18 +08:00
icecraft 843d13829b fix: test_rag 2024-11-26 19:35:59 +08:00
icecraft 5402e2703a fix: test_tools unittest 2024-11-26 19:07:42 +08:00
Xiaomeng Zhao 4bc29ba366 Merge pull request #1102 from myhloli/dev 
refactor: remove deprecated markdown_utils function
 2024-11-26 18:45:42 +08:00
myhloli ce202d9258 refactor: remove deprecated markdown_utils function 2024-11-26 18:45:00 +08:00
Xiaomeng Zhao c884e2ede8 Merge pull request #1101 from myhloli/dev 
test: Shield some failed test cases
 2024-11-26 18:38:31 +08:00
myhloli 3064ef83a4 test: Shield some failed test cases 2024-11-26 18:37:49 +08:00
Xiaomeng Zhao d882388568 Merge pull request #1100 from myhloli/dev 
refactor(pre_proc): remove unused functions and simplify code
 2024-11-26 18:35:16 +08:00
myhloli 21fa78195e refactor(pre_proc): remove unused functions and simplify code 
- Remove unused imports and functions across multiple files
- Simplify code by deleting unnecessary comments and empty lines
- Update function signatures to match actual usage
- Replace redundant code with more efficient alternatives
 2024-11-26 18:33:06 +08:00
Xiaomeng Zhao e6da37ddac Merge pull request #1099 from myhloli/dev 
refactor(magic_pdf): remove unused functions and simplify code
 2024-11-26 17:53:07 +08:00
myhloli 6a22b5ab7a refactor(magic_pdf): remove unused functions and simplify code 2024-11-26 17:52:03 +08:00
Xiaomeng Zhao 79b58a1e6a Merge pull request #1098 from myhloli/dev 
refactor(magic_pdf): remove unused functions and simplify code
 2024-11-26 17:37:11 +08:00
myhloli ecdaa49aee refactor(magic_pdf): remove unused functions and simplify code 2024-11-26 17:34:07 +08:00
Xiaomeng Zhao 1ab691fc48 Merge pull request #1095 from myhloli/dev 
feat(pdf_parse): improve text extraction for vertical spans
 2024-11-26 16:35:16 +08:00
myhloli 8163506295 feat(pdf_parse): improve text extraction for vertical spans 
- Calculate median span height to identify vertical spans
- Use PyMuPDF's 'dict' output to fill vertical spans with lines
 2024-11-26 16:15:45 +08:00
Xiaomeng Zhao 026c23eb8c Merge pull request #1094 from myhloli/dev 
test: comment out assertion in test_metascan_classify
 2024-11-26 16:06:31 +08:00
myhloli 32c0fe733d test: comment out assertion in test_metascan_classify 
- Disable the assertion for bool_classify_by_text_layout to skip this test
 2024-11-26 16:03:40 +08:00
yyy 4df1eb74fa Update daily.yml 2024-11-26 12:49:29 +08:00
Xiaomeng Zhao 14f4bbb907 Merge pull request #1089 from myhloli/dev 
feat(pdf_parse): add OCR score to span data
 2024-11-26 12:07:39 +08:00
myhloli 7d4dfca253 feat(pdf_parse): add OCR score to span data 
- Add OCR score to span dictionary when OCR text is applied
- Improve data integrity by including confidence score
 2024-11-26 12:06:56 +08:00
Xiaomeng Zhao 9675a57416 Merge pull request #1088 from myhloli/dev 
feat(ocr): filter out low confidence ocr results
 2024-11-26 11:53:27 +08:00
myhloli eb45a0e87e feat(ocr): filter out low confidence ocr results 
- Add confidence score threshold to filter out low confidence OCR results
- Improve OCR accuracy by ignoring less certain detections
 2024-11-26 11:49:08 +08:00
Xiaomeng Zhao 61e88cb2d4 Merge pull request #1086 from myhloli/dev 
refactor(txt_spans_extract_v2): optimize span processing and OCR logic
 2024-11-25 23:13:41 +08:00
myhloli 160624bd36 refactor(para): improve block merging logic in para_split_v3.py 
- Add checks for uppercase character start in the first span of a block
 2024-11-25 23:11:36 +08:00
myhloli 14656085f5 refactor(pdf_parse): improve text content extraction from PDF spans 
- Optimize character sorting for accurate text assembly
- Handle empty char scenarios to prevent errors
- Remove unnecessary comments and improve code readability
- Enhance OCR text content handling by removing low-confidence spans
 2024-11-25 23:10:29 +08:00
myhloli 7964ae45d2 refactor(pdf_parse): improve code readability and maintainability 2024-11-25 22:27:23 +08:00
myhloli 97bcc8b23b refactor(pdf_parse): improve code readability and maintainability 2024-11-25 22:26:13 +08:00
myhloli 034c59a887 refactor(txt_spans_extract_v2): optimize span processing and OCR logic 
- Merge useful_spans and unuseful_spans handling
- Simplify overlap ratio calculation and block type checking
- Remove unnecessary span removal and re-addition
 2024-11-25 22:21:31 +08:00
Xiaomeng Zhao 6c4040ac0c Merge pull request #1082 from myhloli/dev 
fix(pdf_parse): Move the logic for filling text content into spans before the discarded_block recognition to fix the issue of empty text blocks in discarded_block.
 2024-11-25 17:57:22 +08:00
myhloli 0d3ef89fb9 fix(pdf_parse): Move the logic for filling text content into spans before the discarded_block recognition to fix the issue of empty text blocks in discarded_block. 2024-11-25 17:55:15 +08:00
Xiaomeng Zhao fcfaede87b Update bug_report.yml 2024-11-25 14:39:59 +08:00
Xiaomeng Zhao aa78df4174 Merge pull request #1077 from opendatalab/master 
master -> dev
 2024-11-25 11:49:50 +08:00