Commit Graph
2016 Commits
Author SHA1 Message Date
myhloli c660fdc8f0 feat(llm): add LLM-aided formula and text correction
- Add LLM-aided formula and text correction functionality
- Update config reader to include LLM-aided settings
- Create new LLM-aided processing module
- Update main processing script to incorporate LLM-aided corrections
- Modify download scripts to check for new config version
2024-12-24 14:33:35 +08:00
myhloli c638fc5d1f fix(pdf): improve ligature handling and text extraction
- Move ligature replacement function to pdf_parse_union_core_v2.py
- Optimize ligature replacement using a more efficient approach
- Modify text extraction flags to preserve ligatures in PDF content
- Remove unnecessary function from ocr_mkcontent.py
2024-12-13 15:43:05 +08:00
myhloli 6a75d7dce5 perf(layout): optimize layout detection for PDF extraction
- Add initial setup for layout detection
- Implement conditional cropping for tall images
- Skip cropping for wide images to improve performance
- Reuse Image object across layout detection steps
2024-12-12 12:10:05 +08:00
Xiaomeng Zhao 56b0e18b22 Merge pull request #1268 from icecraft/fix/classify 2024-12-12 00:01:58 +08:00
xu rui 712d7d4a8d fix: classif pdf type 2024-12-11 19:50:35 +08:00
Xiaomeng Zhao bdacf29179 Merge pull request #1257 from icecraft/docs/refactor_en_docs
Docs/refactor en docs
2024-12-11 19:09:41 +08:00
Xiaomeng Zhao 2df3e90179 Merge pull request #1267 from opendatalab/master
master->dev
2024-12-11 19:06:58 +08:00
myhloli 391a99860d Update version.py with new version 2024-12-11 10:56:15 +00:00
Xiaomeng Zhao 613074b896 Merge pull request #1266 from opendatalab/release-0.10.6
Release 0.10.6
magic_pdf-0.10.6-released
2024-12-11 18:47:46 +08:00
Xiaomeng Zhao 1c29f99e5a Merge pull request #1265 from opendatalab/dev
Dev->release
2024-12-11 18:47:09 +08:00
Xiaomeng Zhao a502c5c948 Merge pull request #1264 from myhloli/dev
build(docker): add torch and torchvision dependencies
2024-12-11 18:46:32 +08:00
myhloli 28dae58839 build(docker): add torch and torchvision dependencies
- Add torch>=2.2.2,<=2.3.1 to requirements-docker.txt- Add torchvision>=0.17.2,<=0.18.1 to requirements-docker.txt
2024-12-11 18:45:31 +08:00
Xiaomeng Zhao b4f7b53ecb Merge pull request #1261 from opendatalab/release-0.10.6
Release 0.10.6
2024-12-11 18:40:28 +08:00
Xiaomeng Zhao d3b51aa556 Merge pull request #1263 from opendatalab/dev
refactor(draw_bbox): remove redundant '_line_sort' suffix from output filename
2024-12-11 18:34:01 +08:00
Xiaomeng Zhao fcba88b56f Merge pull request #1262 from myhloli/dev
refactor(draw_bbox): remove redundant '_line_sort' suffix from output  filename
2024-12-11 18:33:07 +08:00
myhloli ef78819aa9 refactor(draw_bbox): remove redundant '_line_sort' suffix from output filename
- Updated the filename generation logic in the draw_bbox function
- Removed the unnecessary '_line_sort' suffix from the output PDF filename
2024-12-11 18:31:51 +08:00
myhloli 9efc35ecaa refactor(magic_pdf): remove unused import in pdf_parse_union_core_v2.py
- Remove unused import of ocr_model_init from magic_pdf.model.sub_modules.model_init
- Keep existing functionality and structure intact
2024-12-11 18:13:03 +08:00
Xiaomeng Zhao 0440ee8781 Merge pull request #1260 from opendatalab/dev
fix: dup classify pdf type & improve layout detection for DocLayout_YOLO model
2024-12-11 18:02:43 +08:00
Xiaomeng Zhao 327fdf906f Merge pull request #1259 from myhloli/dev
feat(layout): improve layout detection for DocLayout_YOLO model
2024-12-11 17:46:58 +08:00
myhloli f5d812b313 feat(layout): improve layout detection for DocLayout_YOLO model
- Implement image cropping and pasting technique to enhance layout detection
- Adjust detected polygons to original image coordinates
- Add comments for better code readability
2024-12-11 17:28:00 +08:00
xu rui 302a695078 feat: remove pipe_auto_mode 2024-12-11 17:00:17 +08:00
icecraft 3062217d41 fix: fix ut 2024-12-11 16:56:25 +08:00
xu rui b04867f90a docs: check links in doc 2024-12-11 16:56:25 +08:00
xu rui cece8f5375 feat: support ms-office and images file in command line tools 2024-12-11 16:56:25 +08:00
xu rui 7dc3b0a9a2 docs: add quick_start example 2024-12-11 16:56:25 +08:00
xu rui 1d32722f5b fix: not create empty directory 2024-12-11 16:56:25 +08:00
xu rui f6af67eb11 feat: support convert ppt/pptx/doc/docx 2024-12-11 16:56:25 +08:00
xu rui f3ceebc41b fix: read_api list files 2024-12-11 16:56:25 +08:00
xu rui 3cd51d4941 feat: rewrite code snippet 2024-12-11 16:56:25 +08:00
xu rui 6ca86beaa3 docs: rewrite install and usage docs 2024-12-11 16:56:25 +08:00
Xiaomeng Zhao fd2f3c58da Merge pull request #1258 from icecraft/fix/dup_classify
fix: dup classify pdf type
2024-12-11 16:47:32 +08:00
icecraft 4e7511fb86 fix: dup classify pdf type 2024-12-11 16:42:16 +08:00
Xiaomeng Zhao fb4686711e Merge pull request #1256 from opendatalab/dev
build(deps): update torch and torchvision version requirements
2024-12-11 14:42:43 +08:00
Xiaomeng Zhao 168a111544 Merge pull request #1255 from myhloli/dev
build(deps): update torch and torchvision version requirements
2024-12-11 14:32:43 +08:00
myhloli 9a96362db7 build(deps): update torch and torchvision version requirements
- Specify torch==2.3.1 and torchvision==0.18.1 for Windows CUDA installation
- Add torch and torchvision version constraints in setup.py:
  - torch>=2.2.2,<=2.3.1
  - torchvision>=0.17.2,<=0.18.1
- Update installation instructions in both English and Chinese README files
2024-12-11 14:31:49 +08:00
Xiaomeng Zhao fdf15a45d6 Merge pull request #1252 from myhloli/dev
fix(detect_invalid_chars):fix the stack error caused by multiple memory releases in PyMuPDF
2024-12-11 01:54:59 +08:00
myhloli 023ed9c8b8 build: enable pdfminer.six dependency
- Uncomment pdfminer.six in requirements.txt
- Specify version 20231228 for pdfminer.six
2024-12-11 01:50:04 +08:00
myhloli 061c03a0db refactor(model): update import paths for PaddleOCR modules
- Change import paths from paddleocr.ppocr to ppocr for utility functions
- Update import paths for logging and utility modules in ppocr_273_mod.py- Modify import paths for tablemaster_paddle.py to use ppstructure instead of paddleocr.ppstructure
2024-12-11 01:49:24 +08:00
myhloli e1be7da644 refactor(magic_pdf): switch to pdfminer for invalid character detection
- Replace MuPDF with pdfminer for detecting invalid characters in PDFs
- Uncomment and update the detect_invalid_chars function to use pdfminer
- Update the check_invalid_chars function in pdf_meta_scan.py to use the new implementation
2024-12-11 01:41:35 +08:00
myhloli 01cd633ded refactor(tablemaster): update import paths for TableSystem and init_args
- Change import path for TableSystem from 'ppstructure.table.predict_table' to 'paddleocr.ppstructure.table.predict_table'
- Change import path for init_args from 'ppstructure.utility' to 'paddleocr.ppstructure.utility'
2024-12-11 01:26:18 +08:00
myhloli 56fad23d67 refactor(magic_pdf): update paddleocr module import paths
- Modify import paths for paddleocr utilities in ocr_utils.py and ppocr_273_mod.py
- Change from `ppocr.utils.utility` to `paddleocr.ppocr.utils.utility`
- Update related import statements in two files to reflect the new path
2024-12-11 01:24:02 +08:00
myhloli 52dfdd5354 refactor(magic_pdf): remove unnecessary comment
- Remove commented-out call to clean_memory() function
- This change simplifies the code by eliminating an unused code snippet
2024-12-10 20:05:22 +08:00
myhloli dd7f6781f6 fix(magic_pdf): disable PaddlePaddle signal handler
- Import paddle module and disable its signal handler to prevent interference with other components
- This change addresses potential conflicts between PaddlePaddle and other libraries or system signals
2024-12-10 18:11:59 +08:00
myhloli 2b6e94424c refactor: comment out clean_memory function call
- Remove the call to clean_memory() function from pdf_parse_union_core_v2.py
- This change may affect memory usage and needs to be tested to ensure proper functionality
2024-12-10 17:50:51 +08:00
Xiaomeng Zhao 8dbfea6d1e Merge pull request #1239 from myhloli/dev
docs(windows): update CUDA installation guide
2024-12-09 18:12:28 +08:00
myhloli ede7d361a4 docs(windows): update CUDA installation guide
- Remove specific version requirements for torch and torchvision
- Simplify installation command in both English and Chinese guides
- Delete important note about version compatibility
2024-12-09 18:11:24 +08:00
Xiaomeng Zhao 4355b6e03f Merge pull request #1238 from myhloli/dev
refactor(magic_pdf): optimize environment setup and dependencies
2024-12-09 18:09:40 +08:00
myhloli a296ea41f9 refactor(magic_pdf): optimize environment setup and dependencies
- Add environment variables to disable albumentations and yolo updates
- Import torchtext and disable deprecation warnings
- Update unimernet to 0.2.2
- Specify ultralytics version as >=8.3.48
- Remove upper version limit for torch
2024-12-09 18:08:27 +08:00
Xiaomeng Zhao 5c3bf21e31 Merge pull request #1232 from myhloli/dev
build(deps): update dependency versions
2024-12-09 14:27:24 +08:00
myhloli 2ae1039408 build(deps): update dependency versions
- Update ultralytics to >=8.3.47
2024-12-09 14:26:48 +08:00