1919 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
xu rui	91d825b29a	docs: fix table format	2024-12-03 14:06:58 +08:00
xu rui	f6bd47de6a	docs: add dataset method description	2024-12-03 14:06:58 +08:00
xu rui	d44e7a28f4	refactor: add docs	2024-12-03 14:06:58 +08:00
icecraft	4a82d6a07a	feat: add function definitions	2024-12-03 14:06:58 +08:00
icecraft	a3a720ea87	refactor: isolate inference and pipeline	2024-12-03 14:06:58 +08:00
Xiaomeng Zhao	fdf47155e3	Merge pull request #1170 from opendatalab/master ... master->dev	2024-12-02 14:26:41 +08:00
myhloli	b9f3435cb7	Update version.py with new version	2024-12-02 06:14:50 +00:00
Xiaomeng Zhao	c175001d6a	Merge pull request #1165 from opendatalab/release-0.10.5 ... Release 0.10.5 magic_pdf-0.10.5-released	2024-12-02 11:44:51 +08:00
Xiaomeng Zhao	a35785b9e5	Merge pull request #1167 from opendatalab/dev ... Dev -> 0.10.5	2024-12-02 11:44:16 +08:00
Xiaomeng Zhao	a7296f78f0	Merge pull request #1166 from myhloli/dev ... fix(pre_proc): prevent errors when imageWriter is None	2024-12-02 11:41:57 +08:00
Xiaomeng Zhao	ed822634df	Merge pull request #1164 from myhloli/dev ... refactor(para): adjust line height multiplier for block splitting,fix(pre_proc): prevent errors when imageWriter is None	2024-12-02 11:32:54 +08:00
myhloli	b0529b6fbd	fix: reduce maximum image size ... - Decrease the maximum width and height from 9000 to 4500 pixels - This change aims to prevent excessive resource usage when rendering PDFs	2024-12-02 11:27:22 +08:00
myhloli	7f8dc353b0	fix(pre_proc): prevent errors when imageWriter is None ... - Updated cut_image.py to check for NoneType imageWriter - Prevents AttributeError when imageWriter is not provided	2024-12-02 11:20:49 +08:00
Xiaomeng Zhao	384e03799c	Merge pull request #1156 from myhloli/dev ... refactor(para): adjust line height multiplier for block splitting	2024-12-01 02:06:17 +08:00
myhloli	41545a13c6	refactor(para): adjust line height multiplier for block splitting ... - Decrease the line height multiplier from 0.8 to 0.7 for both left and right sides - This modification aims to improve the accuracy of paragraph splitting	2024-12-01 02:05:49 +08:00
Xiaomeng Zhao	b17084febb	Merge pull request #1154 from LollipopsAndWine/dev	2024-11-30 17:05:52 +08:00
houlinfeng	f11f3d6032	fix: 修复文件名错误	2024-11-30 16:52:03 +08:00
myhloli	f8828be7eb	Update version.py with new version	2024-11-29 18:48:40 +00:00
Xiaomeng Zhao	b03a7fae5e	Merge pull request #1153 from opendatalab/release-0.10.4 ... Release 0.10.4 magic_pdf-0.10.4-released	2024-11-30 02:47:28 +08:00
Xiaomeng Zhao	9726403c69	Merge pull request #1152 from myhloli/dev ... fix(mkcontent): optimize paragraph text merging and language detection	2024-11-30 02:45:02 +08:00
myhloli	b3127233f0	refactor: modify bbox processing for layout separation ... - Remove overlap between bboxes for block separation - Sort bboxes by combined x and y coordinates for better layout handling - Comment out previous overlap removal function	2024-11-30 02:33:26 +08:00
myhloli	b80befe9cf	refactor(mkcontent): optimize paragraph text merging and language detection ... - Extract language detection to block level instead of line level - Improve logic for handling Chinese, Japanese, and Korean languages - Refactor code for better readability and performance - Optimize handling of hyphenated words at line ends	2024-11-30 02:16:38 +08:00
myhloli	ea35fa6b60	Merge remote-tracking branch 'origin/dev' into dev	2024-11-30 01:14:26 +08:00
myhloli	c8cabb3cf6	feat(ocr_mkcontent): add language detection for line spacing ... - Introduce language detection to determine line spacing based on language context - Implement different spacing rules for Chinese/Japanese/Korean and Western texts - Adjust span content handling based on detected language and span type	2024-11-30 01:14:12 +08:00
Xiaomeng Zhao	78c9014073	Merge pull request #1147 from opendatalab/master ... master->dev	2024-11-29 16:44:47 +08:00
myhloli	d19911f113	Update version.py with new version	2024-11-29 08:03:01 +00:00
Xiaomeng Zhao	b3fbedf055	Merge pull request #1143 from opendatalab/release-0.10.3 ... Release 0.10.3 magic_pdf-0.10.3-released	2024-11-29 16:01:36 +08:00
Xiaomeng Zhao	66bd0f8b69	Merge pull request #1141 from myhloli/dev ... refactor(ocr): Fix the error of paddleocr failing to initialize in a multi-threaded environment	2024-11-29 12:03:48 +08:00
myhloli	7f2f2c0f28	refactor(ocr): Fix the error of paddleocr failing to initialize in a multi-threaded environment	2024-11-29 12:02:48 +08:00
Xiaomeng Zhao	68c455309d	Merge pull request #1140 from myhloli/dev ... refactor(pdf_parse): adjust character-axis alignment algorithm	2024-11-29 12:00:36 +08:00
myhloli	d4345b6e39	refactor(pdf_parse): adjust character-axis alignment algorithm ... - Introduce `span_height_radio` parameter to calculate_char_in_span function - Replace fixed ratio with dynamic ratio for character and span axis alignment - Improve flexibility and accuracy of character placement within spans	2024-11-29 11:59:52 +08:00
Xiaomeng Zhao	086b48b7ae	Merge pull request #1139 from myhloli/dev ... fix(ocr_mkcontent): handle empty paragraphs on pages	2024-11-29 11:59:03 +08:00
myhloli	782e6571bc	fix(ocr_mkcontent): handle empty paragraphs on pages ... - Add empty paragraph handling for pages with no content - Append an empty markdown object when a page has no paragraphs - Increment page number even if no content is present	2024-11-29 11:58:34 +08:00
Xiaomeng Zhao	4adabc37ac	Merge pull request #1138 from myhloli/dev ... feat(pdf_parse): add line start flag detection and optimize line stop flag logic	2024-11-28 23:13:47 +08:00
myhloli	949d0867fb	feat(pdf_parse): add line start flag detection and optimize line stop flag logic ... - Add LINE_START_FLAG tuple to identify starting flags of a line - Modify calculate_char_in_span function to handle both line start and stop flags - Remove redundant char_is_line_stop_flag variable and simplify logic - Improve line flag detection to enhance text extraction accuracy	2024-11-28 23:12:37 +08:00
Xiaomeng Zhao	a1cff28c74	Merge pull request #1137 from myhloli/dev ... refactor(pdf_check): improve character detection using PyMuPDF	2024-11-28 22:36:30 +08:00
myhloli	ac88815620	refactor(pdf_check): improve character detection using PyMuPDF ... - Replace pdfminer with PyMuPDF for character detection - Implement new method detect_invalid_chars_by_pymupdf - Update check_invalid_chars in pdf_meta_scan.py to use new method - Add __replace_0xfffd function in pdf_parse_union_core_v2.py to handle special characters - Remove unused imports and update requirements.txt	2024-11-28 22:34:23 +08:00
Xiaomeng Zhao	b4dfa0f92f	Merge pull request #1136 from myhloli/dev ... refactor(ocr): improve text processing and span handling	2024-11-28 19:39:28 +08:00
myhloli	88c0854a65	refactor(ocr): improve text processing and span handling ... - Remove unused language detection code - Simplify text content processing logic - Update span sorting and text extraction in pdf_parse_union_core_v2.py	2024-11-28 19:38:30 +08:00
Xiaomeng Zhao	c295587b9e	Merge pull request #1135 from myhloli/dev ... feat(pdf_parse): filter out skewed text lines	2024-11-28 18:53:06 +08:00
myhloli	37da8c44c4	feat(pdf_parse): filter out skewed text lines ... - Add direction filtering to ignore highly skewed text lines - Improve text extraction accuracy by focusing on non-skewed content	2024-11-28 18:52:18 +08:00
Xiaomeng Zhao	5ecafbfa7d	Merge pull request #1134 from myhloli/dev ... refactor(para): improve language detection and block splitting	2024-11-28 18:07:23 +08:00
myhloli	f674b8d413	refactor(para): improve language detection and block splitting ... - Add language detection for each block of text - Implement language-specific logic for right margin alignment - Introduce logging for debugging purposes	2024-11-28 18:06:17 +08:00
Xiaomeng Zhao	e22fa18b46	Merge pull request #1132 from myhloli/dev ... fix(Hybrid OCR):Enable Hybrid OCR for Empty Spans That Contain a Certain Number of Placeholders but No Actual Text	2024-11-28 15:34:00 +08:00
myhloli	08392d63a0	fix(Hybrid OCR):Enable Hybrid OCR for Empty Spans That Contain a Certain Number of Placeholders but No Actual Text	2024-11-28 15:29:42 +08:00
Xiaomeng Zhao	f09c1cd284	Merge pull request #1130 from myhloli/dev ... fix(lite_model): Adapt iite Mode to the Hybrid OCR Mode in Version 0.10	2024-11-28 15:27:52 +08:00
myhloli	9b4d77dcd4	fix(lite_model): Adapt iite Mode to the Hybrid OCR Mode in Version 0.10	2024-11-28 15:06:54 +08:00
Xiaomeng Zhao	89c7bd0419	Merge pull request #1121 from opendatalab/master ... master -> dev	2024-11-27 18:33:05 +08:00
myhloli	52ef1bc782	Update version.py with new version	2024-11-27 10:31:09 +00:00
Xiaomeng Zhao	8afff9aee8	Merge pull request #1120 from opendatalab/release-0.10.2 ... Release 0.10.2 magic_pdf-0.10.2-released	2024-11-27 18:16:02 +08:00