Commit Graph
10 Commits
Author SHA1 Message Date
myhloli e1be7da644 refactor(magic_pdf): switch to pdfminer for invalid character detection
- Replace MuPDF with pdfminer for detecting invalid characters in PDFs
- Uncomment and update the detect_invalid_chars function to use pdfminer
- Update the check_invalid_chars function in pdf_meta_scan.py to use the new implementation
2024-12-11 01:41:35 +08:00
icecraft a3a720ea87 refactor: isolate inference and pipeline 2024-12-03 14:06:58 +08:00
myhloli ac88815620 refactor(pdf_check): improve character detection using PyMuPDF
- Replace pdfminer with PyMuPDF for character detection
- Implement new method detect_invalid_chars_by_pymupdf
- Update check_invalid_chars in pdf_meta_scan.py to use new method
- Add __replace_0xfffd function in pdf_parse_union_core_v2.py to handle special characters
- Remove unused imports and update requirements.txt
2024-11-28 22:34:23 +08:00
myhloli 2db3c26374 refactor(libs): remove unused imports and functions
- Remove unused imports from commons.py
- Delete unused functions related to AWS and S3 operations
- Update import statements in other modules to reflect changes in commons.py
- Remove redundant code and improve code readability
2024-11-27 14:51:30 +08:00
icecraft b492c19c4c refactor: move some constants or enums defs to config folder 2024-11-19 16:56:53 +08:00
赵小蒙 df14c61f6f update: Enhance the capability to detect garbled document issues 2024-06-19 16:03:21 +08:00
赵小蒙 084dc22ab1 update AVG_TEXT_LEN_THRESHOLD 200->100 2024-06-18 10:46:54 +08:00
赵小蒙 a5e22396c1 delete svgs check 2024-04-26 17:41:33 +08:00
赵小蒙 1b9d65b3d3 1、Trace类的key增加前置下划线
2、实现UNIPipe
2024-04-11 17:43:00 +08:00
赵小蒙 d5dbed7325 目录重构 2024-03-01 16:07:51 +08:00