Commit Graph
6917 Commits
Author SHA1 Message Date
myhloli 31553d29ea ci: add PR gates and separate optional backend tests 2026-10-24 01:16:43 +08:00
Xiaomeng Zhao aa205720b8 Merge pull request #5661 from myhloli/dev
Unify task uploads and results, enhance API and ONNX features
2026-10-23 11:53:54 +08:00
myhloli 28494e633c refactor(api): unify task file fields and generate optional client IDs 2026-10-23 11:53:26 +08:00
myhloli 918454607b fix(router): isolate forwarded proxy headers from workers 2026-10-23 11:33:50 +08:00
myhloli 5ccbc08bdf Merge remote-tracking branch 'origin/dev' into dev 2026-10-23 11:15:31 +08:00
myhloli a6f252779c fix(router): handle cascading task uploads and result leases 2026-10-23 11:15:17 +08:00
Xiaomeng Zhao 06c87ffe9c Merge branch 'opendatalab:dev' into dev 2026-10-23 10:54:09 +08:00
myhloli 412d2c9b86 refactor(gradio): require 6.21 and simplify event chains 2026-10-23 01:11:35 +08:00
myhloli 65ae44f833 test: align DocVortex integration with native-only 0.6 2026-10-22 23:24:00 +08:00
myhloli f013067b96 fix(tasks): harden result leases and SDK request handling 2026-10-22 22:50:48 +08:00
myhloli be6e78a914 chore(deps): require docvortex >=0.6.0,<0.7 2026-10-22 21:52:04 +08:00
myhloli 7f250e2331 feat(api)!: make Tasks the only asynchronous parsing API
Converge the SDK, Gradio, remote CLI, Doclib and Router on Tasks with hash-based uploads, per-file result URLs, task listing and optional model output. Preserve Files and the synchronous file_parse wrapper while removing standalone Uploads/Jobs and local-path HTTP inputs.

Migrate contract, lifecycle, transport and real-document regression coverage; update documentation and HTTP examples. Validation: 1452 passed, 3 conditional skips, plus targeted final PDF checks.

BREAKING CHANGE: clients and workers must use Tasks; standalone Uploads/Jobs endpoints and local-source/inline HTTP options have been removed.
2026-10-22 21:39:42 +08:00
myhloli 92613d738d chore: align Python 3.11-3.14 support and release checks 2026-10-22 21:06:05 +08:00
myhloli 7ce2f07302 fix(cli): preserve full-document output after parsing
Refresh completed non-PDF ranges, keep whole-document content and cursor reads, and require DocVortex 0.5.18 for cross-page table alignment.

Refs opendatalab/MinerU#5654 and opendatalab/MinerU#5652.
2026-10-22 21:04:59 +08:00
myhloli f59acf1ffa feat(api): add recoverable synchronous file parsing service 2026-10-22 18:42:54 +08:00
myhloli 57485941fb feat(api): process JSON tasks and deliver per-file outputs 2026-10-22 18:41:58 +08:00
Xiaomeng Zhao 1681c8e916 fix(llama): serialize Linux Xe-LPG Vulkan submissions 2026-10-22 15:08:19 +08:00
Xiaomeng Zhao 345f630eb1 feat(llama): calibrate integrated GPU routing and serialize legacy Intel Vulkan
Reuse platform ONNX decisions when no discrete GPU is available. Bind and verify the actual Vulkan devices before applying the legacy Intel submission workaround in engine and server paths.

Preserve explicit device settings and offline behavior, and include regression coverage and hardware validation records.
2026-10-22 02:43:32 +08:00
Xiaomeng Zhao 354566d8af feat(onnx): add adaptive calibration and Intel WebGPU architecture policy 2026-10-21 23:17:57 +08:00
Xiaomeng Zhao a2a4960b08 feat(onnx): add Linux WebGPU and compact OCR recognition outputs 2026-10-21 18:56:45 +08:00
myhloli 3702cd7c56 feat(api): support URL inputs in v1 tasks 2026-10-21 15:00:29 +08:00
myhloli ee3ba4dc40 feat(onnx): route UNet through shared device selection 2026-10-20 16:04:11 +08:00
myhloli c666e08dc7 feat(api): unify task uploads and result download links
Accept form manifests and per-file uploads with resumable task state and optional source hashes. Publish result URLs only after final ZIP artifacts are ready, and align Router handling with the same protocol.

Remove synchronous parsing and task result routes, drop fixed local upload limits, and update the HTTP client, documentation, and regression coverage.
2026-10-20 10:48:28 +08:00
Xiaomeng Zhao e0019caba0 Merge pull request #5650 from myhloli/dev
Dev
2026-10-19 16:41:22 +08:00
myhloli 09b37cc7df fix(layout): raise inline formula confidence threshold to 0.45
Keep all other class thresholds unchanged across Torch and ONNX.

Refs #5646
2026-10-19 15:48:41 +08:00
myhloli 48524bfabb feat(onnx): automatically select CPU or DirectML devices
Add auto/cpu/gpu configuration, Windows adapter discovery, isolated rec calibration, atomic preference persistence, and vendor session presets. Route layout, rec, and full MFR batches through DirectML with CPU fallback.

Use mutually exclusive Windows x64 DirectML dependencies and guard ORT/Magika distribution identity. Include calibration fixtures, three-machine results, the 0.8 threshold with ten UHD 770 repeats, and installation documentation.

Validation: 762 MinerU tests and 71 DocVortex tests passed; dependency checks and Windows Python 3.10-3.14 resolution passed. The 19-page OCR result matches the CPU baseline. DocVortex PyPI acceptance remains pending a compatible release; pinned source acceptance passed.
2026-10-19 11:02:29 +08:00
Xiaomeng Zhao 05c64acb6c Merge pull request #5639 from myhloli/dev
Dev
2026-10-13 02:04:18 +08:00
myhloli c3d34f379f test(perf): add CPU and MPS detector benchmark tools
Freeze real detection inputs and replay isolated device, fusion, shape, microbatch, and NNPACK strategies with output parity checks and memory accounting. Add demo PDF Basic/OCR stage timing and keep explicit NNPACK-on controls independent of the production default.

Refs #5632
2026-10-13 01:45:37 +08:00
myhloli d05105e925 perf(runtime): disable NNPACK by default for CPU convolutions
Disable NNPACK during the first lazy Torch initialization while preserving later explicit overrides and GPU/ONNX execution paths. Verify the process setting across worker threads without loading Torch on uninitialized lightweight paths.

Apple M4 CPU FP32 Basic/OCR on demo1 and demo2 retained byte-identical outputs across two order-rotated comparisons. Median parsing time fell from 216.67s to 105.63s, primarily in MFR. Related regression checks: 148 passed; default SDK demo2 replay matched the accepted outputs.

Refs #5632
2026-10-13 01:45:37 +08:00
myhloli 0dbefb400e fix(ocr): skip degenerate text crops before recognition
Filter narrow formula-mask fragments before cropping and keep valid boxes aligned with recognition results. Add coverage for zero-size crops, formula-mask endpoints, and empty recognition batches.
2026-10-13 01:45:09 +08:00
Xiaomeng Zhao 4b47ac619e Merge pull request #5638 from myhloli/dev
Dev
2026-10-12 20:46:46 +08:00
myhloli daf7ee2b21 fix(doclib): handle health-check timeouts on Python 3.10 2026-10-12 20:09:58 +08:00
myhloli 1b2be98968 fix(ocr): skip zero-sized seal crops without misaligning results 2026-10-12 19:37:59 +08:00
myhloli 11ea32032f fix(pdf): restore native drop caps and word boundaries
Recover unclaimed oversized initial letters only at the start of a text paragraph, restore PDFium U+0002 line-end hyphens before text assembly, and consume DocVortex ligature-space geometry without changing OCR routing.

Refs opendatalab/MinerU#5635 and opendatalab/MinerU#5636

Validation: 431 focused regression tests passed. The three-page issue sample passes standard auto/txt and Flash txt with identical Python/Rust ModelJson, MiddleJson and Markdown outputs; source and before/after line boxes were reviewed.
2026-10-12 16:33:48 +08:00
myhloli 77698670a5 feat: update issue template and sync published release version 2026-10-11 18:40:17 +08:00
myhloli a87030c1f3 chore(deps): bump mineru-llama-cpp and mineru-vl-utils minimum versions 2026-10-11 18:23:52 +08:00
myhloli d30ed2d230 fix(doclib): invalidate managed tier health during switches
Invalidate stale local capabilities before config changes return, wake the health worker, and reject superseded config/probe results until the target tier is ready. Preserve the actual child startup state for graceful shutdown.

Refs #5272
2026-10-11 11:37:10 +08:00
myhloli aca9b06da3 feat(doclib): add paginated filename search and unified page selectors
Apply filename prefix matching to the final query token and paginate the
complete filtered candidate set with stable ordering and accurate totals.
Reuse input page-range syntax in read locators, including rN and all, and
preserve selected-page boundaries during block and character continuation.
Update CLI/SDK/HTTP contracts, bilingual help, documentation, and regressions.

Validation: 667 Doclib/CLI tests passed; Ruff and formatting checks passed.
Refs #5366, #5337, #5365.
2026-10-11 11:22:21 +08:00
myhloli dc60e7c672 chore(release): prepare MinerU 4.0.12 2026-10-11 10:15:30 +08:00
myhloli 3d0326e299 fix(router): preserve retention and usage and validate output requests 2026-10-11 10:15:30 +08:00
Xiaomeng Zhao d90e43a8f6 Merge pull request #5630 from opendatalab/dev
Dev
2026-10-11 10:11:23 +08:00
myhloli 75c181a387 fix(ci): translate retention help and isolate optional Torch tests 2026-10-11 04:27:47 +08:00
myhloli 193d193aa8 fix(ci): require DocVortex 0.5.16 and repair docs and coverage checks 2026-10-11 04:19:10 +08:00
myhloli 78379ff635 fix(pdf): reuse shared CJK space handling for native text
Use DocVortex generated-space evidence in Python span filling and rotated-line paths, while keeping the Rust fast path aligned. Document list-tail continuation semantics and add spacing, code protection, rotation, and Python/Rust parity regressions.
2026-10-11 00:26:04 +08:00
myhloli 13bc9cf33e test(pdf): preserve rejected native table inputs for model fallback 2026-10-10 22:36:35 +08:00
myhloli b479e42d90 fix(layout): restore PP-DocLayoutV2 model class thresholds
Align output filtering with reading-order candidate validity in Torch and ONNX. Remove the uniform conf override and cover classification threshold boundaries and multi-label candidates.
2026-10-10 12:07:39 +08:00
myhloli 197795402f Record verified closure of six repaired pull requests 2026-10-10 01:30:07 +08:00
myhloli 27ecc16f0f Record authorized fix pushes and original PR closure scope 2026-10-10 01:27:27 +08:00
myhloli 5b477601a0 Document PR triage outcomes and service retention and recovery settings 2026-10-10 01:17:33 +08:00
myhloli 7b19a73a20 Record loaded source paths in CPU acceptance measurements; Refs #5477 2026-10-10 01:14:30 +08:00