feat: add model source configuration and auto-detection logic

docs/en/usage/model_source.md

@@ -2,13 +2,14 @@
 
 MinerU uses `HuggingFace` and `ModelScope` as model repositories. Users can switch model sources or use local models as needed.
 
-- `HuggingFace` is the default model source, providing excellent loading speed and high stability globally.
+- `auto` is the default model source policy. It first requests `https://huggingface.co/models` to check whether Hugging Face is reachable. If reachable, MinerU uses `HuggingFace`; otherwise, it falls back to `ModelScope`.
+- `HuggingFace` provides excellent loading speed and high stability globally.
 - `ModelScope` is the best choice for users in mainland China, providing seamlessly compatible `hf` SDK modules, suitable for users who cannot access HuggingFace.
 
 ## Methods to Switch Model Sources
 
 ### Configure via Environment Variables
-MinerU configures model sources through the `MINERU_MODEL_SOURCE` environment variable. This applies to all command line tools and API calls.
+MinerU configures model sources through the `MINERU_MODEL_SOURCE` environment variable. This applies to all command line tools and API calls. Supported values are `huggingface`, `modelscope`, and `local`. The environment variable has higher priority than `model-source` in `mineru.json`. Do not set this environment variable to `auto`; unset it if you want MinerU to choose a source automatically.
 ```bash
 export MINERU_MODEL_SOURCE=modelscope
 mineru -p <input_path> -o <output_path>
@@ -21,6 +22,14 @@ os.environ["MINERU_MODEL_SOURCE"] = "modelscope"
 >[!TIP]
 > MinerU no longer provides a CLI flag for model source selection. Model sources set through environment variables take effect in the current terminal session until the terminal is closed or the environment variable is modified.
 
+### Configure via Configuration File
+If `MINERU_MODEL_SOURCE` is not set, MinerU reads the `model-source` field from `mineru.json` in the user directory. `model-source` supports fixed values `huggingface` and `modelscope`, and also supports the template's first-run placeholder value `auto`. When the value is `auto` or the field is missing, MinerU probes the actual source first. After the first auto probe resolves an actual source, MinerU writes `model-source` back as `huggingface` or `modelscope` to avoid switching sources on later startups due to network fluctuations.
+```json
+{
+    "model-source": "auto"
+}
+```
+
 ## Using Local Models
 
 ### 1. Download Models to Local Storage
@@ -32,12 +41,12 @@ or use the interactive command line tool to select model downloads:
 mineru-models-download
 ```
 > [!NOTE]
->- After download completion, the model path will be output in the current terminal window and automatically written to `mineru.json` in the user directory.
->- You can also create it by copying the [configuration template file](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json) to your user directory and renaming it to `mineru.json`.
+>- After download completion, the model path will be output in the current terminal window and automatically written to `mineru.json` in the user directory. The `model-source` field records the actual remote source used for this download, either `huggingface` or `modelscope`.
+>- You can also create it by copying the [configuration template file](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json) to your user directory and renaming it to `mineru.json`. The template sets `model-source` to `auto`, so MinerU auto-detects once and writes back the resolved source on first use.
 >- After downloading models locally, you can freely move the model folder to other locations while updating the model path in `mineru.json`.
 >- If you deploy the model folder to another server, please ensure you move the `mineru.json` file to the user directory of the new device and configure the model path correctly.
 >- If you need to update model files, you can run the `mineru-models-download` command again. Model updates do not support custom paths currently - if you haven't moved the local model folder, model files will be incrementally updated; if you have moved the model folder, model files will be re-downloaded to the default location and `mineru.json` will be updated.
->- `mineru-models-download` must use a remote model source to perform a real download. If your current shell already sets `MINERU_MODEL_SOURCE=local`, this command will temporarily ignore that value for this invocation and use your selected `huggingface` or `modelscope` source instead.
+>- `mineru-models-download` must use a remote model source to perform a real download. If your current shell already sets `MINERU_MODEL_SOURCE=local`, this command will temporarily ignore that value for this invocation and use your selected `auto`, `huggingface`, or `modelscope` source instead.
 
 ### 2. Use Local Models for Parsing
 

docs/zh/usage/model_source.md

@@ -2,13 +2,14 @@
 
 MinerU使用 `HuggingFace` 和 `ModelScope` 作为模型仓库，用户可以根据需要切换模型源或使用本地模型。
 
-- `HuggingFace` 是默认的模型源，在全球范围内提供了优异的加载速度和极高稳定性。
+- `auto` 是默认的模型源策略，会先请求 `https://huggingface.co/models` 探测 HuggingFace 是否可访问；可访问时使用 `HuggingFace`，不可访问时自动回退到 `ModelScope`。
+- `HuggingFace` 在全球范围内提供了优异的加载速度和极高稳定性。
 - `ModelScope` 是中国大陆地区用户的最佳选择，提供了无缝兼容的SDK模块，适用于无法访问`HuggingFace`的用户。
 
 ## 模型源的切换方法
 
 ### 通过环境变量切换
-MinerU 通过 `MINERU_MODEL_SOURCE` 环境变量配置模型源，这适用于所有命令行工具和 API 调用。
+MinerU 通过 `MINERU_MODEL_SOURCE` 环境变量配置模型源，这适用于所有命令行工具和 API 调用。支持的取值为 `huggingface`、`modelscope` 和 `local`，环境变量优先级高于 `mineru.json` 中的 `model-source`。请不要将环境变量设置为 `auto`；如需自动选择来源，请删除该环境变量。
 ```bash
 export MINERU_MODEL_SOURCE=modelscope
 mineru -p <input_path> -o <output_path>
@@ -21,6 +22,14 @@ os.environ["MINERU_MODEL_SOURCE"] = "modelscope"
 >[!TIP]
 > MinerU 已不再提供用于切换模型源的命令行参数。通过环境变量设置的模型源会在当前终端会话中生效，直到终端关闭或环境变量被修改。
 
+### 通过配置文件切换
+如果未设置 `MINERU_MODEL_SOURCE`，MinerU 会读取用户目录下 `mineru.json` 中的 `model-source` 字段。`model-source` 支持固定值 `huggingface`、`modelscope`，也支持模板中的首次解析占位值 `auto`。当值为 `auto` 或字段缺失时，会先自动探测实际来源；首次自动探测完成后，会将 `model-source` 写回为 `huggingface` 或 `modelscope`，避免后续启动时因网络波动反复切换来源。
+```json
+{
+    "model-source": "auto"
+}
+```
+
 
 ## 使用本地模型
 
@@ -33,12 +42,12 @@ mineru-models-download --help
 mineru-models-download
 ```
 > [!NOTE]
->- 下载完成后，模型路径会在当前终端窗口输出，并自动写入用户目录下的 `mineru.json`。
->- 您也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建配置文件。
+>- 下载完成后，模型路径会在当前终端窗口输出，并自动写入用户目录下的 `mineru.json`。配置文件中的 `model-source` 会记录本次实际使用的远端来源，即 `huggingface` 或 `modelscope`。
+>- 您也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建配置文件；模板中的 `model-source` 默认为 `auto`，首次使用时会自动探测并写回实际来源。
 >- 模型下载到本地后，您可以自由移动模型文件夹到其他位置，同时需要在 `mineru.json` 中更新模型路径。
 >- 如您将模型文件夹部署到其他服务器上，请确保将 `mineru.json`文件一同移动到新设备的用户目录中并正确配置模型路径。
 >- 如您需要更新模型文件，可以再次运行 `mineru-models-download` 命令，模型更新暂不支持自定义路径，如您没有移动本地模型文件夹，模型文件会增量更新；如您移动了模型文件夹，模型文件会重新下载到默认位置并更新`mineru.json`。
->- `mineru-models-download` 必须使用远端模型源执行真实下载；如果当前终端已设置 `MINERU_MODEL_SOURCE=local`，该命令会仅在本次执行中临时忽略该值，并改用您选择的 `huggingface` 或 `modelscope` 下载模型。
+>- `mineru-models-download` 必须使用远端模型源执行真实下载；如果当前终端已设置 `MINERU_MODEL_SOURCE=local`，该命令会仅在本次执行中临时忽略该值，并改用您选择的 `auto`、`huggingface` 或 `modelscope` 下载模型。
 
 ### 2. 使用本地模型进行解析
 

mineru.template.json

@@ -26,5 +26,6 @@
         "pipeline": "",
         "vlm": ""
     },
-    "config_version": "1.3.1"
-}
+    "model-source": "auto",
+    "config_version": "1.3.2"
+}

mineru/cli/models_download.py

@@ -1,69 +1,39 @@
 # Copyright (c) Opendatalab. All rights reserved.
 from contextlib import contextmanager
-import json
 import os
 import sys
 import click
-import requests
 from loguru import logger
 
 from mineru.utils.enum_class import ModelPath
-from mineru.utils.models_download_utils import auto_download_and_get_model_root_path
+from mineru.utils.models_download_utils import (
+    CONFIG_TEMPLATE_URL,
+    auto_download_and_get_model_root_path,
+    download_and_modify_json,
+    get_tools_config_file_path,
+    resolve_model_source,
+)
 
 MODEL_SOURCE_ENV_VAR = 'MINERU_MODEL_SOURCE'
-REMOTE_MODEL_SOURCES = ('huggingface', 'modelscope')
+REMOTE_MODEL_SOURCES = ('auto', 'huggingface', 'modelscope')
 
 
-def download_json(url):
-    """下载JSON文件"""
-    response = requests.get(url)
-    response.raise_for_status()
-    return response.json()
-
-
-def download_and_modify_json(url, local_filename, modifications):
-    """下载JSON并修改内容"""
-    if os.path.exists(local_filename):
-        data = json.load(open(local_filename))
-        config_version = data.get('config_version', '0.0.0')
-        if config_version < '1.3.1':
-            data = download_json(url)
-    else:
-        data = download_json(url)
-
-    # 修改内容
-    for key, value in modifications.items():
-        if key in data:
-            if isinstance(data[key], dict):
-                # 如果是字典，合并新值
-                data[key].update(value)
-            else:
-                # 否则直接替换
-                data[key] = value
-
-    # 保存修改后的内容
-    with open(local_filename, 'w', encoding='utf-8') as f:
-        json.dump(data, f, ensure_ascii=False, indent=4)
-
-
-def configure_model(model_dir, model_type):
+def configure_model(model_dir, model_type, model_source):
     """配置模型"""
-    json_url = 'https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/mineru.template.json'
-    config_file_name = os.getenv('MINERU_TOOLS_CONFIG_JSON', 'mineru.json')
-    home_dir = os.path.expanduser('~')
-    config_file = os.path.join(home_dir, config_file_name)
+    config_file = get_tools_config_file_path()
 
     json_mods = {
         'models-dir': {
             f'{model_type}': model_dir
-        }
+        },
+        'model-source': model_source,
     }
 
-    download_and_modify_json(json_url, config_file, json_mods)
+    download_and_modify_json(CONFIG_TEMPLATE_URL, config_file, json_mods)
     logger.info(f'The configuration file has been successfully configured, the path is: {config_file}')
 
 
-def download_pipeline_models():
+def download_pipeline_models(model_source):
     """下载Pipeline模型"""
     model_paths = [
         ModelPath.pp_doclayout_v2,
@@ -79,14 +49,14 @@ def download_pipeline_models():
         logger.info(f"Downloading model: {model_path}")
         download_finish_path = auto_download_and_get_model_root_path(model_path, repo_mode='pipeline')
     logger.info(f"Pipeline models downloaded successfully to: {download_finish_path}")
-    configure_model(download_finish_path, "pipeline")
+    configure_model(download_finish_path, "pipeline", model_source)
 
 
-def download_vlm_models():
+def download_vlm_models(model_source):
     """下载VLM模型"""
     download_finish_path = auto_download_and_get_model_root_path("/", repo_mode='vlm')
     logger.info(f"VLM models downloaded successfully to: {download_finish_path}")
-    configure_model(download_finish_path, "vlm")
+    configure_model(download_finish_path, "vlm", model_source)
 
 
 def get_effective_download_model_source(requested_model_source):
@@ -98,12 +68,12 @@ def get_effective_download_model_source(requested_model_source):
             f"`mineru-models-download` will temporarily use '{requested_model_source}' "
             f"to perform a real download."
         )
-        return requested_model_source
+        return resolve_model_source(requested_model_source, allow_auto=True)
 
     if current_model_source is None:
-        return requested_model_source
+        return resolve_model_source(requested_model_source, allow_auto=True)
 
-    return current_model_source
+    return resolve_model_source(current_model_source)
 
 
 @contextmanager
@@ -151,7 +121,7 @@ def download_models(model_source, model_type):
         model_source = click.prompt(
             "Please select the model download source: ",
             type=click.Choice(REMOTE_MODEL_SOURCES),
-            default='huggingface'
+            default='auto'
         )
 
     effective_model_source = get_effective_download_model_source(model_source)
@@ -169,12 +139,12 @@ def download_models(model_source, model_type):
     try:
         with temporary_model_source(effective_model_source):
             if model_type == 'pipeline':
-                download_pipeline_models()
+                download_pipeline_models(effective_model_source)
             elif model_type == 'vlm':
-                download_vlm_models()
+                download_vlm_models(effective_model_source)
             elif model_type == 'all':
-                download_pipeline_models()
-                download_vlm_models()
+                download_pipeline_models(effective_model_source)
+                download_vlm_models(effective_model_source)
             else:
                 click.echo(f"Unsupported model type: {model_type}", err=True)
                 sys.exit(1)

mineru/utils/config_reader.py

@@ -30,6 +30,36 @@ def read_config():
         return config
 
 
+def get_configured_model_source(default: str | None = None) -> str | None:
+    """读取配置文件中的固定模型来源配置，auto 或缺失时返回默认值。"""
+    supported_sources = {'huggingface', 'modelscope'}
+    config = read_config()
+    if config is None:
+        return default
+
+    model_source = config.get('model-source')
+    if model_source is None:
+        return default
+    if not isinstance(model_source, str):
+        logger.warning(
+            f"'model-source' in {CONFIG_FILE_NAME} must be a string, use {default} as default"
+        )
+        return default
+
+    normalized_model_source = model_source.strip().lower()
+    if not normalized_model_source:
+        return default
+    if normalized_model_source == "auto":
+        return default
+    if normalized_model_source in supported_sources:
+        return normalized_model_source
+
+    logger.warning(
+        f"Unsupported 'model-source' in {CONFIG_FILE_NAME}: {model_source}, use {default} as default"
+    )
+    return default
+
+
 def get_s3_config(bucket_name: str):
     """~/magic-pdf.json 读出来."""
     config = read_config()

mineru/utils/models_download_utils.py

@@ -1,13 +1,167 @@
 # Copyright (c) Opendatalab. All rights reserved.
+import json
 import os
 from functools import lru_cache
 
 from huggingface_hub import snapshot_download as hf_snapshot_download
+from loguru import logger
 from modelscope import snapshot_download as ms_snapshot_download
+import requests
 
-from mineru.utils.config_reader import get_local_models_dir
+from mineru.utils.config_reader import get_configured_model_source, get_local_models_dir
 from mineru.utils.enum_class import ModelPath
 
+MODEL_SOURCE_ENV_VAR = 'MINERU_MODEL_SOURCE'
+CONFIG_TEMPLATE_URL = 'https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/mineru.template.json'
+MINERU_CONFIG_VERSION = '1.3.2'
+HUGGINGFACE_MODELS_PAGE_URL = "https://huggingface.co/models"
+HUGGINGFACE_MODELS_PAGE_TIMEOUT = 3
+HUGGINGFACE_MODELS_PAGE_MAX_ATTEMPTS = 2
+REMOTE_MODEL_SOURCES = ("huggingface", "modelscope")
+
+
+def get_tools_config_file_path() -> str:
+    """获取 MinerU 工具配置文件路径，支持环境变量指定绝对或相对路径。"""
+    config_file_name = os.getenv('MINERU_TOOLS_CONFIG_JSON', 'mineru.json')
+    if os.path.isabs(config_file_name):
+        return config_file_name
+    return os.path.join(os.path.expanduser('~'), config_file_name)
+
+
+def download_json(url):
+    """下载 JSON 文件并返回解析后的内容。"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.json()
+
+
+def is_config_version_outdated(config_version):
+    """判断本地配置版本是否低于当前模板版本。"""
+    def version_tuple(version):
+        """将版本号字符串转换为可比较的整数元组。"""
+        parts = []
+        for part in str(version).split('.'):
+            parts.append(int(part) if part.isdigit() else 0)
+        return tuple(parts)
+
+    current_version = version_tuple(config_version)
+    target_version = version_tuple(MINERU_CONFIG_VERSION)
+    max_len = max(len(current_version), len(target_version))
+    current_version += (0,) * (max_len - len(current_version))
+    target_version += (0,) * (max_len - len(target_version))
+    return current_version < target_version
+
+
+def merge_config_dict(base_config: dict, override_config: dict, skip_keys: set[str] | None = None) -> dict:
+    """递归合并配置字典，用 override_config 覆盖 base_config 并保留新模板字段。"""
+    skip_keys = skip_keys or set()
+    merged_config = dict(base_config)
+    for key, value in override_config.items():
+        if key in skip_keys:
+            continue
+        base_value = merged_config.get(key)
+        if isinstance(base_value, dict) and isinstance(value, dict):
+            merged_config[key] = merge_config_dict(base_value, value, skip_keys=skip_keys)
+        else:
+            merged_config[key] = value
+    return merged_config
+
+
+def download_and_modify_json(url, local_filename, modifications):
+    """下载或读取 JSON 配置，并按 modifications 合并更新后写回。"""
+    if os.path.exists(local_filename):
+        with open(local_filename, encoding='utf-8') as f:
+            data = json.load(f)
+        config_version = data.get('config_version', '0.0.0')
+        if is_config_version_outdated(config_version):
+            template_data = download_json(url)
+            data = merge_config_dict(template_data, data, skip_keys={'config_version'})
+    else:
+        data = download_json(url)
+
+    data = merge_config_dict(data, modifications)
+
+    with open(local_filename, 'w', encoding='utf-8') as f:
+        json.dump(data, f, ensure_ascii=False, indent=4)
+
+
+def persist_resolved_model_source(model_source: str) -> None:
+    """将 auto 解析出的实际模型来源写入配置文件，避免下次启动时再次受网络波动影响。"""
+    if model_source not in REMOTE_MODEL_SOURCES:
+        return
+    try:
+        download_and_modify_json(
+            CONFIG_TEMPLATE_URL,
+            get_tools_config_file_path(),
+            {'model-source': model_source},
+        )
+    except Exception as exc:
+        logger.warning(f"Failed to persist resolved model source '{model_source}': {exc}")
+
+
+@lru_cache(maxsize=1)
+def resolve_auto_model_source() -> str:
+    """通过 Hugging Face 模型列表页探测 auto 应该使用的实际模型来源。"""
+    last_error = None
+    for _ in range(HUGGINGFACE_MODELS_PAGE_MAX_ATTEMPTS):
+        try:
+            response = requests.get(
+                HUGGINGFACE_MODELS_PAGE_URL,
+                timeout=HUGGINGFACE_MODELS_PAGE_TIMEOUT,
+            )
+            if 200 <= response.status_code < 400:
+                return "huggingface"
+            last_error = f"status_code={response.status_code}"
+        except Exception as exc:
+            last_error = str(exc)
+
+    logger.warning(
+        f"Failed to access {HUGGINGFACE_MODELS_PAGE_URL}: {last_error}, fallback to modelscope."
+    )
+    return "modelscope"
+
+
+def resolve_model_source(model_source: str | None = None, allow_auto: bool = False) -> str:
+    """将环境变量或配置文件中的模型来源解析为实际可下载的来源。"""
+    if model_source is None:
+        model_source = os.getenv(MODEL_SOURCE_ENV_VAR)
+        if isinstance(model_source, str) and model_source.strip().lower() == "auto":
+            raise ValueError(
+                f"{MODEL_SOURCE_ENV_VAR}=auto is not supported. "
+                f"Unset {MODEL_SOURCE_ENV_VAR} to use auto detection once, "
+                "or set it to huggingface/modelscope/local."
+            )
+    if model_source is None:
+        model_source = get_configured_model_source()
+    if model_source is None:
+        model_source = "auto"
+        allow_auto = True
+
+    if not isinstance(model_source, str):
+        logger.warning(f"Unsupported model source type: {type(model_source)}, fallback to auto.")
+        model_source = "auto"
+        allow_auto = True
+
+    normalized_model_source = model_source.strip().lower()
+    if normalized_model_source == "local":
+        return "local"
+    if normalized_model_source == "auto":
+        if not allow_auto:
+            raise ValueError(
+                "model source auto is only supported for internal default detection "
+                "or explicit download command selection."
+            )
+        resolved_model_source = resolve_auto_model_source()
+        persist_resolved_model_source(resolved_model_source)
+        return resolved_model_source
+    if normalized_model_source in REMOTE_MODEL_SOURCES:
+        return normalized_model_source
+
+    logger.warning(f"Unsupported model source: {model_source}, fallback to auto.")
+    resolved_model_source = resolve_auto_model_source()
+    persist_resolved_model_source(resolved_model_source)
+    return resolved_model_source
+
 
 @lru_cache(maxsize=None)
 def _snapshot_download_cached(model_source: str, repo_mode: str, repo: str, relative_path: str) -> str:
@@ -40,7 +194,7 @@ def auto_download_and_get_model_root_path(relative_path: str, repo_mode='pipelin
     :param relative_path: 文件或目录相对路径
     :return: 本地文件绝对路径或相对路径
     """
-    model_source = os.getenv('MINERU_MODEL_SOURCE', "huggingface")
+    model_source = resolve_model_source()
 
     if model_source == 'local':
         local_models_config = get_local_models_dir()
@@ -53,22 +207,19 @@ def auto_download_and_get_model_root_path(relative_path: str, repo_mode='pipelin
     repo_mapping = {
         'pipeline': {
             'huggingface': ModelPath.pipeline_root_hf,
-            'modelscope': ModelPath.pipeline_root_modelscope,
-            'default': ModelPath.pipeline_root_hf
+            'modelscope': ModelPath.pipeline_root_modelscope
         },
         'vlm': {
             'huggingface': ModelPath.vlm_root_hf,
-            'modelscope': ModelPath.vlm_root_modelscope,
-            'default': ModelPath.vlm_root_hf
+            'modelscope': ModelPath.vlm_root_modelscope
         }
     }
 
     if repo_mode not in repo_mapping:
         raise ValueError(f"Unsupported repo_mode: {repo_mode}, must be 'pipeline' or 'vlm'")
 
-    # 如果没有指定model_source或值不是'modelscope'，则使用默认值
-    repo = repo_mapping[repo_mode].get(model_source, repo_mapping[repo_mode]['default'])
-
+    # model_source 已解析为实际远端来源后，再选择对应仓库。
+    repo = repo_mapping[repo_mode][model_source]
 
     if repo_mode == 'pipeline':
         relative_path = relative_path.strip('/')