diff --git a/docs/en/usage/model_source.md b/docs/en/usage/model_source.md index 9fb86ada..9d2d61f3 100644 --- a/docs/en/usage/model_source.md +++ b/docs/en/usage/model_source.md @@ -2,13 +2,14 @@ MinerU uses `HuggingFace` and `ModelScope` as model repositories. Users can switch model sources or use local models as needed. -- `HuggingFace` is the default model source, providing excellent loading speed and high stability globally. +- `auto` is the default model source policy. It first requests `https://huggingface.co/models` to check whether Hugging Face is reachable. If reachable, MinerU uses `HuggingFace`; otherwise, it falls back to `ModelScope`. +- `HuggingFace` provides excellent loading speed and high stability globally. - `ModelScope` is the best choice for users in mainland China, providing seamlessly compatible `hf` SDK modules, suitable for users who cannot access HuggingFace. ## Methods to Switch Model Sources ### Configure via Environment Variables -MinerU configures model sources through the `MINERU_MODEL_SOURCE` environment variable. This applies to all command line tools and API calls. +MinerU configures model sources through the `MINERU_MODEL_SOURCE` environment variable. This applies to all command line tools and API calls. Supported values are `huggingface`, `modelscope`, and `local`. The environment variable has higher priority than `model-source` in `mineru.json`. Do not set this environment variable to `auto`; unset it if you want MinerU to choose a source automatically. ```bash export MINERU_MODEL_SOURCE=modelscope mineru -p -o @@ -21,6 +22,14 @@ os.environ["MINERU_MODEL_SOURCE"] = "modelscope" >[!TIP] > MinerU no longer provides a CLI flag for model source selection. Model sources set through environment variables take effect in the current terminal session until the terminal is closed or the environment variable is modified. +### Configure via Configuration File +If `MINERU_MODEL_SOURCE` is not set, MinerU reads the `model-source` field from `mineru.json` in the user directory. `model-source` supports fixed values `huggingface` and `modelscope`, and also supports the template's first-run placeholder value `auto`. When the value is `auto` or the field is missing, MinerU probes the actual source first. After the first auto probe resolves an actual source, MinerU writes `model-source` back as `huggingface` or `modelscope` to avoid switching sources on later startups due to network fluctuations. +```json +{ + "model-source": "auto" +} +``` + ## Using Local Models ### 1. Download Models to Local Storage @@ -32,12 +41,12 @@ or use the interactive command line tool to select model downloads: mineru-models-download ``` > [!NOTE] ->- After download completion, the model path will be output in the current terminal window and automatically written to `mineru.json` in the user directory. ->- You can also create it by copying the [configuration template file](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json) to your user directory and renaming it to `mineru.json`. +>- After download completion, the model path will be output in the current terminal window and automatically written to `mineru.json` in the user directory. The `model-source` field records the actual remote source used for this download, either `huggingface` or `modelscope`. +>- You can also create it by copying the [configuration template file](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json) to your user directory and renaming it to `mineru.json`. The template sets `model-source` to `auto`, so MinerU auto-detects once and writes back the resolved source on first use. >- After downloading models locally, you can freely move the model folder to other locations while updating the model path in `mineru.json`. >- If you deploy the model folder to another server, please ensure you move the `mineru.json` file to the user directory of the new device and configure the model path correctly. >- If you need to update model files, you can run the `mineru-models-download` command again. Model updates do not support custom paths currently - if you haven't moved the local model folder, model files will be incrementally updated; if you have moved the model folder, model files will be re-downloaded to the default location and `mineru.json` will be updated. ->- `mineru-models-download` must use a remote model source to perform a real download. If your current shell already sets `MINERU_MODEL_SOURCE=local`, this command will temporarily ignore that value for this invocation and use your selected `huggingface` or `modelscope` source instead. +>- `mineru-models-download` must use a remote model source to perform a real download. If your current shell already sets `MINERU_MODEL_SOURCE=local`, this command will temporarily ignore that value for this invocation and use your selected `auto`, `huggingface`, or `modelscope` source instead. ### 2. Use Local Models for Parsing diff --git a/docs/zh/usage/model_source.md b/docs/zh/usage/model_source.md index ff73cd87..53678679 100644 --- a/docs/zh/usage/model_source.md +++ b/docs/zh/usage/model_source.md @@ -2,13 +2,14 @@ MinerU使用 `HuggingFace` 和 `ModelScope` 作为模型仓库,用户可以根据需要切换模型源或使用本地模型。 -- `HuggingFace` 是默认的模型源,在全球范围内提供了优异的加载速度和极高稳定性。 +- `auto` 是默认的模型源策略,会先请求 `https://huggingface.co/models` 探测 HuggingFace 是否可访问;可访问时使用 `HuggingFace`,不可访问时自动回退到 `ModelScope`。 +- `HuggingFace` 在全球范围内提供了优异的加载速度和极高稳定性。 - `ModelScope` 是中国大陆地区用户的最佳选择,提供了无缝兼容的SDK模块,适用于无法访问`HuggingFace`的用户。 ## 模型源的切换方法 ### 通过环境变量切换 -MinerU 通过 `MINERU_MODEL_SOURCE` 环境变量配置模型源,这适用于所有命令行工具和 API 调用。 +MinerU 通过 `MINERU_MODEL_SOURCE` 环境变量配置模型源,这适用于所有命令行工具和 API 调用。支持的取值为 `huggingface`、`modelscope` 和 `local`,环境变量优先级高于 `mineru.json` 中的 `model-source`。请不要将环境变量设置为 `auto`;如需自动选择来源,请删除该环境变量。 ```bash export MINERU_MODEL_SOURCE=modelscope mineru -p -o @@ -21,6 +22,14 @@ os.environ["MINERU_MODEL_SOURCE"] = "modelscope" >[!TIP] > MinerU 已不再提供用于切换模型源的命令行参数。通过环境变量设置的模型源会在当前终端会话中生效,直到终端关闭或环境变量被修改。 +### 通过配置文件切换 +如果未设置 `MINERU_MODEL_SOURCE`,MinerU 会读取用户目录下 `mineru.json` 中的 `model-source` 字段。`model-source` 支持固定值 `huggingface`、`modelscope`,也支持模板中的首次解析占位值 `auto`。当值为 `auto` 或字段缺失时,会先自动探测实际来源;首次自动探测完成后,会将 `model-source` 写回为 `huggingface` 或 `modelscope`,避免后续启动时因网络波动反复切换来源。 +```json +{ + "model-source": "auto" +} +``` + ## 使用本地模型 @@ -33,12 +42,12 @@ mineru-models-download --help mineru-models-download ``` > [!NOTE] ->- 下载完成后,模型路径会在当前终端窗口输出,并自动写入用户目录下的 `mineru.json`。 ->- 您也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建配置文件。 +>- 下载完成后,模型路径会在当前终端窗口输出,并自动写入用户目录下的 `mineru.json`。配置文件中的 `model-source` 会记录本次实际使用的远端来源,即 `huggingface` 或 `modelscope`。 +>- 您也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建配置文件;模板中的 `model-source` 默认为 `auto`,首次使用时会自动探测并写回实际来源。 >- 模型下载到本地后,您可以自由移动模型文件夹到其他位置,同时需要在 `mineru.json` 中更新模型路径。 >- 如您将模型文件夹部署到其他服务器上,请确保将 `mineru.json`文件一同移动到新设备的用户目录中并正确配置模型路径。 >- 如您需要更新模型文件,可以再次运行 `mineru-models-download` 命令,模型更新暂不支持自定义路径,如您没有移动本地模型文件夹,模型文件会增量更新;如您移动了模型文件夹,模型文件会重新下载到默认位置并更新`mineru.json`。 ->- `mineru-models-download` 必须使用远端模型源执行真实下载;如果当前终端已设置 `MINERU_MODEL_SOURCE=local`,该命令会仅在本次执行中临时忽略该值,并改用您选择的 `huggingface` 或 `modelscope` 下载模型。 +>- `mineru-models-download` 必须使用远端模型源执行真实下载;如果当前终端已设置 `MINERU_MODEL_SOURCE=local`,该命令会仅在本次执行中临时忽略该值,并改用您选择的 `auto`、`huggingface` 或 `modelscope` 下载模型。 ### 2. 使用本地模型进行解析 diff --git a/mineru.template.json b/mineru.template.json index 7d979ec8..a933e799 100644 --- a/mineru.template.json +++ b/mineru.template.json @@ -26,5 +26,6 @@ "pipeline": "", "vlm": "" }, - "config_version": "1.3.1" -} \ No newline at end of file + "model-source": "auto", + "config_version": "1.3.2" +} diff --git a/mineru/cli/models_download.py b/mineru/cli/models_download.py index cc1ab48a..da7bd468 100644 --- a/mineru/cli/models_download.py +++ b/mineru/cli/models_download.py @@ -1,69 +1,39 @@ # Copyright (c) Opendatalab. All rights reserved. from contextlib import contextmanager -import json import os import sys import click -import requests from loguru import logger from mineru.utils.enum_class import ModelPath -from mineru.utils.models_download_utils import auto_download_and_get_model_root_path +from mineru.utils.models_download_utils import ( + CONFIG_TEMPLATE_URL, + auto_download_and_get_model_root_path, + download_and_modify_json, + get_tools_config_file_path, + resolve_model_source, +) MODEL_SOURCE_ENV_VAR = 'MINERU_MODEL_SOURCE' -REMOTE_MODEL_SOURCES = ('huggingface', 'modelscope') +REMOTE_MODEL_SOURCES = ('auto', 'huggingface', 'modelscope') -def download_json(url): - """下载JSON文件""" - response = requests.get(url) - response.raise_for_status() - return response.json() - - -def download_and_modify_json(url, local_filename, modifications): - """下载JSON并修改内容""" - if os.path.exists(local_filename): - data = json.load(open(local_filename)) - config_version = data.get('config_version', '0.0.0') - if config_version < '1.3.1': - data = download_json(url) - else: - data = download_json(url) - - # 修改内容 - for key, value in modifications.items(): - if key in data: - if isinstance(data[key], dict): - # 如果是字典,合并新值 - data[key].update(value) - else: - # 否则直接替换 - data[key] = value - - # 保存修改后的内容 - with open(local_filename, 'w', encoding='utf-8') as f: - json.dump(data, f, ensure_ascii=False, indent=4) - - -def configure_model(model_dir, model_type): +def configure_model(model_dir, model_type, model_source): """配置模型""" - json_url = 'https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/mineru.template.json' - config_file_name = os.getenv('MINERU_TOOLS_CONFIG_JSON', 'mineru.json') - home_dir = os.path.expanduser('~') - config_file = os.path.join(home_dir, config_file_name) + config_file = get_tools_config_file_path() json_mods = { 'models-dir': { f'{model_type}': model_dir - } + }, + 'model-source': model_source, } - download_and_modify_json(json_url, config_file, json_mods) + download_and_modify_json(CONFIG_TEMPLATE_URL, config_file, json_mods) logger.info(f'The configuration file has been successfully configured, the path is: {config_file}') -def download_pipeline_models(): +def download_pipeline_models(model_source): """下载Pipeline模型""" model_paths = [ ModelPath.pp_doclayout_v2, @@ -79,14 +49,14 @@ def download_pipeline_models(): logger.info(f"Downloading model: {model_path}") download_finish_path = auto_download_and_get_model_root_path(model_path, repo_mode='pipeline') logger.info(f"Pipeline models downloaded successfully to: {download_finish_path}") - configure_model(download_finish_path, "pipeline") + configure_model(download_finish_path, "pipeline", model_source) -def download_vlm_models(): +def download_vlm_models(model_source): """下载VLM模型""" download_finish_path = auto_download_and_get_model_root_path("/", repo_mode='vlm') logger.info(f"VLM models downloaded successfully to: {download_finish_path}") - configure_model(download_finish_path, "vlm") + configure_model(download_finish_path, "vlm", model_source) def get_effective_download_model_source(requested_model_source): @@ -98,12 +68,12 @@ def get_effective_download_model_source(requested_model_source): f"`mineru-models-download` will temporarily use '{requested_model_source}' " f"to perform a real download." ) - return requested_model_source + return resolve_model_source(requested_model_source, allow_auto=True) if current_model_source is None: - return requested_model_source + return resolve_model_source(requested_model_source, allow_auto=True) - return current_model_source + return resolve_model_source(current_model_source) @contextmanager @@ -151,7 +121,7 @@ def download_models(model_source, model_type): model_source = click.prompt( "Please select the model download source: ", type=click.Choice(REMOTE_MODEL_SOURCES), - default='huggingface' + default='auto' ) effective_model_source = get_effective_download_model_source(model_source) @@ -169,12 +139,12 @@ def download_models(model_source, model_type): try: with temporary_model_source(effective_model_source): if model_type == 'pipeline': - download_pipeline_models() + download_pipeline_models(effective_model_source) elif model_type == 'vlm': - download_vlm_models() + download_vlm_models(effective_model_source) elif model_type == 'all': - download_pipeline_models() - download_vlm_models() + download_pipeline_models(effective_model_source) + download_vlm_models(effective_model_source) else: click.echo(f"Unsupported model type: {model_type}", err=True) sys.exit(1) diff --git a/mineru/utils/config_reader.py b/mineru/utils/config_reader.py index a04d882e..e1d5fa9e 100644 --- a/mineru/utils/config_reader.py +++ b/mineru/utils/config_reader.py @@ -30,6 +30,36 @@ def read_config(): return config +def get_configured_model_source(default: str | None = None) -> str | None: + """读取配置文件中的固定模型来源配置,auto 或缺失时返回默认值。""" + supported_sources = {'huggingface', 'modelscope'} + config = read_config() + if config is None: + return default + + model_source = config.get('model-source') + if model_source is None: + return default + if not isinstance(model_source, str): + logger.warning( + f"'model-source' in {CONFIG_FILE_NAME} must be a string, use {default} as default" + ) + return default + + normalized_model_source = model_source.strip().lower() + if not normalized_model_source: + return default + if normalized_model_source == "auto": + return default + if normalized_model_source in supported_sources: + return normalized_model_source + + logger.warning( + f"Unsupported 'model-source' in {CONFIG_FILE_NAME}: {model_source}, use {default} as default" + ) + return default + + def get_s3_config(bucket_name: str): """~/magic-pdf.json 读出来.""" config = read_config() diff --git a/mineru/utils/models_download_utils.py b/mineru/utils/models_download_utils.py index aa473c93..5c1b4bed 100644 --- a/mineru/utils/models_download_utils.py +++ b/mineru/utils/models_download_utils.py @@ -1,13 +1,167 @@ # Copyright (c) Opendatalab. All rights reserved. +import json import os from functools import lru_cache from huggingface_hub import snapshot_download as hf_snapshot_download +from loguru import logger from modelscope import snapshot_download as ms_snapshot_download +import requests -from mineru.utils.config_reader import get_local_models_dir +from mineru.utils.config_reader import get_configured_model_source, get_local_models_dir from mineru.utils.enum_class import ModelPath +MODEL_SOURCE_ENV_VAR = 'MINERU_MODEL_SOURCE' +CONFIG_TEMPLATE_URL = 'https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/mineru.template.json' +MINERU_CONFIG_VERSION = '1.3.2' +HUGGINGFACE_MODELS_PAGE_URL = "https://huggingface.co/models" +HUGGINGFACE_MODELS_PAGE_TIMEOUT = 3 +HUGGINGFACE_MODELS_PAGE_MAX_ATTEMPTS = 2 +REMOTE_MODEL_SOURCES = ("huggingface", "modelscope") + + +def get_tools_config_file_path() -> str: + """获取 MinerU 工具配置文件路径,支持环境变量指定绝对或相对路径。""" + config_file_name = os.getenv('MINERU_TOOLS_CONFIG_JSON', 'mineru.json') + if os.path.isabs(config_file_name): + return config_file_name + return os.path.join(os.path.expanduser('~'), config_file_name) + + +def download_json(url): + """下载 JSON 文件并返回解析后的内容。""" + response = requests.get(url) + response.raise_for_status() + return response.json() + + +def is_config_version_outdated(config_version): + """判断本地配置版本是否低于当前模板版本。""" + def version_tuple(version): + """将版本号字符串转换为可比较的整数元组。""" + parts = [] + for part in str(version).split('.'): + parts.append(int(part) if part.isdigit() else 0) + return tuple(parts) + + current_version = version_tuple(config_version) + target_version = version_tuple(MINERU_CONFIG_VERSION) + max_len = max(len(current_version), len(target_version)) + current_version += (0,) * (max_len - len(current_version)) + target_version += (0,) * (max_len - len(target_version)) + return current_version < target_version + + +def merge_config_dict(base_config: dict, override_config: dict, skip_keys: set[str] | None = None) -> dict: + """递归合并配置字典,用 override_config 覆盖 base_config 并保留新模板字段。""" + skip_keys = skip_keys or set() + merged_config = dict(base_config) + for key, value in override_config.items(): + if key in skip_keys: + continue + base_value = merged_config.get(key) + if isinstance(base_value, dict) and isinstance(value, dict): + merged_config[key] = merge_config_dict(base_value, value, skip_keys=skip_keys) + else: + merged_config[key] = value + return merged_config + + +def download_and_modify_json(url, local_filename, modifications): + """下载或读取 JSON 配置,并按 modifications 合并更新后写回。""" + if os.path.exists(local_filename): + with open(local_filename, encoding='utf-8') as f: + data = json.load(f) + config_version = data.get('config_version', '0.0.0') + if is_config_version_outdated(config_version): + template_data = download_json(url) + data = merge_config_dict(template_data, data, skip_keys={'config_version'}) + else: + data = download_json(url) + + data = merge_config_dict(data, modifications) + + with open(local_filename, 'w', encoding='utf-8') as f: + json.dump(data, f, ensure_ascii=False, indent=4) + + +def persist_resolved_model_source(model_source: str) -> None: + """将 auto 解析出的实际模型来源写入配置文件,避免下次启动时再次受网络波动影响。""" + if model_source not in REMOTE_MODEL_SOURCES: + return + try: + download_and_modify_json( + CONFIG_TEMPLATE_URL, + get_tools_config_file_path(), + {'model-source': model_source}, + ) + except Exception as exc: + logger.warning(f"Failed to persist resolved model source '{model_source}': {exc}") + + +@lru_cache(maxsize=1) +def resolve_auto_model_source() -> str: + """通过 Hugging Face 模型列表页探测 auto 应该使用的实际模型来源。""" + last_error = None + for _ in range(HUGGINGFACE_MODELS_PAGE_MAX_ATTEMPTS): + try: + response = requests.get( + HUGGINGFACE_MODELS_PAGE_URL, + timeout=HUGGINGFACE_MODELS_PAGE_TIMEOUT, + ) + if 200 <= response.status_code < 400: + return "huggingface" + last_error = f"status_code={response.status_code}" + except Exception as exc: + last_error = str(exc) + + logger.warning( + f"Failed to access {HUGGINGFACE_MODELS_PAGE_URL}: {last_error}, fallback to modelscope." + ) + return "modelscope" + + +def resolve_model_source(model_source: str | None = None, allow_auto: bool = False) -> str: + """将环境变量或配置文件中的模型来源解析为实际可下载的来源。""" + if model_source is None: + model_source = os.getenv(MODEL_SOURCE_ENV_VAR) + if isinstance(model_source, str) and model_source.strip().lower() == "auto": + raise ValueError( + f"{MODEL_SOURCE_ENV_VAR}=auto is not supported. " + f"Unset {MODEL_SOURCE_ENV_VAR} to use auto detection once, " + "or set it to huggingface/modelscope/local." + ) + if model_source is None: + model_source = get_configured_model_source() + if model_source is None: + model_source = "auto" + allow_auto = True + + if not isinstance(model_source, str): + logger.warning(f"Unsupported model source type: {type(model_source)}, fallback to auto.") + model_source = "auto" + allow_auto = True + + normalized_model_source = model_source.strip().lower() + if normalized_model_source == "local": + return "local" + if normalized_model_source == "auto": + if not allow_auto: + raise ValueError( + "model source auto is only supported for internal default detection " + "or explicit download command selection." + ) + resolved_model_source = resolve_auto_model_source() + persist_resolved_model_source(resolved_model_source) + return resolved_model_source + if normalized_model_source in REMOTE_MODEL_SOURCES: + return normalized_model_source + + logger.warning(f"Unsupported model source: {model_source}, fallback to auto.") + resolved_model_source = resolve_auto_model_source() + persist_resolved_model_source(resolved_model_source) + return resolved_model_source + @lru_cache(maxsize=None) def _snapshot_download_cached(model_source: str, repo_mode: str, repo: str, relative_path: str) -> str: @@ -40,7 +194,7 @@ def auto_download_and_get_model_root_path(relative_path: str, repo_mode='pipelin :param relative_path: 文件或目录相对路径 :return: 本地文件绝对路径或相对路径 """ - model_source = os.getenv('MINERU_MODEL_SOURCE', "huggingface") + model_source = resolve_model_source() if model_source == 'local': local_models_config = get_local_models_dir() @@ -53,22 +207,19 @@ def auto_download_and_get_model_root_path(relative_path: str, repo_mode='pipelin repo_mapping = { 'pipeline': { 'huggingface': ModelPath.pipeline_root_hf, - 'modelscope': ModelPath.pipeline_root_modelscope, - 'default': ModelPath.pipeline_root_hf + 'modelscope': ModelPath.pipeline_root_modelscope }, 'vlm': { 'huggingface': ModelPath.vlm_root_hf, - 'modelscope': ModelPath.vlm_root_modelscope, - 'default': ModelPath.vlm_root_hf + 'modelscope': ModelPath.vlm_root_modelscope } } if repo_mode not in repo_mapping: raise ValueError(f"Unsupported repo_mode: {repo_mode}, must be 'pipeline' or 'vlm'") - # 如果没有指定model_source或值不是'modelscope',则使用默认值 - repo = repo_mapping[repo_mode].get(model_source, repo_mapping[repo_mode]['default']) - + # model_source 已解析为实际远端来源后,再选择对应仓库。 + repo = repo_mapping[repo_mode][model_source] if repo_mode == 'pipeline': relative_path = relative_path.strip('/')