mirror of
https://github.com/opendatalab/MinerU.git
synced 2026-09-21 12:42:22 +08:00
+7
-5
@@ -2,8 +2,10 @@
|
||||
|
||||
## Project List
|
||||
|
||||
- [llama_index_rag](./llama_index_rag/README.md): Build a lightweight RAG system based on llama_index
|
||||
- [gradio_app](./gradio_app/README.md): Build a web app based on gradio
|
||||
- ~~[web_demo](./web_demo/README.md): MinerU online [demo](https://opendatalab.com/OpenSourceTools/Extractor/PDF/) localized deployment version~~(Deprecated)
|
||||
- [web_api](./web_api/README.md): Web API Based on FastAPI
|
||||
- [multi_gpu](./multi_gpu/README.md): Multi-GPU parallel processing based on LitServe
|
||||
- Projects compatible with version 2.0:
|
||||
- [gradio_app](./gradio_app/README.md): Web application based on Gradio
|
||||
|
||||
- Projects not yet compatible with version 2.0:
|
||||
- [web_api](./web_api/README.md): Web API based on FastAPI
|
||||
- [multi_gpu](./multi_gpu/README.md): Multi-GPU parallel processing based on LitServe
|
||||
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
|
||||
## 项目列表
|
||||
|
||||
- [llama_index_rag](./llama_index_rag/README_zh-CN.md): 基于 llama_index 构建轻量级 RAG 系统
|
||||
- [gradio_app](./gradio_app/README_zh-CN.md): 基于 Gradio 的 Web 应用
|
||||
- ~~[web_demo](./web_demo/README_zh-CN.md): MinerU在线[demo](https://opendatalab.com/OpenSourceTools/Extractor/PDF/)本地化部署版本~~(已过时)
|
||||
- [web_api](./web_api/README.md): 基于 FastAPI 的 Web API
|
||||
- [multi_gpu](./multi_gpu/README.md): 基于 LitServe 的多 GPU 并行处理
|
||||
- 已兼容2.0版本的项目列表
|
||||
- [gradio_app](./gradio_app/README_zh-CN.md): 基于 Gradio 的 Web 应用
|
||||
|
||||
- 未兼容2.0版本的项目列表
|
||||
- [web_api](./web_api/README.md): 基于 FastAPI 的 Web API
|
||||
- [multi_gpu](./multi_gpu/README.md): 基于 LitServe 的多 GPU 并行处理
|
||||
|
||||
@@ -1,160 +0,0 @@
|
||||
## Installation
|
||||
|
||||
MinerU
|
||||
|
||||
```bash
|
||||
git clone https://github.com/opendatalab/MinerU.git
|
||||
cd MinerU
|
||||
|
||||
conda create -n MinerU python=3.10
|
||||
conda activate MinerU
|
||||
pip install .[full] --extra-index-url https://wheels.myhloli.com
|
||||
```
|
||||
|
||||
Third-party software
|
||||
|
||||
```bash
|
||||
# install
|
||||
pip install llama-index-vector-stores-elasticsearch==0.2.0
|
||||
pip install llama-index-embeddings-dashscope==0.2.0
|
||||
pip install llama-index-core==0.10.68
|
||||
pip install einops==0.7.0
|
||||
pip install transformers-stream-generator==0.0.5
|
||||
pip install accelerate==0.33.0
|
||||
|
||||
# uninstall
|
||||
pip uninstall transformer-engine
|
||||
```
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
```
|
||||
export DASHSCOPE_API_KEY={some_key}
|
||||
export ES_USER={some_es_user}
|
||||
export ES_PASSWORD={some_es_password}
|
||||
export ES_URL=http://{es_url}:9200
|
||||
```
|
||||
For instructions on obtaining a DASHSCOPE_API_KEY, refer to [documentation](https://help.aliyun.com/zh/dashscope/opening-service)
|
||||
|
||||
## Usage
|
||||
|
||||
### Data Ingestion
|
||||
|
||||
```bash
|
||||
python data_ingestion.py -p some.pdf # load data from pdf
|
||||
|
||||
or
|
||||
|
||||
python data_ingestion.py -p /opt/data/some_pdf_directory/ # load data from multiples pdf which under the directory of {some_pdf_directory}
|
||||
```
|
||||
|
||||
### Query
|
||||
|
||||
```bash
|
||||
python query.py --question '{the_question_you_want_to_ask}'
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
````bash
|
||||
# Start the es service
|
||||
docker compose up -d
|
||||
|
||||
or
|
||||
|
||||
docker-compose up -d
|
||||
|
||||
|
||||
# Set environment variables
|
||||
export ES_USER=elastic
|
||||
export ES_PASSWORD=llama_index
|
||||
export ES_URL=http://127.0.0.1:9200
|
||||
export DASHSCOPE_API_KEY={some_key}
|
||||
|
||||
|
||||
# Ingest data
|
||||
python data_ingestion.py example/data/declaration_of_the_rights_of_man_1789.pdf
|
||||
|
||||
|
||||
# Ask a question
|
||||
python query.py -q 'how about the rights of men'
|
||||
|
||||
## outputs
|
||||
Please answer the question based on the content within ```:
|
||||
```
|
||||
I. Men are born, and always continue, free and equal in respect of their rights. Civil distinctions, therefore, can be founded only on public utility.
|
||||
```
|
||||
My question is:how about the rights of men。
|
||||
|
||||
question: how about the rights of men
|
||||
answer: The statement implies that men are born free and equal in terms of their rights. Civil distinctions should only be based on public utility. However, it does not specify what those rights are. It is up to society and individual countries to determine and protect the specific rights of their citizens.
|
||||
|
||||
````
|
||||
|
||||
## Development
|
||||
|
||||
`MinerU` provides a `RAG` integration interface, allowing users to specify a single input `pdf` file or a directory. `MinerU` will automatically parse the input files and return an iterable interface for retrieving the data.
|
||||
|
||||
|
||||
### API Interface
|
||||
|
||||
```python
|
||||
from magic_pdf.integrations.rag.type import Node
|
||||
|
||||
class RagPageReader:
|
||||
def get_rel_map(self) -> list[ElementRelation]:
|
||||
# Retrieve the relationships between nodes
|
||||
pass
|
||||
...
|
||||
|
||||
class RagDocumentReader:
|
||||
...
|
||||
|
||||
class DataReader:
|
||||
def __init__(self, path_or_directory: str, method: str, output_dir: str):
|
||||
pass
|
||||
|
||||
def get_documents_count(self) -> int:
|
||||
"""Get the number of pdf documents"""
|
||||
pass
|
||||
|
||||
def get_document_result(self, idx: int) -> RagDocumentReader | None:
|
||||
"""Retrieve the parsed content of a specific pdf"""
|
||||
pass
|
||||
|
||||
|
||||
def get_document_filename(self, idx: int) -> Path:
|
||||
"""Retrieve the path of a specific pdf"""
|
||||
pass
|
||||
|
||||
|
||||
```
|
||||
|
||||
Type Definitions
|
||||
|
||||
```python
|
||||
|
||||
|
||||
class Node(BaseModel):
|
||||
category_type: CategoryType = Field(description='Category') # Category
|
||||
text: str | None = Field(description='Text content', default=None)
|
||||
image_path: str | None = Field(description='Path to image or table (table may be stored as an image)', default=None)
|
||||
anno_id: int = Field(description='Unique ID', default=-1)
|
||||
latex: str | None = Field(description='LaTeX output for equations or tables', default=None)
|
||||
html: str | None = Field(description='HTML output for tables', default=None)
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
Tables can be stored in one of three formats: image, LaTeX, or HTML.
|
||||
`anno_id` is a globally unique ID for each Node. It can be used later to match this Node with other Nodes. The relationships between nodes can be retrieved using the `get_rel_map` method. Users can use `anno_id` to link nodes and construct a RAG index that includes node relationships.
|
||||
|
||||
|
||||
### Node Relationship Matrix
|
||||
|
||||
| | image_body | table_body |
|
||||
| -------------- | ---------- | ---------- |
|
||||
| image_caption | sibling | |
|
||||
| table_caption | | sibling |
|
||||
| table_footnote | | sibling |
|
||||
@@ -1,199 +0,0 @@
|
||||
<details open="open">
|
||||
<summary><h2 style="display: inline-block">目录</h2></summary>
|
||||
<li><a href="#介绍">介绍</a></li>
|
||||
<li><a href="#安装">安装</a></li>
|
||||
<li><a href="#示例">示例</a></li>
|
||||
<li><a href="#开发">开发</a></li>
|
||||
</ol>
|
||||
</details>
|
||||
|
||||
## 介绍
|
||||
|
||||
`MinerU` 提供数据 `API接口` 以支持用户导入数据到 `RAG` 系统。本项目将基于`通义千问`展示如何构建一个轻量级的 `RAG` 系统。
|
||||
|
||||
<p align="center">
|
||||
<img src="rag_data_api.png" width="300px" style="vertical-align:middle;">
|
||||
</p>
|
||||
|
||||
## 安装
|
||||
|
||||
环境要求
|
||||
|
||||
```text
|
||||
NVIDIA A100 80GB,
|
||||
Centos 7 3.10.0-957.el7.x86_64
|
||||
|
||||
Client: Docker Engine - Community
|
||||
Version: 24.0.5
|
||||
API version: 1.43
|
||||
Go version: go1.20.6
|
||||
Git commit: ced0996
|
||||
Built: Fri Jul 21 20:39:02 2023
|
||||
OS/Arch: linux/amd64
|
||||
Context: default
|
||||
|
||||
Server: Docker Engine - Community
|
||||
Engine:
|
||||
Version: 24.0.5
|
||||
API version: 1.43 (minimum version 1.12)
|
||||
Go version: go1.20.6
|
||||
Git commit: a61e2b4
|
||||
Built: Fri Jul 21 20:38:05 2023
|
||||
OS/Arch: linux/amd64
|
||||
Experimental: false
|
||||
containerd:
|
||||
Version: 1.6.25
|
||||
GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
|
||||
runc:
|
||||
Version: 1.1.10
|
||||
GitCommit: v1.1.10-0-g18a0cb0
|
||||
docker-init:
|
||||
Version: 0.19.0
|
||||
GitCommit: de40ad0
|
||||
```
|
||||
|
||||
请参考[文档](../../README_zh-CN.md) 安装 MinerU
|
||||
|
||||
第三方软件
|
||||
|
||||
```bash
|
||||
# install
|
||||
pip install modelscope==1.14.0
|
||||
pip install llama-index-vector-stores-elasticsearch==0.2.0
|
||||
pip install llama-index-embeddings-dashscope==0.2.0
|
||||
pip install llama-index-core==0.10.68
|
||||
pip install einops==0.7.0
|
||||
pip install transformers-stream-generator==0.0.5
|
||||
pip install accelerate==0.33.0
|
||||
|
||||
# uninstall
|
||||
pip uninstall transformer-engine
|
||||
```
|
||||
|
||||
|
||||
## 示例
|
||||
|
||||
````bash
|
||||
cd projects/llama_index_rag
|
||||
|
||||
docker compose up -d
|
||||
|
||||
or
|
||||
|
||||
docker-compose up -d
|
||||
|
||||
|
||||
# 配置环境变量
|
||||
|
||||
export ES_USER=elastic
|
||||
export ES_PASSWORD=llama_index
|
||||
export ES_URL=http://127.0.0.1:9200
|
||||
export DASHSCOPE_API_KEY={some_key}
|
||||
|
||||
|
||||
DASHSCOPE_API_KEY 开通参考[文档](https://help.aliyun.com/zh/dashscope/opening-service)
|
||||
|
||||
# 未导入数据,查询问题。返回通义千问默认答案
|
||||
python query.py -q 'how about the rights of men'
|
||||
|
||||
## outputs
|
||||
question: how about the rights of men
|
||||
answer: The topic of men's rights often refers to discussions around legal, social, and political issues that affect men specifically or differently from women. Movements related to men's rights advocate for addressing areas where men face discrimination or unique challenges, such as:
|
||||
|
||||
Child Custody: Ensuring that men have equal opportunities for custody of their children following divorce or separation.
|
||||
Domestic Violence: Recognizing that men can also be victims of domestic abuse and ensuring they have access to support services.
|
||||
Mental Health and Suicide Rates: Addressing the higher rates of suicide among men and providing mental health resources.
|
||||
Military Conscription: In some countries, only men are required to register for military service, which is seen as a gender-based obligation.
|
||||
Workplace Safety: Historically, more men than women have been employed in high-risk occupations, leading to higher workplace injury and death rates.
|
||||
Parental Leave: Advocating for paternity leave policies that allow men to take time off work for family care.
|
||||
Men's rights activism often intersects with broader discussions on gender equality and aims to promote fairness and equity across genders. It's important to note that while advocating for these issues, it should be done in a way that does not detract from or oppose the goals of gender equality and the rights of other groups. The focus should be on creating a fair society where everyone has equal opportunities and protections under the law.
|
||||
|
||||
|
||||
# 导入数据
|
||||
python data_ingestion.py -p example/data/
|
||||
|
||||
or
|
||||
|
||||
python data_ingestion.py -p example/data/declaration_of_the_rights_of_man_1789.pdf
|
||||
|
||||
|
||||
# 导入数据后,查询问题。通义千问模型会根据 RAG 系统的检索结果,结合上下文,给出答案。
|
||||
|
||||
python query.py -q 'how about the rights of men'
|
||||
|
||||
## outputs
|
||||
请基于```内的内容回答问题。"
|
||||
```
|
||||
I. Men are born, and always continue, free and equal in respect of their rights. Civil distinctions, therefore, can be founded only on public utility.
|
||||
```
|
||||
我的问题是:how about the rights of men。
|
||||
|
||||
question: how about the rights of men
|
||||
answer: The statement implies that men are born free and equal in terms of their rights. Civil distinctions should only be based on public utility. However, it does not specify what those rights are. It is up to society and individual countries to determine and protect the specific rights of their citizens.
|
||||
|
||||
````
|
||||
|
||||
## 开发
|
||||
|
||||
`MinerU` 提供了 `RAG` 集成接口,用户可以通过指定输入单个 `pdf` 文件或者某个目录。`MinerU` 会自动解析输入文件并返回可以迭代的接口用于获取数据
|
||||
|
||||
### API 接口
|
||||
|
||||
```python
|
||||
from magic_pdf.integrations.rag.type import Node
|
||||
|
||||
class RagPageReader:
|
||||
def get_rel_map(self) -> list[ElementRelation]:
|
||||
# 获取节点间的关系
|
||||
pass
|
||||
...
|
||||
|
||||
class RagDocumentReader:
|
||||
...
|
||||
|
||||
class DataReader:
|
||||
def __init__(self, path_or_directory: str, method: str, output_dir: str):
|
||||
pass
|
||||
|
||||
def get_documents_count(self) -> int:
|
||||
"""获取 pdf 文档数量"""
|
||||
pass
|
||||
|
||||
def get_document_result(self, idx: int) -> RagDocumentReader | None:
|
||||
"""获取某个 pdf 的解析内容"""
|
||||
pass
|
||||
|
||||
|
||||
def get_document_filename(self, idx: int) -> Path:
|
||||
"""获取某个 pdf 的具体路径"""
|
||||
pass
|
||||
|
||||
|
||||
```
|
||||
|
||||
类型定义
|
||||
|
||||
```python
|
||||
|
||||
class Node(BaseModel):
|
||||
category_type: CategoryType = Field(description='类别') # 类别
|
||||
text: str | None = Field(description='文本内容',
|
||||
default=None)
|
||||
image_path: str | None = Field(description='图或者表格(表可能用图片形式存储)的存储路径',
|
||||
default=None)
|
||||
anno_id: int = Field(description='unique id', default=-1)
|
||||
latex: str | None = Field(description='公式或表格 latex 解析结果', default=None)
|
||||
html: str | None = Field(description='表格的 html 解析结果', default=None)
|
||||
|
||||
```
|
||||
|
||||
表格存储形式可能会是 图片、latex、html 三种形式之一。
|
||||
anno_id 是该 Node 的在全局唯一ID。后续可以用于匹配该 Node 和其他 Node 的关系。节点的关系可以通过方法 `get_rel_map` 获取。用户可以用 `anno_id` 匹配节点之间的关系,并用于构建具备节点的关系的 rag index。
|
||||
|
||||
### 节点类型关系矩阵
|
||||
|
||||
| | image_body | table_body |
|
||||
| -------------- | ---------- | ---------- |
|
||||
| image_caption | sibling | |
|
||||
| table_caption | | sibling |
|
||||
| table_footnote | | sibling |
|
||||
@@ -1,68 +0,0 @@
|
||||
import os
|
||||
|
||||
import click
|
||||
from llama_index.core.schema import TextNode
|
||||
from llama_index.embeddings.dashscope import (DashScopeEmbedding,
|
||||
DashScopeTextEmbeddingModels,
|
||||
DashScopeTextEmbeddingType)
|
||||
from llama_index.vector_stores.elasticsearch import ElasticsearchStore
|
||||
|
||||
from magic_pdf.integrations.rag.api import DataReader
|
||||
|
||||
es_vec_store = ElasticsearchStore(
|
||||
index_name='rag_index',
|
||||
es_url=os.getenv('ES_URL', 'http://127.0.0.1:9200'),
|
||||
es_user=os.getenv('ES_USER', 'elastic'),
|
||||
es_password=os.getenv('ES_PASSWORD', 'llama_index'),
|
||||
)
|
||||
|
||||
|
||||
# Create embeddings
|
||||
# text_type=`document` to build index
|
||||
def embed_node(node):
|
||||
embedder = DashScopeEmbedding(
|
||||
model_name=DashScopeTextEmbeddingModels.TEXT_EMBEDDING_V2,
|
||||
text_type=DashScopeTextEmbeddingType.TEXT_TYPE_DOCUMENT,
|
||||
)
|
||||
|
||||
result_embeddings = embedder.get_text_embedding(node.text)
|
||||
node.embedding = result_embeddings
|
||||
return node
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
'-p',
|
||||
'--path',
|
||||
'path',
|
||||
type=click.Path(exists=True),
|
||||
required=True,
|
||||
help='local pdf filepath or directory',
|
||||
)
|
||||
def cli(path):
|
||||
output_dir = '/tmp/magic_pdf/integrations/rag/'
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
documents = DataReader(path, 'ocr', output_dir)
|
||||
|
||||
# build nodes
|
||||
nodes = []
|
||||
|
||||
for idx in range(documents.get_documents_count()):
|
||||
doc = documents.get_document_result(idx)
|
||||
if doc is None: # something wrong happens when parse pdf !
|
||||
continue
|
||||
|
||||
for page in iter(
|
||||
doc): # iterate documents from initial page to last page !
|
||||
for element in iter(page): # iterate the element from all page !
|
||||
if element.text is None:
|
||||
continue
|
||||
nodes.append(
|
||||
embed_node(
|
||||
TextNode(text=element.text,
|
||||
metadata={'purpose': 'demo'})))
|
||||
es_vec_store.add(nodes)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
cli()
|
||||
@@ -1,24 +0,0 @@
|
||||
services:
|
||||
es:
|
||||
container_name: es
|
||||
image: docker.elastic.co/elasticsearch/elasticsearch:8.11.3
|
||||
volumes:
|
||||
- esdata01:/usr/share/elasticsearch/data
|
||||
ports:
|
||||
- 9200:9200
|
||||
environment:
|
||||
- node.name=es
|
||||
- ELASTIC_PASSWORD=llama_index
|
||||
- bootstrap.memory_lock=false
|
||||
- discovery.type=single-node
|
||||
- xpack.security.enabled=true
|
||||
- xpack.security.http.ssl.enabled=false
|
||||
- xpack.security.transport.ssl.enabled=false
|
||||
ulimits:
|
||||
memlock:
|
||||
soft: -1
|
||||
hard: -1
|
||||
restart: always
|
||||
volumes:
|
||||
esdata01:
|
||||
driver: local
|
||||
Binary file not shown.
@@ -1,84 +0,0 @@
|
||||
import os
|
||||
|
||||
import click
|
||||
from llama_index.core.vector_stores.types import VectorStoreQuery
|
||||
from llama_index.embeddings.dashscope import (DashScopeEmbedding,
|
||||
DashScopeTextEmbeddingModels,
|
||||
DashScopeTextEmbeddingType)
|
||||
from llama_index.vector_stores.elasticsearch import (AsyncDenseVectorStrategy,
|
||||
ElasticsearchStore)
|
||||
# initialize qwen 7B model
|
||||
from modelscope import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
|
||||
|
||||
es_vector_store = ElasticsearchStore(
|
||||
index_name='rag_index',
|
||||
es_url=os.getenv('ES_URL', 'http://127.0.0.1:9200'),
|
||||
es_user=os.getenv('ES_USER', 'elastic'),
|
||||
es_password=os.getenv('ES_PASSWORD', 'llama_index'),
|
||||
retrieval_strategy=AsyncDenseVectorStrategy(),
|
||||
)
|
||||
|
||||
|
||||
def embed_text(text):
|
||||
embedder = DashScopeEmbedding(
|
||||
model_name=DashScopeTextEmbeddingModels.TEXT_EMBEDDING_V2,
|
||||
text_type=DashScopeTextEmbeddingType.TEXT_TYPE_DOCUMENT,
|
||||
)
|
||||
return embedder.get_text_embedding(text)
|
||||
|
||||
|
||||
def search(vector_store: ElasticsearchStore, query: str):
|
||||
query_vec = VectorStoreQuery(query_embedding=embed_text(query))
|
||||
result = vector_store.query(query_vec)
|
||||
return '\n'.join([node.text for node in result.nodes])
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
'-q',
|
||||
'--question',
|
||||
'question',
|
||||
required=True,
|
||||
help='ask what you want to know!',
|
||||
)
|
||||
def cli(question):
|
||||
tokenizer = AutoTokenizer.from_pretrained('qwen/Qwen-7B-Chat',
|
||||
revision='v1.0.5',
|
||||
trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained('qwen/Qwen-7B-Chat',
|
||||
revision='v1.0.5',
|
||||
device_map='auto',
|
||||
trust_remote_code=True,
|
||||
fp32=True).eval()
|
||||
model.generation_config = GenerationConfig.from_pretrained(
|
||||
'Qwen/Qwen-7B-Chat', revision='v1.0.5', trust_remote_code=True)
|
||||
|
||||
# define a prompt template for the vectorDB-enhanced LLM generation
|
||||
def answer_question(question, context, model):
|
||||
if context == '':
|
||||
prompt = question
|
||||
else:
|
||||
prompt = f'''请基于```内的内容回答问题。"
|
||||
```
|
||||
{context}
|
||||
```
|
||||
我的问题是:{question}。
|
||||
'''
|
||||
history = None
|
||||
print(prompt)
|
||||
response, history = model.chat(tokenizer, prompt, history=None)
|
||||
return response
|
||||
|
||||
answer = answer_question(question, search(es_vector_store, question),
|
||||
model)
|
||||
print(f'question: {question}\n'
|
||||
f'answer: {answer}')
|
||||
|
||||
|
||||
"""
|
||||
|
||||
python query.py -q 'how about the rights of men'
|
||||
"""
|
||||
|
||||
if __name__ == '__main__':
|
||||
cli()
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 9.3 KiB |
@@ -1,24 +0,0 @@
|
||||
# Logs
|
||||
logs
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
pnpm-debug.log*
|
||||
lerna-debug.log*
|
||||
|
||||
node_modules
|
||||
dist
|
||||
dist-ssr
|
||||
*.local
|
||||
|
||||
# Editor directories and files
|
||||
.vscode/*
|
||||
!.vscode/extensions.json
|
||||
.idea
|
||||
.DS_Store
|
||||
*.suo
|
||||
*.ntvs*
|
||||
*.njsproj
|
||||
*.sln
|
||||
*.sw?
|
||||
@@ -1,55 +0,0 @@
|
||||
# MinerU web
|
||||
|
||||
## Table of Contents
|
||||
- [Local Frontend Development](#local-frontend-development)
|
||||
- [Technology Stack](#technology-stack)
|
||||
|
||||
## Local Frontend Development
|
||||
|
||||
### Prerequisites
|
||||
- Node.js 18.x
|
||||
- pnpm
|
||||
|
||||
### Installation Steps
|
||||
|
||||
1. Install Node.js 18
|
||||
- Visit the [Node.js official website](https://nodejs.org/) to download and install Node.js version 18.x
|
||||
|
||||
2. Install pnpm
|
||||
```bash
|
||||
npm install -g pnpm
|
||||
3. Clone the repository
|
||||
```git clone https://github.com/opendatalab/MinerU
|
||||
cd ./projects/web
|
||||
```
|
||||
4. Install dependencies
|
||||
```
|
||||
pnpm install
|
||||
```
|
||||
5. Run the development server
|
||||
```
|
||||
pnpm run dev
|
||||
```
|
||||
6. ⚠️ Note: This command is for local development only, do not use for deployment!
|
||||
Open your browser and visit http://localhost:5173 (or another address output in the console)
|
||||
|
||||
7. Ensure that the backend service in ./projects/web_demo is running
|
||||
|
||||
8. If you encounter an error when executing `pnpm install`, you can switch to an alternative package manager.
|
||||
```
|
||||
npm install -g yarn
|
||||
yarn
|
||||
yarn start
|
||||
```
|
||||
|
||||
|
||||
## Building the Project
|
||||
```
|
||||
pnpm run build
|
||||
```
|
||||
## Technology Stack
|
||||
- React
|
||||
- Tailwind CSS
|
||||
- typeScript
|
||||
- zustand
|
||||
- ahooks
|
||||
@@ -1,61 +0,0 @@
|
||||
# MinerU web
|
||||
|
||||
|
||||
## 目录
|
||||
- [前端本地开发](#前端本地开发)
|
||||
- [技术栈](#技术栈)
|
||||
## 前端本地开发
|
||||
|
||||
### 前置条件
|
||||
- Node.js 18.x
|
||||
- pnpm
|
||||
|
||||
### 安装步骤
|
||||
|
||||
1. 安装 Node.js 18
|
||||
- 访问 [Node.js 官网](https://nodejs.org/) 下载并安装 Node.js 18.x 版本
|
||||
|
||||
2. 安装 pnpm
|
||||
```bash
|
||||
npm install -g pnpm
|
||||
```
|
||||
3. 克隆仓库
|
||||
```
|
||||
1. git clone https://github.com/opendatalab/MinerU
|
||||
2. cd ./projects/web
|
||||
```
|
||||
|
||||
4. 安装依赖
|
||||
```
|
||||
pnpm install
|
||||
```
|
||||
|
||||
5. 运行开发服务器
|
||||
```
|
||||
pnpm run dev
|
||||
```
|
||||
|
||||
6. ⚠️ 注意:此命令仅用于本地开发,不要用于部署!
|
||||
打开浏览器访问 http://localhost:5173(或控制台输出的其他地址)
|
||||
构建项目
|
||||
要构建生产版本,请执行以下命令:
|
||||
|
||||
```
|
||||
pnpm run build
|
||||
```
|
||||
7. 请确保./projects/web_demo后端服务启动
|
||||
|
||||
8. 如果pnpm install执行error,可更换包管理器
|
||||
```
|
||||
npm install -g yarn
|
||||
yarn
|
||||
yarn start
|
||||
```
|
||||
|
||||
## 技术栈
|
||||
|
||||
- React
|
||||
- Tailwind CSS
|
||||
- typeScript
|
||||
- zustand
|
||||
- ahooks
|
||||
Vendored
-28
@@ -1,28 +0,0 @@
|
||||
import js from '@eslint/js'
|
||||
import globals from 'globals'
|
||||
import reactHooks from 'eslint-plugin-react-hooks'
|
||||
import reactRefresh from 'eslint-plugin-react-refresh'
|
||||
import tseslint from 'typescript-eslint'
|
||||
|
||||
export default tseslint.config(
|
||||
{ ignores: ['dist'] },
|
||||
{
|
||||
extends: [js.configs.recommended, ...tseslint.configs.recommended],
|
||||
files: ['**/*.{ts,tsx}'],
|
||||
languageOptions: {
|
||||
ecmaVersion: 2020,
|
||||
globals: globals.browser,
|
||||
},
|
||||
plugins: {
|
||||
'react-hooks': reactHooks,
|
||||
'react-refresh': reactRefresh,
|
||||
},
|
||||
rules: {
|
||||
...reactHooks.configs.recommended.rules,
|
||||
'react-refresh/only-export-components': [
|
||||
'warn',
|
||||
{ allowConstantExport: true },
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
@@ -1,13 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<link rel="icon" type="image/svg+xml" href="/logo.svg" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>MinerU</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/main.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
Generated
-15512
File diff suppressed because it is too large
Load Diff
@@ -1,66 +0,0 @@
|
||||
{
|
||||
"name": "my-react-app",
|
||||
"private": true,
|
||||
"version": "0.0.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite --host ",
|
||||
"build": "tsc --noEmit && vite build",
|
||||
"lint": "eslint .",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"@ant-design/icons": "^5.4.0",
|
||||
"@codemirror/view": "^6.33.0",
|
||||
"@tanstack/react-query": "^5.56.2",
|
||||
"@types/lodash": "^4.17.7",
|
||||
"@types/qs": "^6.9.15",
|
||||
"@types/react-copy-to-clipboard": "^5.0.7",
|
||||
"@types/react-syntax-highlighter": "^15.5.13",
|
||||
"@uiw/codemirror-extensions-langs": "^4.23.0",
|
||||
"@uiw/react-codemirror": "^4.23.0",
|
||||
"ahooks": "^3.8.1",
|
||||
"antd": "^5.20.3",
|
||||
"axios": "^1.7.5",
|
||||
"canvas": "^2.11.2",
|
||||
"classnames": "^2.5.1",
|
||||
"js-cookie": "^3.0.5",
|
||||
"lodash": "^4.17.21",
|
||||
"path2d": "^0.2.1",
|
||||
"qs": "^6.13.0",
|
||||
"react": "^18.3.1",
|
||||
"react-copy-to-clipboard": "^5.1.0",
|
||||
"react-dom": "^18.3.1",
|
||||
"react-intl": "^6.6.8",
|
||||
"react-markdown": "^9.0.1",
|
||||
"react-query": "^3.39.3",
|
||||
"react-router-dom": "^6.26.1",
|
||||
"react-syntax-highlighter": "^15.5.0",
|
||||
"rehype-katex": "^7.0.1",
|
||||
"rehype-raw": "^7.0.0",
|
||||
"remark-gfm": "^4.0.0",
|
||||
"remark-math": "^6.0.0",
|
||||
"zustand": "^4.5.5"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "^9.9.0",
|
||||
"@types/js-cookie": "^3.0.6",
|
||||
"@types/node": "^22.5.1",
|
||||
"@types/react": "^18.3.3",
|
||||
"@types/react-dom": "^18.3.0",
|
||||
"@vitejs/plugin-react": "^4.3.1",
|
||||
"autoprefixer": "^10.4.20",
|
||||
"eslint": "^9.9.0",
|
||||
"eslint-plugin-react-hooks": "^5.1.0-rc.0",
|
||||
"eslint-plugin-react-refresh": "^0.4.9",
|
||||
"globals": "^15.9.0",
|
||||
"less": "^4.2.0",
|
||||
"postcss": "^8.4.41",
|
||||
"sass-embedded": "^1.77.8",
|
||||
"tailwindcss": "^3.4.10",
|
||||
"ts-prune": "^0.10.3",
|
||||
"typescript": "^5.5.3",
|
||||
"typescript-eslint": "^8.0.1",
|
||||
"vite": "^5.4.1"
|
||||
}
|
||||
}
|
||||
Generated
-5951
File diff suppressed because it is too large
Load Diff
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
export default {
|
||||
plugins: {
|
||||
tailwindcss: {},
|
||||
autoprefixer: {},
|
||||
},
|
||||
}
|
||||
Vendored
-1
File diff suppressed because one or more lines are too long
@@ -1,30 +0,0 @@
|
||||
<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<g clip-path="url(#clip0_1924_30132)">
|
||||
<path opacity="0.4" d="M2.1688 9.68828C2.14583 9.59322 2.16383 9.49288 2.21842 9.41173L6.04222 3.7273C6.09089 3.65495 6.1649 3.60346 6.24966 3.58298L16.6007 1.08163C16.7879 1.03639 16.9763 1.15147 17.0216 1.33868L19.658 12.2487C20.7217 16.6504 18.0157 21.0808 13.6141 22.1445C9.21248 23.2082 4.782 20.5022 3.71834 16.1006L2.1688 9.68828Z" fill="url(#paint0_linear_1924_30132)"/>
|
||||
<path d="M5.19531 7.24425C5.19531 7.14645 5.23638 7.05315 5.3085 6.98709L10.3605 2.35987C10.4248 2.30098 10.5089 2.26831 10.5961 2.26831H21.2451C21.4377 2.26831 21.5938 2.42444 21.5938 2.61703V13.8411C21.5938 18.3694 17.9229 22.0404 13.3946 22.0404C8.86624 22.0404 5.19531 18.3694 5.19531 13.8411V7.24425Z" fill="url(#paint1_linear_1924_30132)"/>
|
||||
<path d="M9.87906 7.30192L5.28333 7.01537L10.4111 2.32143L10.626 6.5632C10.6475 6.98306 10.2987 7.32808 9.87906 7.30192Z" fill="#5D76FF"/>
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M15.7002 11.8158V13.0479C15.7002 14.0038 14.9253 14.7787 13.9694 14.7787C13.0135 14.7787 12.2386 14.0038 12.2386 13.0479V9.96771H10.2145V13.3413C10.2145 15.4151 11.8956 17.0962 13.9694 17.0962C16.0432 17.0962 17.7243 15.4151 17.7243 13.3413V9.96771L17.7243 11.8158H15.7002Z" fill="#0028FD"/>
|
||||
<path d="M17.7243 10.9944H18.5457V11.8158L17.7243 11.8158L17.7243 10.9944Z" fill="#0028FD"/>
|
||||
<path d="M15.7002 10.4957H17.0203V11.8158L15.7002 11.8158L15.7002 10.4957Z" fill="#0028FD"/>
|
||||
<path d="M17.0203 9.7917H17.7243V10.4957L17.0203 10.4957L17.0203 9.7917Z" fill="#0028FD"/>
|
||||
<path d="M18.135 8.61828H18.5751V9.05831H18.135V8.61828Z" fill="#0028FD"/>
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M15.4627 11.7367V12.9688C15.4627 13.9246 14.6878 14.6995 13.7319 14.6995C12.776 14.6995 12.0011 13.9246 12.0011 12.9688V9.88854H9.97697V13.2621C9.97697 15.3359 11.6581 17.017 13.7319 17.017C15.8057 17.017 17.4868 15.3359 17.4868 13.2621V9.88854L17.4868 11.7367H15.4627Z" fill="white"/>
|
||||
<path d="M17.4868 10.9153H18.3082V11.7367L17.4868 11.7367L17.4868 10.9153Z" fill="white"/>
|
||||
<path d="M15.4627 10.4166H16.7828V11.7367L15.4627 11.7367L15.4627 10.4166Z" fill="white"/>
|
||||
<path d="M16.7828 9.71253H17.4868V10.4166L16.7828 10.4166L16.7828 9.71253Z" fill="white"/>
|
||||
<path d="M17.8975 8.53912H18.3376V8.97915H17.8975V8.53912Z" fill="white"/>
|
||||
</g>
|
||||
<defs>
|
||||
<linearGradient id="paint0_linear_1924_30132" x1="0.16149" y1="7.29712" x2="19.8967" y2="12.4412" gradientUnits="userSpaceOnUse">
|
||||
<stop stop-color="#1543FE"/>
|
||||
<stop offset="1" stop-color="#8C46FF"/>
|
||||
</linearGradient>
|
||||
<linearGradient id="paint1_linear_1924_30132" x1="3.80582" y1="4.44849" x2="21.7806" y2="14.0843" gradientUnits="userSpaceOnUse">
|
||||
<stop stop-color="#1543FE"/>
|
||||
<stop offset="1" stop-color="#8C46FF"/>
|
||||
</linearGradient>
|
||||
<clipPath id="clip0_1924_30132">
|
||||
<rect width="24" height="24" fill="white"/>
|
||||
</clipPath>
|
||||
</defs>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 2.9 KiB |
-19814
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -1,3 +0,0 @@
|
||||
àRCopyright 1990-2009 Adobe Systems Incorporated.
|
||||
All rights reserved.
|
||||
See ./LICENSEáCNS2-H
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -1,3 +0,0 @@
|
||||
àRCopyright 1990-2009 Adobe Systems Incorporated.
|
||||
All rights reserved.
|
||||
See ./LICENSEá ETen-B5-H` ^
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -1,4 +0,0 @@
|
||||
àRCopyright 1990-2009 Adobe Systems Incorporated.
|
||||
All rights reserved.
|
||||
See ./LICENSE!!�º]aX!!]`�21�> �p�z�$]‚�"R‚d�-Uƒ7�*„␍4„%�+ „Z „{�/…%…<�9K…b�1]†.�"‡‰`]‡,�"]ˆ
|
||||
�"]ˆh�"]‰F�"]Š$�"]‹�"]‹`�"]Œ>�"]��"]�z�"]ŽX�"]�6�"]��"]�r�"]‘P�"]’.�"]“�"]“j�"]”H�"]•&�"]–�"]–b�"]—@�"]˜�"]˜|�"]™Z�"]š8�"]›�"]›t�"]œR�"]�0�"]ž�"]žl�"]ŸJ�"] (�"]¡�"]¡d�"]¢B�"]£ �"X£~�']¤W�"]¥5�"]¦�"]¦q�"]§O�"]¨-�"]©�"]©i�"]ªG�"]«%�"]¬�"]¬a�"]?�"]®�"]®{�"]¯Y�"]°7�"]±�"]±s�"]²Q�"]³/�"]´␍�"]´k�"]µI�"]¶'�"]·�"]·c�"]¸A�"]¹�"]¹}�"]º[�"]»9
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user