mirror of
https://github.com/ooyinet/WeClone.git
synced 2026-08-30 22:40:50 +08:00
Merge remote-tracking branch 'origin/master' into Refactoring-Data-Processing
This commit is contained in:
xming521
@@ -1,200 +1,211 @@
|
||||
|
||||
|
||||
## 核心功能✨
|
||||
- 💬 使用微信聊天记录微调LLM
|
||||
- 🎙️ 使用微信语音消息➕0.5B大模型实现高质量声音克隆 👉[WeClone-audio](https://github.com/xming521/WeClone/tree/master/WeClone-audio)
|
||||
- 🔗 绑定到微信机器人，实现自己的数字分身
|
||||
|
||||
## 特性与说明📋
|
||||
|
||||
> [!TIP]
|
||||
> 新特性：[WeClone-audio](https://github.com/xming521/WeClone/tree/master/WeClone-audio) 模块，支持对微信语音进行克隆。
|
||||
|
||||
|
||||
> [!IMPORTANT]
|
||||
> 微调LLM最终效果很大程度取决于聊天数据的数量和质量
|
||||
|
||||
### 硬件要求
|
||||
|
||||
目前项目默认使用chatglm3-6b模型，LoRA方法对sft阶段微调，大约需要16GB显存。也可以使用[LLaMA Factory](https://github.com/hiyouga/LLaMA-Factory/blob/main/README_zh.md#%E6%A8%A1%E5%9E%8B)支持的其他模型和方法，占用显存更少，需要自行修改模板的system提示词等相关配置。
|
||||
|
||||
需要显存的估算值：
|
||||
| 方法 | 精度 | 7B | 14B | 30B | 70B | `x`B |
|
||||
| ------------------------------- | ---- | ----- | ----- | ----- | ------ | ------- |
|
||||
| Full (`bf16` or `fp16`) | 32 | 120GB | 240GB | 600GB | 1200GB | `18x`GB |
|
||||
| Full (`pure_bf16`) | 16 | 60GB | 120GB | 300GB | 600GB | `8x`GB |
|
||||
| Freeze/LoRA/GaLore/APOLLO/BAdam | 16 | 16GB | 32GB | 64GB | 160GB | `2x`GB |
|
||||
| QLoRA | 8 | 10GB | 20GB | 40GB | 80GB | `x`GB |
|
||||
| QLoRA | 4 | 6GB | 12GB | 24GB | 48GB | `x/2`GB |
|
||||
| QLoRA | 2 | 4GB | 8GB | 16GB | 24GB | `x/4`GB |
|
||||
|
||||
|
||||
### 环境搭建
|
||||
建议使用 [uv](https://docs.astral.sh/uv/)，这是一个非常快速的 Python 环境管理器。安装uv后，您可以使用以下命令创建一个新的Python环境并安装依赖项，注意这不包含xcodec（音频克隆）功能的依赖：
|
||||
```bash
|
||||
git clone https://github.com/xming521/WeClone.git
|
||||
cd WeClone
|
||||
uv venv .venv --python=3.9
|
||||
source .venv/bin/activate
|
||||
uv pip install --group main -e .
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> 训练以及推理相关配置统一在文件[settings.json](settings.json)
|
||||
|
||||
|
||||
### 数据准备
|
||||
|
||||
请使用[PyWxDump](https://github.com/xaoyaoo/PyWxDump)提取微信聊天记录。下载软件并解密数据库后，点击聊天备份，导出类型为CSV，可以导出多个联系人或群聊，然后将导出的位于`wxdump_tmp/export` 的 `csv` 文件夹放在`./data`目录即可，也就是不同人聊天记录的文件夹一起放在 `./data/csv`。 示例数据位于[data/example_chat.csv](data/example_chat.csv)。
|
||||
|
||||
### 数据预处理
|
||||
|
||||
项目默认去除了数据中的手机号、身份证号、邮箱、网址。还提供了一个禁用词词库[blocked_words](make_dataset/blocked_words.json)，可以自行添加需要过滤的词句（会默认去掉包括禁用词的整句）。
|
||||
执行 `./make_dataset/csv_to_json.py` 脚本对数据进行处理。
|
||||
|
||||
在同一人连续回答多句的情况下，有三种处理方式：
|
||||
| 文件 | 处理方式 |
|
||||
| --- | --- |
|
||||
| csv_to_json.py | 用逗号连接 |
|
||||
| csv_to_json-单句回答.py(已废弃) | 只选择最长的回答作为最终数据 |
|
||||
| csv_to_json-单句多轮.py | 放在了提示词的'history'中 |
|
||||
|
||||
### 模型下载
|
||||
|
||||
首选在Hugging Face下载[ChatGLM3](https://huggingface.co/THUDM/chatglm3-6b) 模型。如果您在 Hugging Face 模型的下载中遇到了问题，可以通过下述方法使用魔搭社区，后续训练推理都需要先执行`export USE_MODELSCOPE_HUB=1`来使用魔搭社区的模型。
|
||||
由于模型较大，下载过程比较漫长请耐心等待。
|
||||
|
||||
```bash
|
||||
export USE_MODELSCOPE_HUB=1 # Windows 使用 `set USE_MODELSCOPE_HUB=1`
|
||||
git lfs install
|
||||
git clone https://www.modelscope.cn/ZhipuAI/chatglm3-6b.git
|
||||
```
|
||||
魔搭社区的`modeling_chatglm.py`文件需要更换为Hugging Face的
|
||||
|
||||
### 配置参数并微调模型
|
||||
|
||||
- (可选)修改 [settings.json](settings.json)选择本地下载好的其他模型。
|
||||
|
||||
- 修改`per_device_train_batch_size`以及`gradient_accumulation_steps`来调整显存占用。
|
||||
- 可以根据自己数据集的数量和质量修改`num_train_epochs`、`lora_rank`、`lora_dropout`等参数。
|
||||
|
||||
#### 单卡训练
|
||||
|
||||
运行 `src/train_sft.py` 进行sft阶段微调，本人loss只降到了3.5左右，降低过多可能会过拟合，我使用了大概2万条整合后的有效数据。
|
||||
|
||||
```bash
|
||||
python src/train_sft.py
|
||||
```
|
||||
|
||||
#### 多卡训练
|
||||
|
||||
```bash
|
||||
uv pip install deepspeed
|
||||
deepspeed --num_gpus=使用显卡数量 src/train_sft.py
|
||||
```
|
||||
|
||||
|
||||
### 使用浏览器demo简单推理
|
||||
|
||||
```bash
|
||||
python ./src/web_demo.py
|
||||
```
|
||||
|
||||
### 使用接口进行推理
|
||||
|
||||
```bash
|
||||
python ./src/api_service.py
|
||||
```
|
||||
|
||||
### 使用常见聊天问题测试
|
||||
|
||||
```bash
|
||||
python ./src/api_service.py
|
||||
python ./src/test_model.py
|
||||
```
|
||||
|
||||
### 部署到聊天机器人
|
||||
|
||||
#### AstrBot方案
|
||||
[AstrBot](https://github.com/AstrBotDevs/AstrBot) 是易上手的多平台 LLM 聊天机器人及开发框架 ✨ 平台支持 QQ、QQ频道、Telegram、微信、企微、飞书。
|
||||
|
||||
使用步骤：
|
||||
1. 部署 AstrBot
|
||||
2. 在 AstrBot 中部署消息平台
|
||||
3. 执行 `python ./src/api_service.py ` 启动api服务
|
||||
4. 在 AstrBot 中新增服务提供商，类型选择OpenAI，API Base URL 根据AstrBot部署方式填写（例如docker部署可能为http://172.17.0.1:8005/v1） ，模型填写gpt-3.5-turbo
|
||||
5. 微调后不支持工具调用，请先关掉默认的工具，消息平台发送指令： `/tool off reminder`，否则会没有微调后的效果。
|
||||
6. 根据微调时使用的default_system，在 AstrBot 中设置系统提示词。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<details>
|
||||
<summary>itchat方案（已弃用）</summary>
|
||||
|
||||
> [!IMPORTANT]
|
||||
> 微信有封号风险，建议使用小号，并且必须绑定银行卡才能使用
|
||||
|
||||
```bash
|
||||
python ./src/api_service.py # 先启动api服务
|
||||
python ./src/wechat_bot/main.py
|
||||
```
|
||||
|
||||
默认在终端显示二维码，扫码登录即可。可以私聊或者在群聊中@机器人使用。
|
||||
</details>
|
||||
|
||||
### 截图
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
# 免责声明
|
||||
> [!CAUTION]
|
||||
> 请勿用于非法用途，否则后果自负。
|
||||
<details>
|
||||
<summary>1. 使用目的</summary>
|
||||
|
||||
* 本项目仅供学习交流使用，**请勿用于非法用途**，**请勿用于非法用途**，**请勿用于非法用途**，否则后果自负。
|
||||
* 用户理解并同意，任何违反法律法规、侵犯他人合法权益的行为，均与本项目及其开发者无关，后果由用户自行承担。
|
||||
|
||||
2. 使用期限
|
||||
|
||||
* 您应该在下载保存使用本项目的24小时内，删除本项目的源代码和程序；超出此期限的任何使用行为，一概与本项目及其开发者无关。
|
||||
|
||||
3. 操作规范
|
||||
|
||||
* 本项目仅允许在授权情况下使用数据训练，严禁用于非法目的，否则自行承担所有相关责任；用户如因违反此规定而引发的任何法律责任，将由用户自行承担，与本项目及其开发者无关。
|
||||
* 严禁用于窃取他人隐私，严禁用于窃取他人隐私，严禁用于窃取他人隐私，否则自行承担所有相关责任。
|
||||
|
||||
4. 免责声明接受
|
||||
|
||||
* 下载、保存、进一步浏览源代码或者下载安装、编译使用本程序，表示你同意本警告，并承诺遵守它;
|
||||
|
||||
5. 禁止用于非法测试或渗透
|
||||
|
||||
* 禁止利用本项目的相关技术从事非法测试或渗透，禁止利用本项目的相关代码或相关技术从事任何非法工作，如因此产生的一切不良后果与本项目及其开发者无关。
|
||||
* 任何因此产生的不良后果，包括但不限于数据泄露、系统瘫痪、侵犯隐私等，均与本项目及其开发者无关，责任由用户自行承担。
|
||||
|
||||
6. 免责声明修改
|
||||
|
||||
* 本免责声明可能根据项目运行情况和法律法规的变化进行修改和调整。用户应定期查阅本页面以获取最新版本的免责声明，使用本项目时应遵守最新版本的免责声明。
|
||||
|
||||
7. 其他
|
||||
|
||||
* 除本免责声明规定外，用户在使用本项目过程中应遵守相关的法律法规和道德规范。对于因用户违反相关规定而引发的任何纠纷或损失，本项目及其开发者不承担任何责任。
|
||||
|
||||
* 请用户慎重阅读并理解本免责声明的所有内容，确保在使用本项目时严格遵守相关规定。
|
||||
|
||||
</details>
|
||||
请用户慎重阅读并理解本免责声明的所有内容，确保在使用本项目时严格遵守相关规定。
|
||||
|
||||
<br>
|
||||
<br>
|
||||
<br>
|
||||
|
||||
<div align="center"> 克隆我们，保留那灵魂的芬芳 </div>
|
||||
|
||||
|
||||
## 核心功能✨
|
||||
- 💬 使用微信聊天记录微调LLM
|
||||
- 🎙️ 使用微信语音消息➕0.5B大模型实现高质量声音克隆 👉[WeClone-audio](https://github.com/xming521/WeClone/tree/master/WeClone-audio)
|
||||
- 🔗 绑定到微信、QQ、Telegram、企微、飞书机器人，实现自己的数字分身
|
||||
|
||||
## 特性与说明📋
|
||||
|
||||
> [!TIP]
|
||||
> 新特性：[WeClone-audio](https://github.com/xming521/WeClone/tree/master/WeClone-audio) 模块，支持对微信语音进行克隆。
|
||||
|
||||
|
||||
> [!IMPORTANT]
|
||||
> 微调LLM最终效果很大程度取决于聊天数据的数量和质量
|
||||
|
||||
### 硬件要求
|
||||
|
||||
目前项目默认使用chatglm3-6b模型，LoRA方法对sft阶段微调，大约需要16GB显存。也可以使用[LLaMA Factory](https://github.com/hiyouga/LLaMA-Factory/blob/main/README_zh.md#%E6%A8%A1%E5%9E%8B)支持的其他模型和方法，占用显存更少，需要自行修改模板的system提示词等相关配置。
|
||||
|
||||
需要显存的估算值：
|
||||
| 方法 | 精度 | 7B | 14B | 30B | 70B | `x`B |
|
||||
| ------------------------------- | ---- | ----- | ----- | ----- | ------ | ------- |
|
||||
| Full (`bf16` or `fp16`) | 32 | 120GB | 240GB | 600GB | 1200GB | `18x`GB |
|
||||
| Full (`pure_bf16`) | 16 | 60GB | 120GB | 300GB | 600GB | `8x`GB |
|
||||
| Freeze/LoRA/GaLore/APOLLO/BAdam | 16 | 16GB | 32GB | 64GB | 160GB | `2x`GB |
|
||||
| QLoRA | 8 | 10GB | 20GB | 40GB | 80GB | `x`GB |
|
||||
| QLoRA | 4 | 6GB | 12GB | 24GB | 48GB | `x/2`GB |
|
||||
| QLoRA | 2 | 4GB | 8GB | 16GB | 24GB | `x/4`GB |
|
||||
|
||||
|
||||
### 环境搭建
|
||||
建议使用 [uv](https://docs.astral.sh/uv/)，这是一个非常快速的 Python 环境管理器。安装uv后，您可以使用以下命令创建一个新的Python环境并安装依赖项，注意这不包含xcodec（音频克隆）功能的依赖：
|
||||
```bash
|
||||
git clone https://github.com/xming521/WeClone.git
|
||||
cd WeClone
|
||||
uv venv .venv --python=3.9
|
||||
source .venv/bin/activate
|
||||
uv pip install --group main -e .
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> 训练以及推理相关配置统一在文件[settings.json](settings.json)
|
||||
|
||||
|
||||
### 数据准备
|
||||
|
||||
请使用[PyWxDump](https://github.com/xaoyaoo/PyWxDump)提取微信聊天记录。下载软件并解密数据库后，点击聊天备份，导出类型为CSV，可以导出多个联系人或群聊，然后将导出的位于`wxdump_tmp/export` 的 `csv` 文件夹放在`./data`目录即可，也就是不同人聊天记录的文件夹一起放在 `./data/csv`。 示例数据位于[data/example_chat.csv](data/example_chat.csv)。
|
||||
|
||||
### 数据预处理
|
||||
|
||||
项目默认去除了数据中的手机号、身份证号、邮箱、网址。还提供了一个禁用词词库[blocked_words](make_dataset/blocked_words.json)，可以自行添加需要过滤的词句（会默认去掉包括禁用词的整句）。
|
||||
执行 `./make_dataset/csv_to_json.py` 脚本对数据进行处理。
|
||||
|
||||
在同一人连续回答多句的情况下，有三种处理方式：
|
||||
| 文件 | 处理方式 |
|
||||
| --- | --- |
|
||||
| csv_to_json.py | 用逗号连接 |
|
||||
| csv_to_json-单句回答.py(已废弃) | 只选择最长的回答作为最终数据 |
|
||||
| csv_to_json-单句多轮.py | 放在了提示词的'history'中 |
|
||||
|
||||
### 模型下载
|
||||
|
||||
首选在Hugging Face下载[ChatGLM3](https://huggingface.co/THUDM/chatglm3-6b) 模型。如果您在 Hugging Face 模型的下载中遇到了问题，可以通过下述方法使用魔搭社区，后续训练推理都需要先执行`export USE_MODELSCOPE_HUB=1`来使用魔搭社区的模型。
|
||||
由于模型较大，下载过程比较漫长请耐心等待。
|
||||
|
||||
```bash
|
||||
export USE_MODELSCOPE_HUB=1 # Windows 使用 `set USE_MODELSCOPE_HUB=1`
|
||||
git lfs install
|
||||
git clone https://www.modelscope.cn/ZhipuAI/chatglm3-6b.git
|
||||
```
|
||||
魔搭社区的`modeling_chatglm.py`文件需要更换为Hugging Face的
|
||||
|
||||
### 配置参数并微调模型
|
||||
|
||||
- (可选)修改 [settings.json](settings.json)选择本地下载好的其他模型。
|
||||
|
||||
- 修改`per_device_train_batch_size`以及`gradient_accumulation_steps`来调整显存占用。
|
||||
- 可以根据自己数据集的数量和质量修改`num_train_epochs`、`lora_rank`、`lora_dropout`等参数。
|
||||
|
||||
#### 单卡训练
|
||||
|
||||
运行 `src/train_sft.py` 进行sft阶段微调，本人loss只降到了3.5左右，降低过多可能会过拟合，我使用了大概2万条整合后的有效数据。
|
||||
|
||||
```bash
|
||||
python src/train_sft.py
|
||||
```
|
||||
|
||||
#### 多卡训练
|
||||
|
||||
```bash
|
||||
uv pip install deepspeed
|
||||
deepspeed --num_gpus=使用显卡数量 src/train_sft.py
|
||||
```
|
||||
|
||||
|
||||
### 使用浏览器demo简单推理
|
||||
|
||||
```bash
|
||||
python ./src/web_demo.py
|
||||
```
|
||||
|
||||
### 使用接口进行推理
|
||||
|
||||
```bash
|
||||
python ./src/api_service.py
|
||||
```
|
||||
|
||||
### 使用常见聊天问题测试
|
||||
|
||||
```bash
|
||||
python ./src/api_service.py
|
||||
python ./src/test_model.py
|
||||
```
|
||||
|
||||
### 部署到聊天机器人
|
||||
|
||||
#### AstrBot方案
|
||||
[AstrBot](https://github.com/AstrBotDevs/AstrBot) 是易上手的多平台 LLM 聊天机器人及开发框架 ✨ 平台支持 QQ、QQ频道、Telegram、微信、企微、飞书。
|
||||
|
||||
使用步骤：
|
||||
1. 部署 AstrBot
|
||||
2. 在 AstrBot 中部署消息平台
|
||||
3. 执行 `python ./src/api_service.py ` 启动api服务
|
||||
4. 在 AstrBot 中新增服务提供商，类型选择OpenAI，API Base URL 根据AstrBot部署方式填写（例如docker部署可能为http://172.17.0.1:8005/v1） ，模型填写gpt-3.5-turbo
|
||||
5. 微调后不支持工具调用，请先关掉默认的工具，消息平台发送指令： `/tool off reminder`，否则会没有微调后的效果。
|
||||
6. 根据微调时使用的default_system，在 AstrBot 中设置系统提示词。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<details>
|
||||
<summary>itchat方案（已弃用）</summary>
|
||||
|
||||
> [!IMPORTANT]
|
||||
> 微信有封号风险，建议使用小号，并且必须绑定银行卡才能使用
|
||||
|
||||
```bash
|
||||
python ./src/api_service.py # 先启动api服务
|
||||
python ./src/wechat_bot/main.py
|
||||
```
|
||||
|
||||
默认在终端显示二维码，扫码登录即可。可以私聊或者在群聊中@机器人使用。
|
||||
</details>
|
||||
|
||||
### 截图
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
# 免责声明
|
||||
> [!CAUTION]
|
||||
> 请勿用于非法用途，否则后果自负。
|
||||
<details>
|
||||
<summary>1. 使用目的</summary>
|
||||
|
||||
* 本项目仅供学习交流使用，**请勿用于非法用途**，**请勿用于非法用途**，**请勿用于非法用途**，否则后果自负。
|
||||
* 用户理解并同意，任何违反法律法规、侵犯他人合法权益的行为，均与本项目及其开发者无关，后果由用户自行承担。
|
||||
|
||||
2. 使用期限
|
||||
|
||||
* 您应该在下载保存使用本项目的24小时内，删除本项目的源代码和程序；超出此期限的任何使用行为，一概与本项目及其开发者无关。
|
||||
|
||||
3. 操作规范
|
||||
|
||||
* 本项目仅允许在授权情况下使用数据训练，严禁用于非法目的，否则自行承担所有相关责任；用户如因违反此规定而引发的任何法律责任，将由用户自行承担，与本项目及其开发者无关。
|
||||
* 严禁用于窃取他人隐私，严禁用于窃取他人隐私，严禁用于窃取他人隐私，否则自行承担所有相关责任。
|
||||
|
||||
4. 免责声明接受
|
||||
|
||||
* 下载、保存、进一步浏览源代码或者下载安装、编译使用本程序，表示你同意本警告，并承诺遵守它;
|
||||
|
||||
5. 禁止用于非法测试或渗透
|
||||
|
||||
* 禁止利用本项目的相关技术从事非法测试或渗透，禁止利用本项目的相关代码或相关技术从事任何非法工作，如因此产生的一切不良后果与本项目及其开发者无关。
|
||||
* 任何因此产生的不良后果，包括但不限于数据泄露、系统瘫痪、侵犯隐私等，均与本项目及其开发者无关，责任由用户自行承担。
|
||||
|
||||
6. 免责声明修改
|
||||
|
||||
* 本免责声明可能根据项目运行情况和法律法规的变化进行修改和调整。用户应定期查阅本页面以获取最新版本的免责声明，使用本项目时应遵守最新版本的免责声明。
|
||||
|
||||
7. 其他
|
||||
|
||||
* 除本免责声明规定外，用户在使用本项目过程中应遵守相关的法律法规和道德规范。对于因用户违反相关规定而引发的任何纠纷或损失，本项目及其开发者不承担任何责任。
|
||||
|
||||
* 请用户慎重阅读并理解本免责声明的所有内容，确保在使用本项目时严格遵守相关规定。
|
||||
|
||||
</details>
|
||||
请用户慎重阅读并理解本免责声明的所有内容，确保在使用本项目时严格遵守相关规定。
|
||||
|
||||
<br>
|
||||
<br>
|
||||
<br>
|
||||
|
||||
## ⭐ Star History
|
||||
> [!TIP]
|
||||
> 如果本项目对您有帮助，或者您关注本项目的未来发展，请给项目 Star，谢谢
|
||||
|
||||
<div align="center">
|
||||
|
||||
[](https://www.star-history.com/#xming521/WeClone&Date)
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<div align="center"> 克隆我们，保留那灵魂的芬芳 </div>
|
||||
|
||||
@@ -26,7 +26,7 @@ WeClone Audio使用uv作为包管理器。
|
||||
# 为 PyWxDump 创建 Python 环境和安装依赖
|
||||
#
|
||||
uv venv .venv-wx --python=3.9
|
||||
source .venv-wx/bin/activate
|
||||
.venv-wx\Scripts\activate
|
||||
# 安装 wx 依赖组
|
||||
uv pip install --group wx -e .
|
||||
```
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
API_KEY=your_api_key_here
|
||||
PORT=5050
|
||||
|
||||
DEFAULT_VOICE=en-US-AvaNeural
|
||||
DEFAULT_RESPONSE_FORMAT=mp3
|
||||
DEFAULT_SPEED=1.0
|
||||
|
||||
DEFAULT_LANGUAGE=en-US
|
||||
|
||||
REQUIRE_API_KEY=True
|
||||
|
||||
REMOVE_FILTER=False
|
||||
|
||||
EXPAND_API=True
|
||||
@@ -0,0 +1,62 @@
|
||||
import re
|
||||
import emoji
|
||||
|
||||
def prepare_tts_input_with_context(text: str) -> str:
|
||||
"""
|
||||
Prepares text for a TTS API by cleaning Markdown and adding minimal contextual hints
|
||||
for certain Markdown elements like headers. Preserves paragraph separation.
|
||||
|
||||
Args:
|
||||
text (str): The raw text containing Markdown or other formatting.
|
||||
|
||||
Returns:
|
||||
str: Cleaned text with contextual hints suitable for TTS input.
|
||||
"""
|
||||
|
||||
# Remove emojis
|
||||
text = emoji.replace_emoji(text, replace='')
|
||||
|
||||
# Add context for headers
|
||||
def header_replacer(match):
|
||||
level = len(match.group(1)) # Number of '#' symbols
|
||||
header_text = match.group(2).strip()
|
||||
if level == 1:
|
||||
return f"Title — {header_text}\n"
|
||||
elif level == 2:
|
||||
return f"Section — {header_text}\n"
|
||||
else:
|
||||
return f"Subsection — {header_text}\n"
|
||||
|
||||
text = re.sub(r"^(#{1,6})\s+(.*)", header_replacer, text, flags=re.MULTILINE)
|
||||
|
||||
# Announce links (currently commented out for potential future use)
|
||||
# text = re.sub(r"\[([^\]]+)\]\((https?:\/\/[^\)]+)\)", r"\1 (link: \2)", text)
|
||||
|
||||
# Remove links while keeping the link text
|
||||
text = re.sub(r"\[([^\]]+)\]\([^\)]+\)", r"\1", text)
|
||||
|
||||
# Describe inline code
|
||||
text = re.sub(r"`([^`]+)`", r"code snippet: \1", text)
|
||||
|
||||
# Remove bold/italic symbols but keep the content
|
||||
text = re.sub(r"(\*\*|__|\*|_)", '', text)
|
||||
|
||||
# Remove code blocks (multi-line) with a description
|
||||
text = re.sub(r"```([\s\S]+?)```", r"(code block omitted)", text)
|
||||
|
||||
# Remove image syntax but add alt text if available
|
||||
text = re.sub(r"!\[([^\]]*)\]\([^\)]+\)", r"Image: \1", text)
|
||||
|
||||
# Remove HTML tags
|
||||
text = re.sub(r"</?[^>]+(>|$)", '', text)
|
||||
|
||||
# Normalize line breaks
|
||||
text = re.sub(r"\n{2,}", '\n\n', text) # Ensure consistent paragraph separation
|
||||
|
||||
# Replace multiple spaces within lines
|
||||
text = re.sub(r" {2,}", ' ', text)
|
||||
|
||||
# Trim leading and trailing whitespace from the whole text
|
||||
text = text.strip()
|
||||
|
||||
return text
|
||||
@@ -0,0 +1,5 @@
|
||||
flask
|
||||
gevent
|
||||
python-dotenv
|
||||
edge-tts
|
||||
emoji
|
||||
@@ -0,0 +1,167 @@
|
||||
# server.py
|
||||
|
||||
from flask import Flask, request, send_file, jsonify
|
||||
from gevent.pywsgi import WSGIServer
|
||||
from dotenv import load_dotenv
|
||||
import os
|
||||
|
||||
from handle_text import prepare_tts_input_with_context
|
||||
from tts_handler import generate_speech, get_models, get_voices
|
||||
from utils import getenv_bool, require_api_key, AUDIO_FORMAT_MIME_TYPES
|
||||
|
||||
app = Flask(__name__)
|
||||
load_dotenv()
|
||||
|
||||
API_KEY = os.getenv('API_KEY', 'your_api_key_here')
|
||||
PORT = int(os.getenv('PORT', 5050))
|
||||
|
||||
DEFAULT_VOICE = os.getenv('DEFAULT_VOICE', 'en-US-AvaNeural')
|
||||
DEFAULT_RESPONSE_FORMAT = os.getenv('DEFAULT_RESPONSE_FORMAT', 'mp3')
|
||||
DEFAULT_SPEED = float(os.getenv('DEFAULT_SPEED', 1.0))
|
||||
|
||||
REMOVE_FILTER = getenv_bool('REMOVE_FILTER', False)
|
||||
EXPAND_API = getenv_bool('EXPAND_API', True)
|
||||
|
||||
# DEFAULT_MODEL = os.getenv('DEFAULT_MODEL', 'tts-1')
|
||||
|
||||
@app.route('/v1/audio/speech', methods=['POST'])
|
||||
@app.route('/audio/speech', methods=['POST']) # Add this line for the alias
|
||||
@require_api_key
|
||||
def text_to_speech():
|
||||
data = request.json
|
||||
if not data or 'input' not in data:
|
||||
return jsonify({"error": "Missing 'input' in request body"}), 400
|
||||
|
||||
text = data.get('input')
|
||||
|
||||
if not REMOVE_FILTER:
|
||||
text = prepare_tts_input_with_context(text)
|
||||
|
||||
# model = data.get('model', DEFAULT_MODEL)
|
||||
voice = data.get('voice', DEFAULT_VOICE)
|
||||
|
||||
response_format = data.get('response_format', DEFAULT_RESPONSE_FORMAT)
|
||||
speed = float(data.get('speed', DEFAULT_SPEED))
|
||||
|
||||
mime_type = AUDIO_FORMAT_MIME_TYPES.get(response_format, "audio/mpeg")
|
||||
|
||||
# Generate the audio file in the specified format with speed adjustment
|
||||
output_file_path = generate_speech(text, voice, response_format, speed)
|
||||
|
||||
# Return the file with the correct MIME type
|
||||
return send_file(output_file_path, mimetype=mime_type, as_attachment=True, download_name=f"speech.{response_format}")
|
||||
|
||||
@app.route('/v1/models', methods=['GET', 'POST'])
|
||||
@app.route('/models', methods=['GET', 'POST'])
|
||||
@require_api_key
|
||||
def list_models():
|
||||
return jsonify({"data": get_models()})
|
||||
|
||||
@app.route('/v1/voices', methods=['GET', 'POST'])
|
||||
@app.route('/voices', methods=['GET', 'POST'])
|
||||
@require_api_key
|
||||
def list_voices():
|
||||
specific_language = None
|
||||
|
||||
data = request.args if request.method == 'GET' else request.json
|
||||
if data and ('language' in data or 'locale' in data):
|
||||
specific_language = data.get('language') if 'language' in data else data.get('locale')
|
||||
|
||||
return jsonify({"voices": get_voices(specific_language)})
|
||||
|
||||
@app.route('/v1/voices/all', methods=['GET', 'POST'])
|
||||
@app.route('/voices/all', methods=['GET', 'POST'])
|
||||
@require_api_key
|
||||
def list_all_voices():
|
||||
return jsonify({"voices": get_voices('all')})
|
||||
|
||||
"""
|
||||
Support for ElevenLabs and Azure AI Speech
|
||||
(currently in beta)
|
||||
"""
|
||||
|
||||
# http://localhost:5050/elevenlabs/v1/text-to-speech
|
||||
# http://localhost:5050/elevenlabs/v1/text-to-speech/en-US-AndrewNeural
|
||||
@app.route('/elevenlabs/v1/text-to-speech/<voice_id>', methods=['POST'])
|
||||
@require_api_key
|
||||
def elevenlabs_tts(voice_id):
|
||||
if not EXPAND_API:
|
||||
return jsonify({"error": f"Endpoint not allowed"}), 500
|
||||
|
||||
# Parse the incoming JSON payload
|
||||
try:
|
||||
payload = request.json
|
||||
if not payload or 'text' not in payload:
|
||||
return jsonify({"error": "Missing 'text' in request body"}), 400
|
||||
except Exception as e:
|
||||
return jsonify({"error": f"Invalid JSON payload: {str(e)}"}), 400
|
||||
|
||||
text = payload['text']
|
||||
|
||||
if not REMOVE_FILTER:
|
||||
text = prepare_tts_input_with_context(text)
|
||||
|
||||
voice = voice_id # ElevenLabs uses the voice_id in the URL
|
||||
|
||||
# Use default settings for edge-tts
|
||||
response_format = 'mp3'
|
||||
speed = DEFAULT_SPEED # Optional customization via payload.get('speed', DEFAULT_SPEED)
|
||||
|
||||
# Generate speech using edge-tts
|
||||
try:
|
||||
output_file_path = generate_speech(text, voice, response_format, speed)
|
||||
except Exception as e:
|
||||
return jsonify({"error": f"TTS generation failed: {str(e)}"}), 500
|
||||
|
||||
# Return the generated audio file
|
||||
return send_file(output_file_path, mimetype="audio/mpeg", as_attachment=True, download_name="speech.mp3")
|
||||
|
||||
# tts.speech.microsoft.com/cognitiveservices/v1
|
||||
# https://{region}.tts.speech.microsoft.com/cognitiveservices/v1
|
||||
# http://localhost:5050/azure/cognitiveservices/v1
|
||||
@app.route('/azure/cognitiveservices/v1', methods=['POST'])
|
||||
@require_api_key
|
||||
def azure_tts():
|
||||
if not EXPAND_API:
|
||||
return jsonify({"error": f"Endpoint not allowed"}), 500
|
||||
|
||||
# Parse the SSML payload
|
||||
try:
|
||||
ssml_data = request.data.decode('utf-8')
|
||||
if not ssml_data:
|
||||
return jsonify({"error": "Missing SSML payload"}), 400
|
||||
|
||||
# Extract the text and voice from SSML
|
||||
from xml.etree import ElementTree as ET
|
||||
root = ET.fromstring(ssml_data)
|
||||
text = root.find('.//{http://www.w3.org/2001/10/synthesis}voice').text
|
||||
voice = root.find('.//{http://www.w3.org/2001/10/synthesis}voice').get('name')
|
||||
except Exception as e:
|
||||
return jsonify({"error": f"Invalid SSML payload: {str(e)}"}), 400
|
||||
|
||||
# Use default settings for edge-tts
|
||||
response_format = 'mp3'
|
||||
speed = DEFAULT_SPEED
|
||||
|
||||
if not REMOVE_FILTER:
|
||||
text = prepare_tts_input_with_context(text)
|
||||
|
||||
# Generate speech using edge-tts
|
||||
try:
|
||||
output_file_path = generate_speech(text, voice, response_format, speed)
|
||||
except Exception as e:
|
||||
return jsonify({"error": f"TTS generation failed: {str(e)}"}), 500
|
||||
|
||||
# Return the generated audio file
|
||||
return send_file(output_file_path, mimetype="audio/mpeg", as_attachment=True, download_name="speech.mp3")
|
||||
|
||||
print(f" Edge TTS (Free Azure TTS) Replacement for OpenAI's TTS API")
|
||||
print(f" ")
|
||||
print(f" * Serving OpenAI Edge TTS")
|
||||
print(f" * Server running on http://localhost:{PORT}")
|
||||
print(f" * TTS Endpoint: http://localhost:{PORT}/v1/audio/speech")
|
||||
print(f" ")
|
||||
|
||||
if __name__ == '__main__':
|
||||
http_server = WSGIServer(('0.0.0.0', PORT), app)
|
||||
http_server.serve_forever()
|
||||
@@ -0,0 +1,133 @@
|
||||
import edge_tts
|
||||
import asyncio
|
||||
import tempfile
|
||||
import subprocess
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# Language default (environment variable)
|
||||
DEFAULT_LANGUAGE = os.getenv('DEFAULT_LANGUAGE', 'en-US')
|
||||
|
||||
# OpenAI voice names mapped to edge-tts equivalents
|
||||
voice_mapping = {
|
||||
'alloy': 'en-US-AvaNeural',
|
||||
'echo': 'en-US-AndrewNeural',
|
||||
'fable': 'en-GB-SoniaNeural',
|
||||
'onyx': 'en-US-EricNeural',
|
||||
'nova': 'en-US-SteffanNeural',
|
||||
'shimmer': 'en-US-EmmaNeural'
|
||||
}
|
||||
|
||||
def is_ffmpeg_installed():
|
||||
"""Check if FFmpeg is installed and accessible."""
|
||||
try:
|
||||
subprocess.run(['ffmpeg', '-version'], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
return True
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
return False
|
||||
|
||||
async def _generate_audio(text, voice, response_format, speed):
|
||||
"""Generate TTS audio and optionally convert to a different format."""
|
||||
# Determine if the voice is an OpenAI-compatible voice or a direct edge-tts voice
|
||||
edge_tts_voice = voice_mapping.get(voice, voice) # Use mapping if in OpenAI names, otherwise use as-is
|
||||
|
||||
# Generate the TTS output in mp3 format first
|
||||
temp_output_file = tempfile.NamedTemporaryFile(delete=False, suffix=".mp3")
|
||||
|
||||
# Convert speed to SSML rate format
|
||||
try:
|
||||
speed_rate = speed_to_rate(speed) # Convert speed value to "+X%" or "-X%"
|
||||
except Exception as e:
|
||||
print(f"Error converting speed: {e}. Defaulting to +0%.")
|
||||
speed_rate = "+0%"
|
||||
|
||||
# Generate the MP3 file
|
||||
communicator = edge_tts.Communicate(text=text, voice=edge_tts_voice, rate=speed_rate)
|
||||
await communicator.save(temp_output_file.name)
|
||||
|
||||
# If the requested format is mp3, return the generated file directly
|
||||
if response_format == "mp3":
|
||||
return temp_output_file.name
|
||||
|
||||
# Check if FFmpeg is installed
|
||||
if not is_ffmpeg_installed():
|
||||
print("FFmpeg is not available. Returning unmodified mp3 file.")
|
||||
return temp_output_file.name
|
||||
|
||||
# Create a new temporary file for the converted output
|
||||
converted_output_file = tempfile.NamedTemporaryFile(delete=False, suffix=f".{response_format}")
|
||||
|
||||
# Build the FFmpeg command
|
||||
ffmpeg_command = [
|
||||
"ffmpeg",
|
||||
"-i", temp_output_file.name, # Input file
|
||||
"-c:a", {
|
||||
"aac": "aac",
|
||||
"mp3": "libmp3lame",
|
||||
"wav": "pcm_s16le",
|
||||
"opus": "libopus",
|
||||
"flac": "flac"
|
||||
}.get(response_format, "aac"), # Default to AAC if unknown
|
||||
"-b:a", "192k" if response_format != "wav" else None, # Bitrate not needed for WAV
|
||||
"-f", {
|
||||
"aac": "mp4", # AAC in MP4 container
|
||||
"mp3": "mp3",
|
||||
"wav": "wav",
|
||||
"opus": "ogg",
|
||||
"flac": "flac"
|
||||
}.get(response_format, response_format), # Default to matching format
|
||||
"-y", # Overwrite without prompt
|
||||
converted_output_file.name # Output file
|
||||
]
|
||||
|
||||
try:
|
||||
# Run FFmpeg command and ensure no errors occur
|
||||
subprocess.run(ffmpeg_command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
except subprocess.CalledProcessError as e:
|
||||
raise RuntimeError(f"FFmpeg error during audio conversion: {e}")
|
||||
|
||||
# Clean up the original temporary file
|
||||
Path(temp_output_file.name).unlink(missing_ok=True)
|
||||
|
||||
return converted_output_file.name
|
||||
|
||||
def generate_speech(text, voice, response_format, speed=1.0):
|
||||
return asyncio.run(_generate_audio(text, voice, response_format, speed))
|
||||
|
||||
def get_models():
|
||||
return [
|
||||
{"id": "tts-1", "name": "Text-to-speech v1"},
|
||||
{"id": "tts-1-hd", "name": "Text-to-speech v1 HD"}
|
||||
]
|
||||
|
||||
async def _get_voices(language=None):
|
||||
# List all voices, filter by language if specified
|
||||
all_voices = await edge_tts.list_voices()
|
||||
language = language or DEFAULT_LANGUAGE # Use default if no language specified
|
||||
filtered_voices = [
|
||||
{"name": v['ShortName'], "gender": v['Gender'], "language": v['Locale']}
|
||||
for v in all_voices if language == 'all' or language is None or v['Locale'] == language
|
||||
]
|
||||
return filtered_voices
|
||||
|
||||
def get_voices(language=None):
|
||||
return asyncio.run(_get_voices(language))
|
||||
|
||||
def speed_to_rate(speed: float) -> str:
|
||||
"""
|
||||
Converts a multiplicative speed value to the edge-tts "rate" format.
|
||||
|
||||
Args:
|
||||
speed (float): The multiplicative speed value (e.g., 1.5 for +50%, 0.5 for -50%).
|
||||
|
||||
Returns:
|
||||
str: The formatted "rate" string (e.g., "+50%" or "-50%").
|
||||
"""
|
||||
if speed < 0 or speed > 2:
|
||||
raise ValueError("Speed must be between 0 and 2 (inclusive).")
|
||||
|
||||
# Convert speed to percentage change
|
||||
percentage_change = (speed - 1) * 100
|
||||
|
||||
# Format with a leading "+" or "-" as required
|
||||
return f"{percentage_change:+.0f}%"
|
||||
@@ -0,0 +1,38 @@
|
||||
# utils.py
|
||||
|
||||
from flask import request, jsonify
|
||||
from functools import wraps
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
def getenv_bool(name: str, default: bool = False) -> bool:
|
||||
return os.getenv(name, str(default)).lower() in ("yes", "y", "true", "1", "t")
|
||||
|
||||
API_KEY = os.getenv('API_KEY', 'your_api_key_here')
|
||||
REQUIRE_API_KEY = getenv_bool('REQUIRE_API_KEY', True)
|
||||
|
||||
def require_api_key(f):
|
||||
@wraps(f)
|
||||
def decorated_function(*args, **kwargs):
|
||||
if not REQUIRE_API_KEY:
|
||||
return f(*args, **kwargs)
|
||||
auth_header = request.headers.get('Authorization')
|
||||
if not auth_header or not auth_header.startswith('Bearer '):
|
||||
return jsonify({"error": "Missing or invalid API key"}), 401
|
||||
token = auth_header.split('Bearer ')[1]
|
||||
if token != API_KEY:
|
||||
return jsonify({"error": "Invalid API key"}), 401
|
||||
return f(*args, **kwargs)
|
||||
return decorated_function
|
||||
|
||||
# Mapping of audio format to MIME type
|
||||
AUDIO_FORMAT_MIME_TYPES = {
|
||||
"mp3": "audio/mpeg",
|
||||
"opus": "audio/ogg",
|
||||
"aac": "audio/aac",
|
||||
"flac": "audio/flac",
|
||||
"wav": "audio/wav",
|
||||
"pcm": "audio/L16"
|
||||
}
|
||||
@@ -1,131 +0,0 @@
|
||||
import os
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
import torch
|
||||
import soundfile as sf
|
||||
from xcodec2.modeling_xcodec2 import XCodec2Model
|
||||
import torchaudio
|
||||
|
||||
|
||||
class TextToSpeech:
|
||||
def __init__(self, sample_audio_path, sample_audio_text):
|
||||
self.sample_audio_text = sample_audio_text
|
||||
# 初始化模型
|
||||
llasa_3b = "HKUSTAudio/Llasa-3B"
|
||||
xcodec2 = "HKUSTAudio/xcodec2"
|
||||
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(llasa_3b)
|
||||
self.llasa_3b_model = AutoModelForCausalLM.from_pretrained(
|
||||
llasa_3b,
|
||||
trust_remote_code=True,
|
||||
device_map="auto",
|
||||
)
|
||||
self.llasa_3b_model.eval()
|
||||
|
||||
self.xcodec_model = XCodec2Model.from_pretrained(xcodec2)
|
||||
self.xcodec_model.eval().cuda()
|
||||
|
||||
# 处理音频
|
||||
waveform, sample_rate = torchaudio.load(sample_audio_path)
|
||||
if len(waveform[0]) / sample_rate > 15:
|
||||
print("已将音频裁剪至前15秒。")
|
||||
waveform = waveform[:, : sample_rate * 15]
|
||||
|
||||
# 检查音频是否为立体声
|
||||
if waveform.size(0) > 1:
|
||||
waveform_mono = torch.mean(waveform, dim=0, keepdim=True)
|
||||
else:
|
||||
waveform_mono = waveform
|
||||
|
||||
self.prompt_wav = torchaudio.transforms.Resample(
|
||||
orig_freq=sample_rate, new_freq=16000
|
||||
)(waveform_mono)
|
||||
|
||||
# Encode the prompt wav
|
||||
vq_code_prompt = self.xcodec_model.encode_code(input_waveform=self.prompt_wav)
|
||||
vq_code_prompt = vq_code_prompt[0, 0, :]
|
||||
self.speech_ids_prefix = self.ids_to_speech_tokens(vq_code_prompt)
|
||||
self.speech_end_id = self.tokenizer.convert_tokens_to_ids("<|SPEECH_GENERATION_END|>")
|
||||
|
||||
def ids_to_speech_tokens(self, speech_ids):
|
||||
speech_tokens_str = []
|
||||
for speech_id in speech_ids:
|
||||
speech_tokens_str.append(f"<|s_{speech_id}|>")
|
||||
return speech_tokens_str
|
||||
|
||||
def extract_speech_ids(self, speech_tokens_str):
|
||||
speech_ids = []
|
||||
for token_str in speech_tokens_str:
|
||||
if token_str.startswith("<|s_") and token_str.endswith("|>"):
|
||||
num_str = token_str[4:-2]
|
||||
num = int(num_str)
|
||||
speech_ids.append(num)
|
||||
else:
|
||||
print(f"Unexpected token: {token_str}")
|
||||
return speech_ids
|
||||
|
||||
@torch.inference_mode()
|
||||
def infer(self, target_text):
|
||||
if len(target_text) == 0:
|
||||
return None
|
||||
elif len(target_text) > 300:
|
||||
print("文本过长，请保持在300字符以内。")
|
||||
target_text = target_text[:300]
|
||||
|
||||
input_text = self.sample_audio_text + " " + target_text
|
||||
|
||||
formatted_text = (
|
||||
f"<|TEXT_UNDERSTANDING_START|>{input_text}<|TEXT_UNDERSTANDING_END|>"
|
||||
)
|
||||
|
||||
chat = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Convert the text to speech:" + formatted_text,
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "<|SPEECH_GENERATION_START|>"
|
||||
+ "".join(self.speech_ids_prefix),
|
||||
},
|
||||
]
|
||||
|
||||
input_ids = self.tokenizer.apply_chat_template(
|
||||
chat, tokenize=True, return_tensors="pt", continue_final_message=True
|
||||
)
|
||||
input_ids = input_ids.to("cuda")
|
||||
|
||||
outputs = self.llasa_3b_model.generate(
|
||||
input_ids,
|
||||
max_length=2048,
|
||||
eos_token_id=self.speech_end_id,
|
||||
do_sample=True,
|
||||
top_p=1,
|
||||
temperature=0.8,
|
||||
)
|
||||
generated_ids = outputs[0][input_ids.shape[1] - len(self.speech_ids_prefix): -1]
|
||||
|
||||
speech_tokens = self.tokenizer.batch_decode(
|
||||
generated_ids, skip_special_tokens=True
|
||||
)
|
||||
|
||||
speech_tokens = self.extract_speech_ids(speech_tokens)
|
||||
speech_tokens = torch.tensor(speech_tokens).cuda().unsqueeze(0).unsqueeze(0)
|
||||
|
||||
gen_wav = self.xcodec_model.decode_code(speech_tokens)
|
||||
gen_wav = gen_wav[:, :, self.prompt_wav.shape[1]:]
|
||||
|
||||
return (16000, gen_wav[0, 0, :].cpu().numpy())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 如果遇到问题，请尝试将参考音频转换为WAV或MP3格式，将其裁剪至15秒以内，并缩短提示文本。
|
||||
sample_audio_text = "对，这就是我万人敬仰的太乙真人，虽然有点婴儿肥，但也掩不住我逼人的帅气。"
|
||||
sample_audio_path = os.path.join(os.path.dirname(__file__), "sample.wav")
|
||||
|
||||
tts = TextToSpeech(sample_audio_path, sample_audio_text)
|
||||
target_text = "晚上好啊，吃了吗您"
|
||||
result = tts.infer(target_text)
|
||||
sf.write(os.path.join(os.path.dirname(__file__), "output.wav"), result[1], result[0])
|
||||
target_text = "我是老北京正黄旗！"
|
||||
result = tts.infer(target_text)
|
||||
sf.write(os.path.join(os.path.dirname(__file__), "output1.wav"), result[1], result[0])
|
||||
Reference in New Issue
Block a user