实现terminalcode和文件交互
This commit is contained in:
minorQAQ committed
1 parent
287e1935f6
commit
a5e5aff63f
141 files changed
+2385
-1826
No files matched your search
+179
-24
@@ -1,35 +1,190 @@
|
||||
This bundle is self-contained for running `Qwen/Qwen3.6-35B-A3B-FP8` with
|
||||
`local/vllm-qwen36-nvfp4:v0.19.1-cu130` on another Spark machine.
|
||||
# LLM Server 部署指南
|
||||
|
||||
Contents:
|
||||
- `image/`: Docker image tarball
|
||||
- `modelscope/`: bundled ModelScope cache for the model weights
|
||||
- `chat_template.jinja`: chat template mounted into the container
|
||||
- `start_vllm.sh`: loads the image if needed and starts the container
|
||||
- `stop_vllm.sh`: stops the container
|
||||
本目录包含 4 个模型的推理服务：**ASR（语音识别）**、**TTS（语音合成）**、**Reranker（重排序）**、**Embedding（文本嵌入）**。
|
||||
|
||||
Usage after extraction:
|
||||
## 1. 上传到远程服务器
|
||||
|
||||
```bash
|
||||
cd vllm-qwen36-bundle
|
||||
chmod +x start_vllm.sh stop_vllm.sh
|
||||
./start_vllm.sh
|
||||
scp -P 60007 -r C:\Users\86166\Desktop\Agent_Learning_minor\Agent\llm_server\* minor123@semi-ai.tech:~/llm_server/
|
||||
```
|
||||
|
||||
Optional overrides:
|
||||
- `HOST_PORT=8001 ./start_vllm.sh`
|
||||
- `CONTAINER_NAME=my-vllm ./start_vllm.sh`
|
||||
## 2. 远程服务器环境准备
|
||||
|
||||
The startup script recreates the `Qwen/Qwen3.6-35B-A3B-FP8` symlink inside the
|
||||
bundled ModelScope cache so the bundle remains portable across machines.
|
||||
```bash
|
||||
ssh minor123@6000d
|
||||
cd ~/llm_server
|
||||
|
||||
# 给所有脚本添加执行权限
|
||||
chmod +x *.sh
|
||||
|
||||
*************audio*************
|
||||
# 前台运行，Ctrl+C 直接停止
|
||||
./start_audio.sh
|
||||
# 配置 HuggingFace 镜像加速（国内服务器必需）
|
||||
export HF_ENDPOINT=https://hf-mirror.com
|
||||
export HF_HUB_ENABLE_XET=0
|
||||
echo 'export HF_ENDPOINT=https://hf-mirror.com' >> ~/.bashrc
|
||||
echo 'export HF_HUB_ENABLE_XET=0' >> ~/.bashrc
|
||||
|
||||
# 后台运行
|
||||
nohup ./start_audio.sh > /dev/null 2>&1 &
|
||||
# 安装 modelscope CLI（用于下载模型）
|
||||
pip install modelscope
|
||||
|
||||
# 后台运行时用 stop 停止
|
||||
./stop_audio.sh
|
||||
# 创建虚拟环境（如尚未创建，需安装对应依赖）
|
||||
python3 -m venv myenv
|
||||
source myenv/bin/activate
|
||||
pip install vllm modelscope transformers fastapi uvicorn torch soundfile bitsandbytes
|
||||
pip install genie-tts "huggingface_hub>=0.25,<0.30"
|
||||
```
|
||||
|
||||
## 3. 模型下载
|
||||
|
||||
所有模型统一存放在 `~/llm_server/models/` 目录下。
|
||||
|
||||
### 3.1 安装 modelscope CLI
|
||||
|
||||
```bash
|
||||
pip install modelscope -U
|
||||
```
|
||||
|
||||
### 3.2 Qwen3-ASR-1.7B（语音识别）
|
||||
|
||||
```bash
|
||||
cd ~/llm_server
|
||||
|
||||
# 用于 start_asr.sh (vLLM 加载)
|
||||
modelscope download --model Qwen/Qwen3-ASR-1.7B --local_dir ./models/Qwen/Qwen3-ASR-1.7B
|
||||
```
|
||||
|
||||
> 两个目录内容相同，只是命名不同以适配不同加载方式。
|
||||
|
||||
### 3.3 Qwen3-Embedding-0.6B（文本嵌入）
|
||||
|
||||
```bash
|
||||
cd ~/llm_server
|
||||
modelscope download --model Qwen/Qwen3-Embedding-0.6B --local_dir ./models/Qwen/Qwen3-Embedding-0___6B
|
||||
```
|
||||
|
||||
### 3.4 Qwen3-Reranker-4B（重排序）
|
||||
|
||||
```bash
|
||||
cd ~/llm_server
|
||||
modelscope download --model Qwen/Qwen3-Reranker-4B --local_dir ./models/Qwen/Qwen3-Reranker-4B
|
||||
```
|
||||
|
||||
### 3.5 Genie TTS（语音合成，端口 8902）
|
||||
|
||||
GENIE 是一个基于开源 TTS 项目 **GPT-SoVITS** 构建的轻量级推理引擎，支持中文/日语/英语/韩语，提供极致的 CPU/GPU 推理性能。
|
||||
|
||||
#### Step 1：安装 genie-tts
|
||||
|
||||
```bash
|
||||
pip install genie-tts
|
||||
```
|
||||
|
||||
#### Step 2：下载基础资源文件 GenieData（~391MB）
|
||||
|
||||
```bash
|
||||
git clone https://huggingface.co/High-Logic/Genie
|
||||
```
|
||||
|
||||
**方式三：从本地上传**
|
||||
|
||||
如果已在本地下载好完整 Genie 目录（如 `Agent/Genie/`），直接 scp：
|
||||
|
||||
```bash
|
||||
scp -P 60007 -r C:\Users\86166\Desktop\Agent_Learning_minor\Agent\Genie\* minor123@semi-ai.tech:~/llm_server/models/genie_tts/
|
||||
```
|
||||
|
||||
> 注意：环境变量 `GENIE_DATA_DIR` 已在 `tts_server.py` 中自动指向 `models/genie_tts/GenieData`，无需额外设置。
|
||||
|
||||
#### Step 3：验证文件完整性
|
||||
|
||||
完成后检查以下路径均存在：
|
||||
|
||||
| 文件/目录 | 说明 |
|
||||
|-----------|------|
|
||||
| `models/genie_tts/GenieData/` | 基础资源（词典、声码器等） |
|
||||
| `models/genie_tts/CharacterModels/v2ProPlus/feibi/tts_models/` | feibi 角色 ONNX 模型 |
|
||||
| `models/genie_tts/CharacterModels/v2ProPlus/feibi/prompt_wav/zh_vo_Main_Linaxita_2_1_10_26.wav` | feibi 参考音频 |
|
||||
|
||||
> 如需使用其他预定义角色（mika、ThirtySeven 等），可浏览 [HuggingFace CharacterModels](https://huggingface.co/High-Logic/Genie/tree/main/CharacterModels) 并按相同方式下载。
|
||||
|
||||
### 3.7 完成后的目录结构
|
||||
|
||||
```
|
||||
~/llm_server/
|
||||
├── models/
|
||||
│ ├── Qwen/
|
||||
│ │ ├── Qwen3-ASR-1.7B/ # ASR（vLLM）
|
||||
│ │ ├── Qwen3-Embedding-0___6B/ # Embedding
|
||||
│ │ └── Qwen3-Reranker-4B/ # Reranker
|
||||
│ └── genie_tts/
|
||||
│ ├── GenieData/
|
||||
│ └── CharacterModels/
|
||||
│ └── v2ProPlus/
|
||||
│ └── feibi/
|
||||
│ ├── tts_models/
|
||||
│ └── prompt_wav/
|
||||
│ └── zh_vo_Main_Linaxita_2_1_10_26.wav
|
||||
├── start_asr.sh # 启动 ASR 服务
|
||||
├── start_tts.sh # 启动 TTS 服务
|
||||
├── start_rag_server.sh # 启动 Reranker + Embedding 服务
|
||||
└── ...
|
||||
```
|
||||
|
||||
## 4. 启动各服务
|
||||
|
||||
### 4.1 ASR（语音识别）— 端口 8901
|
||||
|
||||
```bash
|
||||
cd ~/llm_server
|
||||
bash start_asr.sh
|
||||
```
|
||||
|
||||
测试：
|
||||
```bash
|
||||
python3 test_asr.py --audio /path/to/audio.wav --url http://127.0.0.1:8901/v1/chat/completions
|
||||
```
|
||||
|
||||
### 4.2 TTS（语音合成）— 端口 8902
|
||||
|
||||
```bash
|
||||
cd ~/llm_server
|
||||
bash start_tts.sh
|
||||
```
|
||||
|
||||
测试：
|
||||
```bash
|
||||
python3 test_tts.py
|
||||
```
|
||||
> 注意：测试脚本中 `TTS_BASE_URL` 默认为 `http://192.168.137.164:8902`，在远程服务器上需改为 `http://127.0.0.1:8902`。
|
||||
|
||||
### 4.3 Reranker + Embedding — 端口 8903
|
||||
|
||||
```bash
|
||||
cd ~/llm_server
|
||||
bash start_rag_server.sh
|
||||
```
|
||||
|
||||
测试：
|
||||
```bash
|
||||
python3 test_rag.py
|
||||
```
|
||||
|
||||
## 5. 服务端口汇总
|
||||
|
||||
| 服务 | 脚本 | 端口 |
|
||||
|------|------|------|
|
||||
| ASR | `start_asr.sh` | 8901 |
|
||||
| TTS | `start_tts.sh` | 8902 |
|
||||
| Reranker | `start_rag_server.sh` | 8903 |
|
||||
| Embedding | `start_rag_server.sh` | 8903 |
|
||||
|
||||
## 6. 停止服务
|
||||
|
||||
```bash
|
||||
kill $(lsof -t -i:8901) # 停止 ASR
|
||||
kill $(lsof -t -i:8902) # 停止 TTS
|
||||
kill $(lsof -t -i:8903) # 停止 Reranker/Embedding
|
||||
```
|
||||
|
||||
## 7. 路径说明
|
||||
|
||||
所有 Python 脚本中的模型路径已改为基于脚本所在目录的相对路径（通过 `os.path.dirname(os.path.abspath(__file__))` 动态获取），Shell 脚本使用 `${SCRIPT_DIR}` 定位。因此目录整体迁移到任意位置均可正常运行，无需再修改路径。
|
||||
Reference in new issue
Block a user