agent/auto_agent
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

实现terminalcode和文件交互

Browse Source
This commit is contained in:
minorQAQ committed 2026-07-08 17:24:51 +08:00
1 parent 287e1935f6
commit a5e5aff63f
141 files changed
+2385 -1826
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

llm_server/README.md
+179 -24
View File
@@ -1,35 +1,190 @@
This bundle is self-contained for running `Qwen/Qwen3.6-35B-A3B-FP8` with
`local/vllm-qwen36-nvfp4:v0.19.1-cu130` on another Spark machine.
# LLM Server 部署指南
Contents:
- `image/`: Docker image tarball
- `modelscope/`: bundled ModelScope cache for the model weights
- `chat_template.jinja`: chat template mounted into the container
- `start_vllm.sh`: loads the image if needed and starts the container
- `stop_vllm.sh`: stops the container
本目录包含 4 个模型的推理服务：**ASR（语音识别）**、**TTS（语音合成）**、**Reranker（重排序）**、**Embedding（文本嵌入）**。
Usage after extraction:
## 1. 上传到远程服务器
```bash
cd vllm-qwen36-bundle
chmod +x start_vllm.sh stop_vllm.sh
./start_vllm.sh
scp -P 60007 -r C:\Users\86166\Desktop\Agent_Learning_minor\Agent\llm_server\* minor123@semi-ai.tech:~/llm_server/
```
Optional overrides:
- `HOST_PORT=8001 ./start_vllm.sh`
- `CONTAINER_NAME=my-vllm ./start_vllm.sh`
## 2. 远程服务器环境准备
The startup script recreates the `Qwen/Qwen3.6-35B-A3B-FP8` symlink inside the
bundled ModelScope cache so the bundle remains portable across machines.
```bash
ssh minor123@6000d
cd ~/llm_server
# 给所有脚本添加执行权限
chmod +x *.sh
*************audio*************
# 前台运行，Ctrl+C 直接停止
./start_audio.sh
# 配置 HuggingFace 镜像加速（国内服务器必需）
export HF_ENDPOINT=https://hf-mirror.com
export HF_HUB_ENABLE_XET=0
echo 'export HF_ENDPOINT=https://hf-mirror.com' >> ~/.bashrc
echo 'export HF_HUB_ENABLE_XET=0' >> ~/.bashrc
# 后台运行
nohup ./start_audio.sh > /dev/null 2>&1 &
# 安装 modelscope CLI（用于下载模型）
pip install modelscope
# 后台运行时用 stop 停止
./stop_audio.sh
# 创建虚拟环境（如尚未创建，需安装对应依赖）
python3 -m venv myenv
source myenv/bin/activate
pip install vllm modelscope transformers fastapi uvicorn torch soundfile bitsandbytes
pip install genie-tts "huggingface_hub>=0.25,<0.30"
```
## 3. 模型下载
所有模型统一存放在 `~/llm_server/models/` 目录下。
### 3.1 安装 modelscope CLI
```bash
pip install modelscope -U
```
### 3.2 Qwen3-ASR-1.7B（语音识别）
```bash
cd ~/llm_server
# 用于 start_asr.sh (vLLM 加载)
modelscope download --model Qwen/Qwen3-ASR-1.7B --local_dir ./models/Qwen/Qwen3-ASR-1.7B
```
> 两个目录内容相同，只是命名不同以适配不同加载方式。
### 3.3 Qwen3-Embedding-0.6B（文本嵌入）
```bash
cd ~/llm_server
modelscope download --model Qwen/Qwen3-Embedding-0.6B --local_dir ./models/Qwen/Qwen3-Embedding-0___6B
```
### 3.4 Qwen3-Reranker-4B（重排序）
```bash
cd ~/llm_server
modelscope download --model Qwen/Qwen3-Reranker-4B --local_dir ./models/Qwen/Qwen3-Reranker-4B
```
### 3.5 Genie TTS（语音合成，端口 8902）
GENIE 是一个基于开源 TTS 项目 **GPT-SoVITS** 构建的轻量级推理引擎，支持中文/日语/英语/韩语，提供极致的 CPU/GPU 推理性能。
#### Step 1：安装 genie-tts
```bash
pip install genie-tts
```
#### Step 2：下载基础资源文件 GenieData（~391MB）
```bash
git clone https://huggingface.co/High-Logic/Genie
```
**方式三：从本地上传**
如果已在本地下载好完整 Genie 目录（如 `Agent/Genie/`），直接 scp：
```bash
scp -P 60007 -r C:\Users\86166\Desktop\Agent_Learning_minor\Agent\Genie\* minor123@semi-ai.tech:~/llm_server/models/genie_tts/
```
> 注意：环境变量 `GENIE_DATA_DIR` 已在 `tts_server.py` 中自动指向 `models/genie_tts/GenieData`，无需额外设置。
#### Step 3：验证文件完整性
完成后检查以下路径均存在：
| 文件/目录 | 说明 |
|-----------|------|
| `models/genie_tts/GenieData/` | 基础资源（词典、声码器等） |
| `models/genie_tts/CharacterModels/v2ProPlus/feibi/tts_models/` | feibi 角色 ONNX 模型 |
| `models/genie_tts/CharacterModels/v2ProPlus/feibi/prompt_wav/zh_vo_Main_Linaxita_2_1_10_26.wav` | feibi 参考音频 |
> 如需使用其他预定义角色（mika、ThirtySeven 等），可浏览 [HuggingFace CharacterModels](https://huggingface.co/High-Logic/Genie/tree/main/CharacterModels) 并按相同方式下载。
### 3.7 完成后的目录结构
```
~/llm_server/
├── models/
│ ├── Qwen/
│ │ ├── Qwen3-ASR-1.7B/ # ASR（vLLM）
│ │ ├── Qwen3-Embedding-0___6B/ # Embedding
│ │ └── Qwen3-Reranker-4B/ # Reranker
│ └── genie_tts/
│ ├── GenieData/
│ └── CharacterModels/
│ └── v2ProPlus/
│ └── feibi/
│ ├── tts_models/
│ └── prompt_wav/
│ └── zh_vo_Main_Linaxita_2_1_10_26.wav
├── start_asr.sh # 启动 ASR 服务
├── start_tts.sh # 启动 TTS 服务
├── start_rag_server.sh # 启动 Reranker + Embedding 服务
└── ...
```
## 4. 启动各服务
### 4.1 ASR（语音识别）— 端口 8901
```bash
cd ~/llm_server
bash start_asr.sh
```
测试：
```bash
python3 test_asr.py --audio /path/to/audio.wav --url http://127.0.0.1:8901/v1/chat/completions
```
### 4.2 TTS（语音合成）— 端口 8902
```bash
cd ~/llm_server
bash start_tts.sh
```
测试：
```bash
python3 test_tts.py
```
> 注意：测试脚本中 `TTS_BASE_URL` 默认为 `http://192.168.137.164:8902`，在远程服务器上需改为 `http://127.0.0.1:8902`。
### 4.3 Reranker + Embedding — 端口 8903
```bash
cd ~/llm_server
bash start_rag_server.sh
```
测试：
```bash
python3 test_rag.py
```
## 5. 服务端口汇总
| 服务 | 脚本 | 端口 |
|------|------|------|
| ASR | `start_asr.sh` | 8901 |
| TTS | `start_tts.sh` | 8902 |
| Reranker | `start_rag_server.sh` | 8903 |
| Embedding | `start_rag_server.sh` | 8903 |
## 6. 停止服务
```bash
kill $(lsof -t -i:8901) # 停止 ASR
kill $(lsof -t -i:8902) # 停止 TTS
kill $(lsof -t -i:8903) # 停止 Reranker/Embedding
```
## 7. 路径说明
所有 Python 脚本中的模型路径已改为基于脚本所在目录的相对路径（通过 `os.path.dirname(os.path.abspath(__file__))` 动态获取），Shell 脚本使用 `${SCRIPT_DIR}` 定位。因此目录整体迁移到任意位置均可正常运行，无需再修改路径。
llm_server/rag_server.py
+6 -2
View File
@@ -1,3 +1,4 @@
import os
import torch
import torch.nn.functional as F
from fastapi import FastAPI
@@ -6,11 +7,14 @@ from typing import List
from modelscope import AutoTokenizer, AutoModel, AutoModelForCausalLM
from transformers import BitsAndBytesConfig
# 自动获取脚本所在目录作为基础路径
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
# 在 load_models() 里添加量化配置
quantization_config = BitsAndBytesConfig(load_in_8bit=True)
# ===================== 模型路径 =====================
EMBED_MODEL_PATH = "/home/admin/minor_agent/modelscope/models/Qwen/Qwen3-Embedding-0___6B"
RERANK_MODEL_PATH = "/home/admin/minor_agent/modelscope/models/Qwen/Qwen3-Reranker-4B"
EMBED_MODEL_PATH = os.path.join(BASE_DIR, "models", "Qwen", "Qwen3-Embedding-0___6B")
RERANK_MODEL_PATH = os.path.join(BASE_DIR, "models", "Qwen", "Qwen3-Reranker-4B")
# ===================== 初始化 FastAPI =====================
app = FastAPI(title="RAG Backend with 8-bit Qwen3 Models")
llm_server/speech_server.py
-88
View File
@@ -1,88 +0,0 @@
from fastapi import FastAPI, UploadFile, Query
from fastapi.responses import Response
import io
import numpy as np
import soundfile as sf
from contextlib import asynccontextmanager
import torch
import tempfile
import os
# 直接指定本地模型文件夹（绝对路径）
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
ASR_MODEL_PATH = os.path.join(BASE_DIR, "modelscope", "hub", "models", "Qwen", "Qwen3-ASR-1___7B")
TTS_MODEL_PATH = os.path.join(BASE_DIR, "modelscope", "hub", "models", "OpenBMB", "VoxCPM2")
models = {}
@asynccontextmanager
async def lifespan(app: FastAPI):
from transformers import BitsAndBytesConfig
quantization_asr_config = BitsAndBytesConfig(load_in_8bit=True)
print(f"Loading ASR model from {ASR_MODEL_PATH} ...")
from qwen_asr import Qwen3ASRModel
models["asr"] = Qwen3ASRModel.from_pretrained(
ASR_MODEL_PATH,
dtype=torch.bfloat16,
device_map="cuda:0",
max_inference_batch_size=4,
attn_implementation="flash_attention_2",
max_new_tokens=512,
quantization_config=quantization_asr_config
)
from voxcpm import VoxCPM
models["tts"] = VoxCPM.from_pretrained(
TTS_MODEL_PATH,
load_denoiser=False,
)
models["tts_sample_rate"] = models["tts"].tts_model.sample_rate
print("ASR & TTS models loaded.")
yield
del models["asr"], models["tts"]
app = FastAPI(lifespan=lifespan)
@app.post("/asr")
async def speech_to_text(file: UploadFile):
# 保存上传文件为临时 WAV
suffix = os.path.splitext(file.filename)[-1] or ".wav"
with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
tmp.write(await file.read())
tmp_path = tmp.name
try:
# 调用 transcribe，这里直接使用自动语言检测
results = models["asr"].transcribe(
audio=[tmp_path],
language=None, # 自动检测，或指定 ["Chinese", "English"]
return_time_stamps=False, # 如需时间戳可改为 True
)
# 取第一个结果（仅一个音频）
if results:
text = results[0].text
else:
text = ""
finally:
os.unlink(tmp_path) # 清理临时文件
return {"text": text}
@app.get("/tts")
async def text_to_speech(text: str = Query(...)):
with torch.no_grad():
wav = models["tts"].generate(
text=text,
cfg_value=2.0,
inference_timesteps=10,
)
buffer = io.BytesIO()
sf.write(buffer, wav, models["tts_sample_rate"], format="WAV")
return Response(buffer.getvalue(), media_type="audio/wav")
@app.get("/health")
async def health():
return {"status": "ok", "models": list(models.keys())}
llm_server/start_asr.sh
+7 -4
View File
@@ -3,8 +3,8 @@ set -euo pipefail
# 自动获取脚本所在目录
SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VENV_DIR="${SCRIPT_DIR}/.venv_audio"
MODEL_DIR="${SCRIPT_DIR}/modelscope/models/Qwen/Qwen3-ASR-1.7B"
VENV_DIR="${SCRIPT_DIR}/myvenv"
MODEL_DIR="${SCRIPT_DIR}/models/Qwen/Qwen3-ASR-1.7B"
HOST_PORT="${HOST_PORT:-8901}"
# 激活虚拟环境
@@ -26,10 +26,13 @@ cd "${SCRIPT_DIR}"
export VLLM_USE_MODELSCOPE=true
export MODELSCOPE_DOWNLOAD_MODE=local
export MODELSCOPE_HUB_ENABLE_FILE_CHECK=false
export MODELSCOPE_CACHE="${SCRIPT_DIR}/modelscope"
export MODELSCOPE_CACHE="${SCRIPT_DIR}/models"
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
# 固定使用 GPU 0（多 GPU 服务器上所有模型共用一张卡）
export CUDA_VISIBLE_DEVICES=3
# 启动 vLLM 服务
vllm serve "${MODEL_DIR}" \
--port "${HOST_PORT}" \
@@ -40,5 +43,5 @@ vllm serve "${MODEL_DIR}" \
--max-model-len 16384 \
--allowed-local-media-path / \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.10 \
--gpu-memory-utilization 0.15 \
--max-num-batched-tokens 4096
llm_server/start_audio.sh
-26
View File
@@ -1,26 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BUNDLE_DIR="${SCRIPT_DIR}"
VENV_DIR="${BUNDLE_DIR}/.venv_audio"
HOST_PORT="${HOST_PORT:-8100}"
# 激活虚拟环境
source "${VENV_DIR}/bin/activate"
# 检查端口占用
if lsof -i "tcp:${HOST_PORT}" -t >/dev/null 2>&1; then
echo "Port ${HOST_PORT} already in use" >&2
exit 1
fi
echo "Starting audio server on port ${HOST_PORT} ..."
cd "${BUNDLE_DIR}"
export MODELSCOPE_DOWNLOAD_MODE=local
export MODELSCOPE_HUB_ENABLE_FILE_CHECK=false
export MODELSCOPE_CACHE="${BUNDLE_DIR}/modelscope"
# 强制 HuggingFace transformers 离线
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
uvicorn speech_server:app --host 0.0.0.0 --port "${HOST_PORT}" --log-level info
llm_server/start_rag_server.sh
+5 -2
View File
@@ -3,7 +3,7 @@ set -euo pipefail
# 自动获取脚本所在目录
SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VENV_DIR="${VENV_DIR:-${SCRIPT_DIR}/.venv_audio}"
VENV_DIR="${VENV_DIR:-${SCRIPT_DIR}/myvenv}"
HOST_PORT="${HOST_PORT:-8903}"
# 激活虚拟环境
@@ -28,7 +28,10 @@ export TRANSFORMERS_OFFLINE=1
# 指定 modelscope 的缓存目录（如果你的模型保存在此路径下）
# 根据你的下载路径调整，或将其设置为环境变量覆盖
export MODELSCOPE_CACHE="${MODELSCOPE_CACHE:-/home/admin/minor_agent/modelscope}"
export MODELSCOPE_CACHE="${MODELSCOPE_CACHE:-${SCRIPT_DIR}/models}"
# 固定使用 GPU 0（多 GPU 服务器上所有模型共用一张卡）
export CUDA_VISIBLE_DEVICES=3
# 启动 FastAPI 服务
uvicorn rag_server:app \
llm_server/start_tts.sh
+4 -1
View File
@@ -3,7 +3,7 @@ set -euo pipefail
# 自动获取脚本所在目录
SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VENV_DIR="${SCRIPT_DIR}/myenv"
VENV_DIR="${SCRIPT_DIR}/myvenv"
HOST_PORT="${HOST_PORT:-8902}"
# 激活虚拟环境
@@ -27,6 +27,9 @@ export MODELSCOPE_CACHE="${SCRIPT_DIR}/models"
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
# 固定使用 GPU 0（多 GPU 服务器上所有模型共用一张卡）
export CUDA_VISIBLE_DEVICES=3
# 启动 FastAPI 服务，使用 4 个 worker 进程
uvicorn tts_server:app \
--host 0.0.0.0 \
llm_server/test_asr.py
+24 -7
View File
@@ -12,6 +12,8 @@ import requests
import argparse
import os
import sys
import base64
import mimetypes
DEFAULT_URL = "http://127.0.0.1:8901/v1/chat/completions"
@@ -33,12 +35,24 @@ def get_audio_duration_sec(file_path: str) -> float:
return -1.0
# ----------------------------------------------------------------------
def file_to_url(file_path: str) -> str:
"""Convert local file path to a file:// URL accepted by vLLM."""
abs_path = os.path.abspath(file_path)
return "file://" + abs_path.replace("\\", "/")
def file_to_url(file_path: str, use_base64: bool = False) -> str:
"""Convert local file path to a URL accepted by vLLM.
When the server is remote, use base64 data URL instead of file://
since the remote server cannot access local filesystem paths.
"""
if use_base64:
mime_type, _ = mimetypes.guess_type(file_path)
if mime_type is None:
mime_type = "audio/wav"
with open(file_path, "rb") as f:
b64_data = base64.b64encode(f.read()).decode("utf-8")
return f"data:{mime_type};base64,{b64_data}"
else:
abs_path = os.path.abspath(file_path)
return "file://" + abs_path.replace("\\", "/")
def caption_audio(audio_path: str, server_url: str = DEFAULT_URL) -> str:
def caption_audio(audio_path: str, server_url: str = DEFAULT_URL, use_base64: bool = False) -> str:
"""
Send audio to the Qwen3-Omni-30B-A3B-Captioner and return its description.
"""
@@ -54,7 +68,7 @@ def caption_audio(audio_path: str, server_url: str = DEFAULT_URL) -> str:
file=sys.stderr
)
audio_url = file_to_url(audio_path)
audio_url = file_to_url(audio_path, use_base64=use_base64)
# The model expects NO text prompt. Only the audio input is required.
payload = {
@@ -95,14 +109,17 @@ def main():
help="Output text file (if not provided, print to stdout)")
parser.add_argument("--verbose", "-v", action="store_true",
help="Print detailed request/response info")
parser.add_argument("--base64", "-b", action="store_true",
help="Send audio as base64 data URL (use when server is remote)")
args = parser.parse_args()
if args.verbose:
print(f"Audio file: {args.audio}")
print(f"Server URL: {args.url}")
print(f"Base64 mode: {args.base64}")
caption_text = caption_audio(args.audio, args.url)
caption_text = caption_audio(args.audio, args.url, use_base64=args.base64)
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
llm_server/test_asr.wav
BIN
View File
Binary file not shown.
llm_server/test_output.wav
BIN
View File
Binary file not shown.
llm_server/test_tts.py
+1 -1
View File
@@ -13,7 +13,7 @@ from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
# ======================== 配置 ========================
TTS_BASE_URL = "http://192.168.137.164:8902"
TTS_BASE_URL = "http://127.0.0.1:8902"
OUTPUT_WAV = "test_output.wav"
REQUEST_TIMEOUT = 120
MAX_PARALLEL = 4
llm_server/tts_server.py
+7 -4
View File
@@ -5,7 +5,10 @@ Genie TTS 极简服务 —— 单文本直接合成，无分块无内部线程
"""
import os
os.environ["GENIE_DATA_DIR"] = r"/home/dev/Desktop/minor_agent/models/genie_tts/GenieData"
# 自动获取脚本所在目录作为基础路径
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
os.environ["GENIE_DATA_DIR"] = os.path.join(BASE_DIR, "models", "genie_tts", "GenieData")
import tempfile
from contextlib import asynccontextmanager
@@ -16,8 +19,8 @@ import genie_tts as genie
# ======================== 配置区域 ========================
DEFAULT_CHARACTER = "feibi"
DEFAULT_ONNX_MODEL_DIR = r"/home/dev/Desktop/minor_agent/models/genie_tts/CharacterModels/v2ProPlus/feibi/tts_models"
DEFAULT_REFERENCE_AUDIO = "/home/dev/Desktop/minor_agent/models/genie_tts/CharacterModels/v2ProPlus/feibi/prompt_wav/zh_vo_Main_Linaxita_2_1_10_26.wav"
DEFAULT_ONNX_MODEL_DIR = os.path.join(BASE_DIR, "models", "genie_tts", "CharacterModels", "v2ProPlus", "feibi", "tts_models")
DEFAULT_REFERENCE_AUDIO = os.path.join(BASE_DIR, "models", "genie_tts", "CharacterModels", "v2ProPlus", "feibi", "prompt_wav", "zh_vo_Main_Linaxita_2_1_10_26.wav")
DEFAULT_REFERENCE_TEXT = "在此之前，请您务必继续享受旅居拉古那的时光。"
DEFAULT_LANGUAGE = "zh"
TMP_DIR = tempfile.gettempdir() # 临时文件存放目录
@@ -41,7 +44,7 @@ async def lifespan(app: FastAPI):
language=DEFAULT_LANGUAGE
)
print("✅ Genie TTS 模型加载成功！")
print("👉 调用方式：http://localhost:8000/tts?text=你要说的话")
print("👉 调用方式：http://localhost:8902/tts?text=你要说的话")
except Exception as e:
print(f"❌ 服务启动失败: {e}")
raise e