优化前缀缓存机制和UI,并引入一些新工具.preview
This commit is contained in:
1 parent
863abe7c76
commit
6131c2563e
102 files changed
+9087
-4351
No files matched your search
+35
-10
@@ -85,42 +85,64 @@ sudo apt update && sudo apt install -y ffmpeg libavcodec-dev libavformat-dev lib
|
||||
| `LOCAL_MODEL_DIR` / `MODEL_PATH` / `VOXCPM_MODEL_PATH` | 模型目录 | 全部 |
|
||||
| `CUDA_VISIBLE_DEVICES` | GPU 设备号 | 全部 |
|
||||
| `SPECIFIC_GPU` | 单 GPU 指定(写 `CUDA_VISIBLE_DEVICES`) | image_gen / tts |
|
||||
| `SERVER_API_KEY` | API Key 鉴权(空=不校验,任意值放行) | 全部 |
|
||||
| `SERVER_MODELNAME` | 服务模型名(请求 body.model 须匹配;空=不校验) | 全部 |
|
||||
|
||||
> **鉴权与模型名校验**:`SERVER_API_KEY` 为空时服务不校验 API Key(任意值放行);
|
||||
> `SERVER_MODELNAME` 为空时服务不校验请求中的 `model` 字段。
|
||||
> 部署时如需启用,示例:
|
||||
> ```bash
|
||||
> SERVER_API_KEY=sk-xxx SERVER_MODELNAME=Qwen3-ASR-1.7B bash start_asr.sh
|
||||
> ```
|
||||
|
||||
### LLM · 8900
|
||||
|
||||
```bash
|
||||
bash start_llm.sh
|
||||
# 启用鉴权与自定义模型名
|
||||
# SERVER_API_KEY=sk-xxx SERVER_MODELNAME=Qwen/Qwen3.6-35B-A3B-FP8 bash start_llm.sh
|
||||
# 测试
|
||||
python3 test_llm.py --url http://127.0.0.1:8900 # 非流式
|
||||
python3 test_llm.py --url http://127.0.0.1:8900 --stream # 流式
|
||||
```
|
||||
|
||||
> vLLM 加载 Qwen3.6-35B-A3B-FP8,OpenAI 兼容 API,使用 `chat_template.jinja` 格式化对话。
|
||||
> `--served-model-name` 由 `SERVER_MODELNAME` 控制(默认 `Qwen/Qwen3.6-35B-A3B-FP8`);
|
||||
> `SERVER_API_KEY` 非空时传给 vLLM `--api-key`,请求需携带 `Authorization: Bearer <key>`。
|
||||
|
||||
### ASR · 8901
|
||||
|
||||
```bash
|
||||
bash start_asr.sh
|
||||
# 启用鉴权与自定义模型名
|
||||
# SERVER_API_KEY=sk-xxx SERVER_MODELNAME=Qwen3-ASR-1.7B bash start_asr.sh
|
||||
# 测试
|
||||
python3 test_asr.py --audio /path/to/audio.wav --url http://127.0.0.1:8901/v1/chat/completions
|
||||
```
|
||||
|
||||
> vLLM 加载 Qwen3-ASR-1.7B,`--served-model-name` 由 `SERVER_MODELNAME` 控制(默认 `Qwen3-ASR-1.7B`)。
|
||||
|
||||
### TTS · 8902(流式)
|
||||
|
||||
```bash
|
||||
bash start_streaming_tts.sh
|
||||
# 启用鉴权与自定义模型名
|
||||
# SERVER_API_KEY=sk-xxx SERVER_MODELNAME=VoxCPM1.5 bash start_streaming_tts.sh
|
||||
# 测试(流式返回 wav 分块)
|
||||
curl -X POST http://127.0.0.1:8902/stream_tts \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "你好,这是一个流式语音合成测试。", "request_id": "test-1"}'
|
||||
-d '{"model": "VoxCPM1.5", "text": "你好,这是一个流式语音合成测试。", "request_id": "test-1"}'
|
||||
```
|
||||
|
||||
> VoxCPM 流式合成,边生成边返回 wav chunk,配合 `reference_audio.wav` 控制音色。
|
||||
> 配置了 `SERVER_MODELNAME` 时请求 body 须带 `model` 字段;配置了 `SERVER_API_KEY` 时须带 `Authorization: Bearer <key>` 头。
|
||||
|
||||
### RAG · 8903(Embedding + Reranker)
|
||||
|
||||
```bash
|
||||
bash start_rag_server.sh
|
||||
# 启用鉴权与自定义模型名
|
||||
# SERVER_API_KEY=sk-xxx SERVER_MODELNAME=Qwen3-Embedding-0.6B bash start_rag_server.sh
|
||||
python3 test_rag.py
|
||||
```
|
||||
|
||||
@@ -128,10 +150,12 @@ python3 test_rag.py
|
||||
|
||||
```bash
|
||||
LOCAL_MODEL_DIR=./models/Tongyi-MAI/Z-Image-Turbo SPECIFIC_GPU=3 bash start_image_gen.sh
|
||||
# 启用鉴权与自定义模型名
|
||||
# SERVER_API_KEY=sk-xxx SERVER_MODELNAME=Z-Image-Turbo bash start_image_gen.sh
|
||||
# 测试
|
||||
curl -X POST http://127.0.0.1:8904/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"prompt": "一只橘猫坐在窗台上,阳光洒落", "width": 1024, "height": 1024, "return_format": "base64"}' \
|
||||
-d '{"model": "Z-Image-Turbo", "prompt": "一只橘猫坐在窗台上,阳光洒落", "width": 1024, "height": 1024, "return_format": "base64"}' \
|
||||
| python3 -c "import sys,json; d=json.load(sys.stdin); print(f'成功:{d.get(\"success\")} 尺寸:{d.get(\"width\")}x{d.get(\"height\")} 耗时:{d.get(\"elapsed_seconds\",\"?\")}s')"
|
||||
```
|
||||
|
||||
@@ -171,14 +195,15 @@ for p in 8900 8901 8902 8903 8904; do kill $(lsof -t -i:$p); done
|
||||
|
||||
## 五、接入 Agent
|
||||
|
||||
在 minor Agent 配置向导(或 `env_config.json`)中填入对应地址即可启用各能力:
|
||||
四个本地服务(ASR / TTS / RAG / ImageGen)在 Agent 的 `env_config.json` **`models` 列表中注册**(设置 → 环境变量 → LLM 模型列表,或直接编辑 `src/agent/config/env_config.json`),`base_url` 指向本地映射端口,`api_key` 留空(对应服务端未启用鉴权)。Agent 运行时按 `model` 字段匹配取得服务地址:
|
||||
|
||||
| Agent 能力 | 配置字段 | 示例值 |
|
||||
|-----------|----------|--------|
|
||||
| LLM 对话 | `models[].base_url` | `http://localhost:8900/v1` |
|
||||
| 语音识别 | `ASR_BASE_URL` | `http://localhost:8901/v1` |
|
||||
| 语音合成 | `STREAMING_TTS_URL` | `http://localhost:8902` |
|
||||
| 知识库检索 | `RAG_BASE_URL` | `http://localhost:8903` |
|
||||
| 图像生成 | (由 `image_gen` 工具读取) | `http://localhost:8904` |
|
||||
| Agent 能力 | 模型列表条目 | 字段示例 |
|
||||
|-----------|-------------|----------|
|
||||
| 语音识别 | `name: "ASR 语音识别"` | `model: "Qwen3-ASR-1.7B"`, `base_url: "http://localhost:8901"`, `api_key: ""` |
|
||||
| 语音合成 | `name: "TTS 语音合成"` | `model: "VoxCPM1.5"`, `base_url: "http://localhost:8902"`, `api_key: ""` |
|
||||
| 知识库检索 | `name: "RAG 检索"` | `model: "Qwen3-Embedding-0.6B"`, `base_url: "http://localhost:8903"`, `api_key: ""` |
|
||||
| 图像生成 | `name: "图像生成"` | `model: "Z-Image-Turbo"`, `base_url: "http://localhost:8904"`, `api_key: ""` |
|
||||
|
||||
> 💡 若服务端部署时设置了 `SERVER_API_KEY` / `SERVER_MODELNAME`,请把模型条目中的 `api_key` 与 `model` 改成与之一致,Agent 发出的请求会携带 `Authorization: Bearer <api_key>` 与 `model` 字段。
|
||||
>
|
||||
> 💡 不必自建本服务,Agent 也支持任何 **OpenAI 兼容 API**(云端 Qwen / DeepSeek / OpenAI 等),填入 `base_url` + `api_key` 即可。
|
||||
@@ -26,6 +26,8 @@ import torch
|
||||
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
DEFAULT_PORT = 8904
|
||||
DEFAULT_MODEL_ID = "Tongyi-MAI/Z-Image-Turbo"
|
||||
DEFAULT_MODEL_NAME = "Z-Image-Turbo" # 默认服务模型名(校验请求 body.model 用)
|
||||
DEFAULT_API_KEY = "" # 默认空:不校验 API Key(任意值放行)
|
||||
|
||||
# 本地模型路径:优先读环境变量 LOCAL_MODEL_DIR,未设置则回退到远程 ID
|
||||
LOCAL_MODEL_DIR = os.environ.get("LOCAL_MODEL_DIR", "")
|
||||
@@ -45,6 +47,39 @@ warnings.filterwarnings("ignore", category=FutureWarning)
|
||||
from fastapi import FastAPI, Request, HTTPException
|
||||
from fastapi.responses import JSONResponse, Response
|
||||
|
||||
# 服务参数(--api-key / --model-name):
|
||||
# api_key 为空:不校验(任意值放行);model_name 为空:不校验请求 model 字段
|
||||
_model_args: dict = {"api_key": "", "model_name": ""}
|
||||
|
||||
|
||||
def _check_api_key(request: Request):
|
||||
"""校验请求 API Key:服务未配置 key(空)时放行任意值;否则校验 Authorization / X-API-Key。
|
||||
|
||||
校验失败抛出 HTTP 401。
|
||||
"""
|
||||
api_key = _model_args.get("api_key", "")
|
||||
if not api_key:
|
||||
return
|
||||
header = request.headers.get("Authorization", "")
|
||||
if header.startswith("Bearer "):
|
||||
provided = header[len("Bearer "):].strip()
|
||||
else:
|
||||
provided = request.headers.get("X-API-Key", "").strip()
|
||||
if provided != api_key:
|
||||
raise HTTPException(status_code=401, detail="API Key 无效或缺失")
|
||||
|
||||
|
||||
def _check_model_name(body: dict):
|
||||
"""校验请求 modelname:服务未配置模型名(空)时放行任意值;否则要求 body.model 匹配。
|
||||
|
||||
校验失败抛出 HTTP 400。
|
||||
"""
|
||||
model_name = _model_args.get("model_name", "")
|
||||
if not model_name:
|
||||
return
|
||||
if str(body.get("model") or "").strip() != model_name:
|
||||
raise HTTPException(status_code=400, detail=f"模型名不匹配: 期望 '{model_name}',收到 '{body.get('model') or ''}'")
|
||||
|
||||
# Lazy-loaded pipeline singleton
|
||||
_pipe = None
|
||||
_pipe_lock = threading.Lock()
|
||||
@@ -129,6 +164,10 @@ async def generate_image(request: Request):
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail="请求体必须为 JSON")
|
||||
|
||||
# 鉴权与模型名校验
|
||||
_check_api_key(request)
|
||||
_check_model_name(body)
|
||||
|
||||
prompt = (body.get("prompt") or "").strip()
|
||||
if not prompt:
|
||||
raise HTTPException(status_code=400, detail="prompt 不能为空")
|
||||
@@ -194,14 +233,18 @@ async def health():
|
||||
|
||||
|
||||
# ======================== 启动入口 ========================
|
||||
_model_args: dict = {}
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="Z-Image-Turbo Image Generation Server")
|
||||
parser.add_argument("--port", type=int, default=DEFAULT_PORT, help=f"服务端口(默认 {DEFAULT_PORT})")
|
||||
parser.add_argument("--model-name", type=str, default=DEFAULT_MODEL_NAME,
|
||||
help=f"服务模型名(校验请求 body.model;默认 {DEFAULT_MODEL_NAME},留空则不校验)")
|
||||
parser.add_argument("--api-key", type=str, default=DEFAULT_API_KEY,
|
||||
help="API Key(校验 Authorization: Bearer <key> 或 X-API-Key;默认空=任意值放行)")
|
||||
args = parser.parse_args()
|
||||
|
||||
_model_args["port"] = args.port
|
||||
_model_args["model_name"] = args.model_name
|
||||
_model_args["api_key"] = args.api_key
|
||||
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=args.port)
|
||||
@@ -1,7 +1,9 @@
|
||||
import os
|
||||
import argparse
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from fastapi import FastAPI
|
||||
from fastapi import FastAPI, Request, HTTPException
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel
|
||||
from typing import List
|
||||
from modelscope import AutoTokenizer, AutoModel, AutoModelForCausalLM
|
||||
@@ -16,6 +18,37 @@ quantization_config = BitsAndBytesConfig(load_in_8bit=True)
|
||||
EMBED_MODEL_PATH = os.path.join(BASE_DIR, "models", "Qwen", "Qwen3-Embedding-0___6B")
|
||||
RERANK_MODEL_PATH = os.path.join(BASE_DIR, "models", "Qwen", "Qwen3-Reranker-4B")
|
||||
|
||||
# ===================== 服务参数(--api-key / --model-name)=====================
|
||||
# api_key 为空:不校验(任意值放行);model_name 为空:不校验请求 model 字段
|
||||
_model_args: dict = {"api_key": "", "model_name": ""}
|
||||
|
||||
def check_api_key(request: Request):
|
||||
"""校验请求 API Key:服务未配置 key(空)时放行任意值;否则校验 Authorization / X-API-Key。
|
||||
|
||||
校验失败抛出 HTTP 401。
|
||||
"""
|
||||
api_key = _model_args.get("api_key", "")
|
||||
if not api_key:
|
||||
return
|
||||
header = request.headers.get("Authorization", "")
|
||||
if header.startswith("Bearer "):
|
||||
provided = header[len("Bearer "):].strip()
|
||||
else:
|
||||
provided = request.headers.get("X-API-Key", "").strip()
|
||||
if provided != api_key:
|
||||
raise HTTPException(status_code=401, detail="API Key 无效或缺失")
|
||||
|
||||
def check_model_name(model: str | None):
|
||||
"""校验请求 modelname:服务未配置模型名(空)时放行任意值;否则要求请求 model 匹配。
|
||||
|
||||
校验失败抛出 HTTP 400。
|
||||
"""
|
||||
model_name = _model_args.get("model_name", "")
|
||||
if not model_name:
|
||||
return
|
||||
if str(model or "").strip() != model_name:
|
||||
raise HTTPException(status_code=400, detail=f"模型名不匹配: 期望 '{model_name}',收到 '{model or ''}'")
|
||||
|
||||
# ===================== 初始化 FastAPI =====================
|
||||
app = FastAPI(title="RAG Backend with 8-bit Qwen3 Models")
|
||||
|
||||
@@ -87,6 +120,7 @@ def load_models():
|
||||
# ===================== API 请求体定义 =====================
|
||||
class EmbedRequest(BaseModel):
|
||||
task_description: str = "Given a web search query, retrieve relevant passages that answer the query"
|
||||
model: str | None = None # 可选,服务配置了 --model-name 时须匹配
|
||||
queries: List[str]
|
||||
documents: List[str] # 可选,若只查询文本可只传 queries
|
||||
|
||||
@@ -96,6 +130,7 @@ class EmbedResponse(BaseModel):
|
||||
|
||||
class RerankRequest(BaseModel):
|
||||
task_description: str = "Given a web search query, retrieve relevant passages that answer the query"
|
||||
model: str | None = None # 可选,服务配置了 --model-name 时须匹配
|
||||
query: str
|
||||
documents: List[str]
|
||||
|
||||
@@ -104,7 +139,9 @@ class RerankResponse(BaseModel):
|
||||
|
||||
# ===================== Embedding 端点 =====================
|
||||
@app.post("/embed", response_model=EmbedResponse)
|
||||
async def get_embeddings(req: EmbedRequest):
|
||||
async def get_embeddings(req: EmbedRequest, request: Request):
|
||||
check_api_key(request)
|
||||
check_model_name(req.model)
|
||||
# 为 query 加上指令,document 不加指令
|
||||
instructed_queries = [get_detailed_instruct(req.task_description, q) for q in req.queries]
|
||||
# 所有文本拼成一个 batch:先是 queries,后是 documents(如果有)
|
||||
@@ -132,7 +169,9 @@ async def get_embeddings(req: EmbedRequest):
|
||||
|
||||
# ===================== Reranker 端点 =====================
|
||||
@app.post("/rerank", response_model=RerankResponse)
|
||||
async def rerank_documents(req: RerankRequest):
|
||||
async def rerank_documents(req: RerankRequest, request: Request):
|
||||
check_api_key(request)
|
||||
check_model_name(req.model)
|
||||
# 构造带格式的输入对
|
||||
pairs = [
|
||||
format_rerank_instruction(req.task_description, req.query, doc)
|
||||
@@ -166,3 +205,20 @@ async def rerank_documents(req: RerankRequest):
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
# ======================== 启动入口 ========================
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="Qwen3 RAG Server (Embedding + Reranker)")
|
||||
parser.add_argument("--port", type=int, default=8903, help="服务端口(默认 8903)")
|
||||
parser.add_argument("--model-name", type=str, default="Qwen3-Embedding-0.6B",
|
||||
help="服务模型名(校验请求 body.model;默认 Qwen3-Embedding-0.6B,留空则不校验)")
|
||||
parser.add_argument("--api-key", type=str, default="",
|
||||
help="API Key(校验 Authorization: Bearer <key> 或 X-API-Key;默认空=任意值放行)")
|
||||
args = parser.parse_args()
|
||||
|
||||
_model_args["api_key"] = args.api_key
|
||||
_model_args["model_name"] = args.model_name
|
||||
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=args.port)
|
||||
+24
-10
@@ -6,6 +6,9 @@ SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
VENV_DIR="${SCRIPT_DIR}/myenv"
|
||||
MODEL_DIR="${SCRIPT_DIR}/models/Qwen/Qwen3-ASR-1.7B"
|
||||
HOST_PORT="${HOST_PORT:-8901}"
|
||||
# 鉴权与模型名校验:SERVER_API_KEY 为空时 vLLM 不校验 API Key(任意值放行)
|
||||
SERVER_API_KEY="${SERVER_API_KEY:-}"
|
||||
SERVER_MODELNAME="${SERVER_MODELNAME:-Qwen3-ASR-1.7B}"
|
||||
|
||||
# 激活虚拟环境
|
||||
source "${VENV_DIR}/bin/activate"
|
||||
@@ -18,6 +21,10 @@ fi
|
||||
|
||||
echo "Starting vLLM server on port ${HOST_PORT} ..."
|
||||
echo "Model path: ${MODEL_DIR}"
|
||||
echo "Served model name: ${SERVER_MODELNAME}"
|
||||
if [ -n "${SERVER_API_KEY}" ]; then
|
||||
echo "API Key 校验已启用"
|
||||
fi
|
||||
|
||||
# 进入脚本目录
|
||||
cd "${SCRIPT_DIR}"
|
||||
@@ -34,14 +41,21 @@ export TRANSFORMERS_OFFLINE=1
|
||||
export CUDA_VISIBLE_DEVICES=3
|
||||
|
||||
# 启动 vLLM 服务
|
||||
vllm serve "${MODEL_DIR}" \
|
||||
--port "${HOST_PORT}" \
|
||||
--host 0.0.0.0 \
|
||||
--quantization fp8 \
|
||||
--dtype auto \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-model-len 16384 \
|
||||
--allowed-local-media-path / \
|
||||
--tensor-parallel-size 1 \
|
||||
--gpu-memory-utilization 0.15 \
|
||||
ASR_ARGS=(
|
||||
--port "${HOST_PORT}"
|
||||
--host 0.0.0.0
|
||||
--quantization fp8
|
||||
--dtype auto
|
||||
--kv-cache-dtype fp8
|
||||
--max-model-len 16384
|
||||
--allowed-local-media-path /
|
||||
--tensor-parallel-size 1
|
||||
--gpu-memory-utilization 0.15
|
||||
--max-num-batched-tokens 4096
|
||||
--served-model-name "${SERVER_MODELNAME}"
|
||||
)
|
||||
# API Key 为空时不传 --api-key(vLLM 默认放行任意值)
|
||||
if [ -n "${SERVER_API_KEY}" ]; then
|
||||
ASR_ARGS+=(--api-key "${SERVER_API_KEY}")
|
||||
fi
|
||||
vllm serve "${MODEL_DIR}" "${ASR_ARGS[@]}"
|
||||
@@ -4,6 +4,9 @@ set -euo pipefail
|
||||
# ---------- 配置 ----------
|
||||
HOST_PORT="${HOST_PORT:-8904}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# 鉴权与模型名校验:SERVER_API_KEY / SERVER_MODELNAME 为空时服务端不校验(任意值放行)
|
||||
SERVER_API_KEY="${SERVER_API_KEY:-}"
|
||||
SERVER_MODELNAME="${SERVER_MODELNAME:-Z-Image-Turbo}"
|
||||
|
||||
# ---------- Python 虚拟环境 ----------
|
||||
if [ -f "${SCRIPT_DIR}/../myenv/bin/activate" ]; then
|
||||
@@ -20,5 +23,9 @@ if [ -n "${CUDA_DEV}" ]; then
|
||||
fi
|
||||
|
||||
echo "[Z-Image] 启动图像生成服务 (端口=${HOST_PORT})"
|
||||
echo "[Z-Image] 服务模型名: ${SERVER_MODELNAME}"
|
||||
if [ -n "${SERVER_API_KEY}" ]; then
|
||||
echo "[Z-Image] API Key 校验已启用"
|
||||
fi
|
||||
cd "${SCRIPT_DIR}"
|
||||
exec python image_gen_server.py --port "${HOST_PORT}"
|
||||
exec python image_gen_server.py --port "${HOST_PORT}" --model-name "${SERVER_MODELNAME}" --api-key "${SERVER_API_KEY}"
|
||||
+29
-16
@@ -5,6 +5,9 @@ SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
VENV_DIR="${SCRIPT_DIR}/myenv"
|
||||
HOST_PORT="${HOST_PORT:-8900}"
|
||||
MODEL_DIR="${SCRIPT_DIR}/models/Qwen/Qwen3___6-35B-A3B-FP8"
|
||||
# 鉴权与模型名校验:SERVER_API_KEY 为空时 vLLM 不校验 API Key(任意值放行)
|
||||
SERVER_API_KEY="${SERVER_API_KEY:-}"
|
||||
SERVER_MODELNAME="${SERVER_MODELNAME:-Qwen/Qwen3.6-35B-A3B-FP8}"
|
||||
|
||||
# 激活虚拟环境
|
||||
source "${VENV_DIR}/bin/activate"
|
||||
@@ -17,6 +20,10 @@ fi
|
||||
|
||||
echo "Starting vLLM server on port ${HOST_PORT} ..."
|
||||
echo "Model path: ${MODEL_DIR}"
|
||||
echo "Served model name: ${SERVER_MODELNAME}"
|
||||
if [ -n "${SERVER_API_KEY}" ]; then
|
||||
echo "API Key 校验已启用"
|
||||
fi
|
||||
|
||||
# 设置环境变量
|
||||
export VLLM_USE_MODELSCOPE=true
|
||||
@@ -28,19 +35,25 @@ export TRANSFORMERS_OFFLINE=1
|
||||
export CUDA_VISIBLE_DEVICES=3
|
||||
|
||||
# 启动 vLLM 服务
|
||||
vllm serve "${MODEL_DIR}" \
|
||||
--port "${HOST_PORT}" \
|
||||
--host 0.0.0.0 \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
|
||||
--enable-prefix-caching \
|
||||
--max_num_batched_tokens 4096 \
|
||||
--max_num_seqs 8 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--chat-template "${SCRIPT_DIR}/chat_template.jinja" \
|
||||
--safetensors-load-strategy eager \
|
||||
--served-model-name Qwen/Qwen3.6-35B-A3B-FP8
|
||||
VLLM_ARGS=(
|
||||
--port "${HOST_PORT}"
|
||||
--host 0.0.0.0
|
||||
--tensor-parallel-size 1
|
||||
--max-model-len 262144
|
||||
--reasoning-parser qwen3
|
||||
--enable-auto-tool-choice
|
||||
--tool-call-parser qwen3_coder
|
||||
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
|
||||
--enable-prefix-caching
|
||||
--max_num_batched_tokens 4096
|
||||
--max_num_seqs 8
|
||||
--kv-cache-dtype fp8_e4m3
|
||||
--chat-template "${SCRIPT_DIR}/chat_template.jinja"
|
||||
--safetensors-load-strategy eager
|
||||
--served-model-name "${SERVER_MODELNAME}"
|
||||
)
|
||||
# API Key 为空时不传 --api-key(vLLM 默认放行任意值)
|
||||
if [ -n "${SERVER_API_KEY}" ]; then
|
||||
VLLM_ARGS+=(--api-key "${SERVER_API_KEY}")
|
||||
fi
|
||||
vllm serve "${MODEL_DIR}" "${VLLM_ARGS[@]}"
|
||||
@@ -5,6 +5,9 @@ set -euo pipefail
|
||||
SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
VENV_DIR="${VENV_DIR:-${SCRIPT_DIR}/myenv}"
|
||||
HOST_PORT="${HOST_PORT:-8903}"
|
||||
# 鉴权与模型名校验:SERVER_API_KEY / SERVER_MODELNAME 为空时服务端不校验(任意值放行)
|
||||
SERVER_API_KEY="${SERVER_API_KEY:-}"
|
||||
SERVER_MODELNAME="${SERVER_MODELNAME:-Qwen3-Embedding-0.6B}"
|
||||
|
||||
# 激活虚拟环境
|
||||
source "${VENV_DIR}/bin/activate"
|
||||
@@ -16,6 +19,10 @@ if lsof -i "tcp:${HOST_PORT}" -t >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
echo "Starting Qwen3 RAG server on port ${HOST_PORT} ..."
|
||||
echo "Served model name: ${SERVER_MODELNAME}"
|
||||
if [ -n "${SERVER_API_KEY}" ]; then
|
||||
echo "API Key 校验已启用"
|
||||
fi
|
||||
|
||||
# 进入脚本目录
|
||||
cd "${SCRIPT_DIR}"
|
||||
@@ -34,7 +41,7 @@ export MODELSCOPE_CACHE="${MODELSCOPE_CACHE:-${SCRIPT_DIR}/models}"
|
||||
export CUDA_VISIBLE_DEVICES=3
|
||||
|
||||
# 启动 FastAPI 服务
|
||||
uvicorn rag_server:app \
|
||||
--host 0.0.0.0 \
|
||||
exec python rag_server.py \
|
||||
--port "${HOST_PORT}" \
|
||||
--workers 1
|
||||
--model-name "${SERVER_MODELNAME}" \
|
||||
--api-key "${SERVER_API_KEY}"
|
||||
@@ -7,6 +7,9 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# VoxCPM 模型路径(可覆盖)
|
||||
VOXCPM_MODEL_PATH="${VOXCPM_MODEL_PATH:-${SCRIPT_DIR}/models/VoxCPM1__5}"
|
||||
# 鉴权与模型名校验:SERVER_API_KEY / SERVER_MODELNAME 为空时服务端不校验(任意值放行)
|
||||
SERVER_API_KEY="${SERVER_API_KEY:-}"
|
||||
SERVER_MODELNAME="${SERVER_MODELNAME:-VoxCPM1.5}"
|
||||
|
||||
# ---------- Python 虚拟环境 ----------
|
||||
if [ -f "${SCRIPT_DIR}/../myenv/bin/activate" ]; then
|
||||
@@ -21,7 +24,13 @@ export CUDA_VISIBLE_DEVICES="${CUDA_DEV}"
|
||||
|
||||
echo "[VoxCPM TTS] 启动流式 TTS 服务 (端口=${HOST_PORT}, CUDA=${CUDA_DEV})"
|
||||
echo "[VoxCPM TTS] 模型路径: ${VOXCPM_MODEL_PATH}"
|
||||
echo "[VoxCPM TTS] 服务模型名: ${SERVER_MODELNAME}"
|
||||
if [ -n "${SERVER_API_KEY}" ]; then
|
||||
echo "[VoxCPM TTS] API Key 校验已启用"
|
||||
fi
|
||||
cd "${SCRIPT_DIR}"
|
||||
exec python streaming_tts_server.py \
|
||||
--port "${HOST_PORT}" \
|
||||
--model-path "${VOXCPM_MODEL_PATH}"
|
||||
--model-path "${VOXCPM_MODEL_PATH}" \
|
||||
--model-name "${SERVER_MODELNAME}" \
|
||||
--api-key "${SERVER_API_KEY}"
|
||||
@@ -39,6 +39,8 @@ except ImportError:
|
||||
# 默认参数(可通过命令行覆盖)
|
||||
DEFAULT_MODEL_PATH = os.path.join(BASE_DIR, "models", "VoxCPM1__5")
|
||||
DEFAULT_PORT = 8902
|
||||
DEFAULT_MODEL_NAME = "VoxCPM1.5" # 默认服务模型名(校验请求 body.model 用)
|
||||
DEFAULT_API_KEY = "" # 默认空:不校验 API Key(任意值放行)
|
||||
|
||||
# 参考音频(用于统一音色,同一人物说的一段话 + 对应文本)
|
||||
# 如果配置了这对参数,所有合成都会克隆该音色,各句音色保持一致
|
||||
@@ -161,6 +163,37 @@ def _build_sse(data: dict) -> str:
|
||||
return f"data: {json.dumps(data, ensure_ascii=False)}\n\n"
|
||||
|
||||
|
||||
def _check_api_key(request: Request) -> str | None:
|
||||
"""校验请求 API Key:服务未配置 key(空)时放行任意值;否则校验 Authorization / X-API-Key。
|
||||
|
||||
返回 None 表示通过,否则返回错误提示文本。
|
||||
"""
|
||||
api_key = _model_args.get("api_key", "")
|
||||
if not api_key:
|
||||
return None
|
||||
header = request.headers.get("Authorization", "")
|
||||
if header.startswith("Bearer "):
|
||||
provided = header[len("Bearer "):].strip()
|
||||
else:
|
||||
provided = request.headers.get("X-API-Key", "").strip()
|
||||
if provided == api_key:
|
||||
return None
|
||||
return "API Key 无效或缺失"
|
||||
|
||||
|
||||
def _check_model_name(body: dict) -> str | None:
|
||||
"""校验请求 modelname:服务未配置模型名(空)时放行任意值;否则要求 body.model 匹配。
|
||||
|
||||
返回 None 表示通过,否则返回错误提示文本。
|
||||
"""
|
||||
model_name = _model_args.get("model_name", "")
|
||||
if not model_name:
|
||||
return None
|
||||
if str(body.get("model") or "").strip() == model_name:
|
||||
return None
|
||||
return f"模型名不匹配: 期望 '{model_name}',收到 '{body.get('model') or ''}'"
|
||||
|
||||
|
||||
def _stream_audio_events(text: str, request_id: str, model_path: str):
|
||||
"""生成 SSE 音频流事件(同步,在独立线程中运行)。"""
|
||||
import numpy as np
|
||||
@@ -237,7 +270,7 @@ def _stream_audio_events(text: str, request_id: str, model_path: str):
|
||||
# ======================== FastAPI ========================
|
||||
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
from fastapi.responses import JSONResponse, StreamingResponse
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -260,12 +293,22 @@ app = FastAPI(title="VoxCPM Streaming TTS", lifespan=lifespan)
|
||||
@app.post("/stream_tts")
|
||||
async def stream_tts(request: Request):
|
||||
"""流式 TTS 端点:接收文本,SSE 流式返回 PCM f32le 音频块。"""
|
||||
# 鉴权校验(服务配置了 API Key 时)
|
||||
auth_error = _check_api_key(request)
|
||||
if auth_error:
|
||||
return JSONResponse(status_code=401, content={"detail": auth_error})
|
||||
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
form = await request.form()
|
||||
body = dict(form)
|
||||
|
||||
# 模型名校验(服务配置了模型名时)
|
||||
model_error = _check_model_name(body)
|
||||
if model_error:
|
||||
return JSONResponse(status_code=400, content={"detail": model_error})
|
||||
|
||||
text = (body.get("text") or "").strip()
|
||||
request_id = body.get("request_id") or f"tts-{int(time.time() * 1000)}"
|
||||
|
||||
@@ -317,9 +360,15 @@ if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="VoxCPM Streaming TTS Server")
|
||||
parser.add_argument("--port", type=int, default=DEFAULT_PORT, help=f"服务端口(默认 {DEFAULT_PORT})")
|
||||
parser.add_argument("--model-path", type=str, default=DEFAULT_MODEL_PATH, help="VoxCPM 模型目录")
|
||||
parser.add_argument("--model-name", type=str, default=DEFAULT_MODEL_NAME,
|
||||
help=f"服务模型名(校验请求 body.model;默认 {DEFAULT_MODEL_NAME},留空则不校验)")
|
||||
parser.add_argument("--api-key", type=str, default=DEFAULT_API_KEY,
|
||||
help="API Key(校验 Authorization: Bearer <key> 或 X-API-Key;默认空=任意值放行)")
|
||||
args = parser.parse_args()
|
||||
|
||||
_model_args["model_path"] = args.model_path
|
||||
_model_args["model_name"] = args.model_name
|
||||
_model_args["api_key"] = args.api_key
|
||||
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=args.port)
|
||||
Reference in new issue
Block a user