initial open-source release

llm_server/README.md

@@ -0,0 +1,184 @@
+# LLM 推理服务
+
+为 [minor Agent](../README.md) 提供本地化推理后端，包含 **6 个独立服务**，均跑在 GPU 服务器上，通过 OpenAI 兼容 / 自定义 HTTP API 暴露。可整体部署，也可按需启用。
+
+## 服务总览
+
+| 服务 | 端口 | 模型 | 启动脚本 | Python 服务端 | 关键端点 |
+|------|:----:|------|----------|---------------|----------|
+| **LLM** | 8900 | Qwen3.6-35B-A3B-FP8 | `start_llm.sh` | vLLM 内置 | `/v1/chat/completions` |
+| **ASR** | 8901 | Qwen3-ASR-1.7B | `start_asr.sh` | vLLM 内置 | `/v1/chat/completions` |
+| **TTS** | 8902 | VoxCPM 1.5 / 2 | `start_streaming_tts.sh` | `streaming_tts_server.py` | `/stream_tts` |
+| **RAG** | 8903 | Embedding-0.6B + Reranker-4B | `start_rag_server.sh` | `rag_server.py` | `/embed` `/rerank` |
+| **Image Gen** | 8904 | Z-Image-Turbo | `start_image_gen.sh` | `image_gen_server.py` | `/generate` |
+
+> 所有脚本 / Python 文件均通过 `os.path.dirname(__file__)` 与 `${SCRIPT_DIR}` 动态定位模型路径，**目录整体迁移无需改代码**。
+
+---
+
+## 一、部署流程
+
+### 1. 上传到服务器
+
+```bash
+scp -P <SSH_PORT> -r Agent/llm_server/* <user>@<your-gpu-server>:~/llm_server/
+ssh -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -p <SSH_PORT> <user>@<your-gpu-server>
+```
+
+### 2. 服务器环境
+
+```bash
+cd ~/llm_server
+chmod +x *.sh
+
+# HuggingFace 镜像加速（国内必需）
+export HF_ENDPOINT=https://hf-mirror.com
+export HF_HUB_ENABLE_XET=0
+echo 'export HF_ENDPOINT=https://hf-mirror.com' >> ~/.bashrc
+echo 'export HF_HUB_ENABLE_XET=0' >> ~/.bashrc
+
+python3 -m venv myenv && source myenv/bin/activate
+
+# 一次性安装全部依赖
+pip install vllm modelscope transformers fastapi uvicorn torch numpy torchaudio soundfile \
+            bitsandbytes huggingface_hub voxcpm diffusers
+sudo apt update && sudo apt install -y ffmpeg libavcodec-dev libavformat-dev libavutil-dev \
+            libavdevice-dev libavfilter-dev libswscale-dev libswresample-dev
+```
+
+### 3. 下载模型
+
+所有模型统一存放 `~/llm_server/models/`：
+
+| 模型 | 命令 |
+|------|------|
+| Qwen3.6-35B-A3B-FP8 (LLM) | `modelscope download --model Qwen/Qwen3.6-35B-A3B-FP8 --local_dir ./models/Qwen/Qwen3___6-35B-A3B-FP8` |
+| Qwen3-ASR-1.7B | `modelscope download --model Qwen/Qwen3-ASR-1.7B --local_dir ./models/Qwen/Qwen3-ASR-1.7B` |
+| Qwen3-Embedding-0.6B | `modelscope download --model Qwen/Qwen3-Embedding-0.6B --local_dir ./models/Qwen/Qwen3-Embedding-0___6B` |
+| Qwen3-Reranker-4B | `modelscope download --model Qwen/Qwen3-Reranker-4B --local_dir ./models/Qwen/Qwen3-Reranker-4B` |
+| VoxCPM 1.5 (推荐) | `modelscope download --model OpenBMB/VoxCPM1.5 --local_dir ./models/VoxCPM1__5` |
+| VoxCPM 2 (音质更佳) | `modelscope download --model OpenBMB/VoxCPM2 --local_dir ./models/VoxCPM2` |
+| Z-Image-Turbo | `modelscope download --model Tongyi-MAI/Z-Image-Turbo --local_dir ./models/Tongyi-MAI/Z-Image-Turbo` |
+
+完成后目录结构：
+
+```
+~/llm_server/models/
+├── Qwen/
+│   ├── Qwen3___6-35B-A3B-FP8/      # LLM
+│   ├── Qwen3-ASR-1.7B/             # ASR
+│   ├── Qwen3-Embedding-0___6B/     # Embedding
+│   └── Qwen3-Reranker-4B/          # Reranker
+├── VoxCPM1__5/                     # TTS
+└── Tongyi-MAI/Z-Image-Turbo/       # Image Gen
+```
+
+---
+
+## 二、启动与测试
+
+每个服务独立启动，互不依赖。所有 `start_*.sh` 支持环境变量覆盖：
+
+| 环境变量 | 作用 | 适用脚本 |
+|----------|------|----------|
+| `HOST_PORT` | 服务端口 | 全部 |
+| `LOCAL_MODEL_DIR` / `MODEL_PATH` / `VOXCPM_MODEL_PATH` | 模型目录 | 全部 |
+| `CUDA_VISIBLE_DEVICES` | GPU 设备号 | 全部 |
+| `SPECIFIC_GPU` | 单 GPU 指定（写 `CUDA_VISIBLE_DEVICES`） | image_gen / tts |
+
+### LLM · 8900
+
+```bash
+bash start_llm.sh
+# 测试
+python3 test_llm.py --url http://127.0.0.1:8900            # 非流式
+python3 test_llm.py --url http://127.0.0.1:8900 --stream   # 流式
+```
+
+> vLLM 加载 Qwen3.6-35B-A3B-FP8，OpenAI 兼容 API，使用 `chat_template.jinja` 格式化对话。
+
+### ASR · 8901
+
+```bash
+bash start_asr.sh
+# 测试
+python3 test_asr.py --audio /path/to/audio.wav --url http://127.0.0.1:8901/v1/chat/completions
+```
+
+### TTS · 8902（流式）
+
+```bash
+bash start_streaming_tts.sh
+# 测试（流式返回 wav 分块）
+curl -X POST http://127.0.0.1:8902/stream_tts \
+  -H "Content-Type: application/json" \
+  -d '{"text": "你好，这是一个流式语音合成测试。", "request_id": "test-1"}'
+```
+
+> VoxCPM 流式合成，边生成边返回 wav chunk，配合 `reference_audio.wav` 控制音色。
+
+### RAG · 8903（Embedding + Reranker）
+
+```bash
+bash start_rag_server.sh
+python3 test_rag.py
+```
+
+### Image Gen · 8904
+
+```bash
+LOCAL_MODEL_DIR=./models/Tongyi-MAI/Z-Image-Turbo SPECIFIC_GPU=3 bash start_image_gen.sh
+# 测试
+curl -X POST http://127.0.0.1:8904/generate \
+  -H "Content-Type: application/json" \
+  -d '{"prompt": "一只橘猫坐在窗台上，阳光洒落", "width": 1024, "height": 1024, "return_format": "base64"}' \
+  | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'成功:{d.get(\"success\")} 尺寸:{d.get(\"width\")}x{d.get(\"height\")} 耗时:{d.get(\"elapsed_seconds\",\"?\")}s')"
+```
+
+> Z-Image-Turbo 基于 Diffusers，**9 步推理**生成 1024×1024 图片，`return_format=base64` 直接返回内联数据。
+
+---
+
+## 三、本地端口映射（开发调试）
+
+将远程端口映射到本地，用 `http://localhost:8900` 直连远程服务：
+
+```bash
+# 一次性映射全部端口
+ssh -fN -o ServerAliveInterval=60 -o ServerAliveCountMax=3 \
+  -L 8900:0.0.0.0:8900 -L 8901:0.0.0.0:8901 -L 8902:0.0.0.0:8902 \
+  -L 8903:0.0.0.0:8903 -L 8904:0.0.0.0:8904 \
+  <user>@<your-gpu-server> -p <SSH_PORT>
+```
+
+验证映射（HTTP 200 即通）：
+
+```bash
+for p in 8900 8901 8902 8903 8904; do
+  curl -o /dev/null -s -w "$p: %{http_code}\n" http://localhost:$p/health
+done
+```
+
+---
+
+## 四、停止服务
+
+```bash
+for p in 8900 8901 8902 8903 8904; do kill $(lsof -t -i:$p); done
+```
+
+---
+
+## 五、接入 Agent
+
+在 minor Agent 配置向导（或 `env_config.json`）中填入对应地址即可启用各能力：
+
+| Agent 能力 | 配置字段 | 示例值 |
+|-----------|----------|--------|
+| LLM 对话 | `models[].base_url` | `http://localhost:8900/v1` |
+| 语音识别 | `ASR_BASE_URL` | `http://localhost:8901/v1` |
+| 语音合成 | `STREAMING_TTS_URL` | `http://localhost:8902` |
+| 知识库检索 | `RAG_BASE_URL` | `http://localhost:8903` |
+| 图像生成 | （由 `image_gen` 工具读取） | `http://localhost:8904` |
+
+> 💡 不必自建本服务，Agent 也支持任何 **OpenAI 兼容 API**（云端 Qwen / DeepSeek / OpenAI 等），填入 `base_url` + `api_key` 即可。

llm_server/chat_template.jinja

@@ -0,0 +1,123 @@
+{%- set enable_thinking = enable_thinking | default(false) %}
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {{- "<|vision_start|><|image_pad|><|vision_end|>" }}
+                {%- elif content.type == 'audio' or 'audio' in content or 'audio_url' in content %}
+                    {{- "<|audio_start|><|audio_pad|><|audio_end|>" }}
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {{- "<|vision_start|><|video_pad|><|vision_end|>" }}
+                {%- elif content.type == 'text' %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+    {{- '\n\n' }}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {{- '<|im_start|>system\n' +"<|vision_start|><|image_pad|><|vision_end|>"+ '<|im_end|>\n' }}
+                {%- elif content.type == 'audio' or 'audio' in content or 'audio_url' in content %}
+                    {{- '<|im_start|>system\n' +"<|audio_start|><|audio_pad|><|audio_end|>"+ '<|im_end|>\n' }}
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {{- '<|im_start|>system\n' +"<|vision_start|><|video_pad|><|vision_end|>"+ '<|im_end|>\n' }}
+                {%- elif content.type == 'text' %}
+                    {{- '<|im_start|>system\n' +content.text+ '<|im_end|>\n' }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = namespace(text="") %}
+        {%- for mcontent in message.content %}
+            {%- if mcontent.type == 'image' or 'image' in mcontent or 'image_url' in mcontent %}
+                {%- set content.text = content.text~"<|vision_start|><|image_pad|><|vision_end|>" %}
+            {%- elif mcontent.type == 'audio' or 'audio' in mcontent or 'audio_url' in mcontent %}
+                {%- set content.text = content.text~"<|audio_start|><|audio_pad|><|audio_end|>" %}
+            {%- elif mcontent.type == 'video' or 'video' in mcontent %}
+                {%- set content.text = content.text~"<|vision_start|><|video_pad|><|vision_end|>" %}
+            {%- elif mcontent.type == 'text' %}
+                {%- set content.text = content.text~mcontent.text %}
+            {%- endif %}
+        {%- endfor %}
+        {%- set content = content.text %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = "" %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+            {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+            {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+        {%- endif %}
+    {%- endif %}
+    {%- if loop.index0 > ns.last_query_index %}
+        {%- if loop.last or (not loop.last and reasoning_content) %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip("\n") + '\n</think>\n\n' + content.lstrip('\n') }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+    {%- else %}
+        {{- '<|im_start|>' + message.role + '\n' + content }}
+    {%- endif %}
+    {%- if message.tool_calls %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if (loop.first and content) or (not loop.first) %}{{- '\n' }}{%- endif %}
+            {%- if tool_call.function %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {%- if tool_call.arguments is string %}
+                {{- tool_call.arguments }}
+            {%- else %}
+                {{- tool_call.arguments | tojson }}
+            {%- endif %}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}{{- '<|im_start|>user' }}{%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}{{- '<|im_end|>\n' }}{%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}{{- '<think>\n\n</think>\n\n' }}{%- endif %}
+{%- endif %}

llm_server/image_gen_server.py

@@ -0,0 +1,207 @@
+#!/usr/bin/env python3
+"""
+Z-Image-Turbo 图像生成服务 —— FastAPI 封装。
+
+使用 Tongyi-MAI/Z-Image-Turbo 模型（基于 Diffusers），默认 1024×1024 输出。
+
+启动方式:
+    python image_gen_server.py --port 8904
+"""
+
+import os
+import sys
+import json
+import base64
+import time
+import argparse
+import warnings
+import threading
+from contextlib import asynccontextmanager
+from pathlib import Path
+from io import BytesIO
+
+import torch
+
+# ======================== 配置区域 ========================
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+DEFAULT_PORT = 8904
+DEFAULT_MODEL_ID = "Tongyi-MAI/Z-Image-Turbo"
+
+# 本地模型路径：优先读环境变量 LOCAL_MODEL_DIR，未设置则回退到远程 ID
+LOCAL_MODEL_DIR = os.environ.get("LOCAL_MODEL_DIR", "")
+MODEL_PATH = LOCAL_MODEL_DIR if LOCAL_MODEL_DIR and os.path.isdir(LOCAL_MODEL_DIR) else DEFAULT_MODEL_ID
+
+# GPU 选择：CUDA_VISIBLE_DEVICES 或环境变量 SPECIFIC_GPU
+SPECIFIC_GPU = os.environ.get("SPECIFIC_GPU", "")
+if SPECIFIC_GPU:
+    os.environ["CUDA_VISIBLE_DEVICES"] = SPECIFIC_GPU
+    print(f"[Z-Image] 已设置 CUDA_VISIBLE_DEVICES={SPECIFIC_GPU}")
+
+# 兼容性：将前导 warnings 移到 torch 导入后
+warnings.filterwarnings("ignore", category=FutureWarning)
+
+
+# ======================== FastAPI ========================
+from fastapi import FastAPI, Request, HTTPException
+from fastapi.responses import JSONResponse, Response
+
+# Lazy-loaded pipeline singleton
+_pipe = None
+_pipe_lock = threading.Lock()
+_device: str | None = None
+
+
+def _get_device() -> str:
+    if torch.cuda.is_available():
+        return "cuda"
+    return "cpu"
+
+
+def _get_pipe():
+    """延迟加载 Z-Image pipeline（线程安全）。
+    使用 Diffusers DiffusionPipeline + trust_remote_code 加载模型仓库中的自定义 ZImagePipeline。
+    """
+    global _pipe, _device
+    if _pipe is not None:
+        return _pipe, _device
+
+    with _pipe_lock:
+        if _pipe is not None:
+            return _pipe, _device
+
+        from diffusers import DiffusionPipeline
+
+        _device = _get_device()
+        load_kwargs = {
+            "torch_dtype": torch.bfloat16 if _device == "cuda" else torch.float32,
+            "low_cpu_mem_usage": False,
+            "trust_remote_code": True,
+        }
+        # 本地路径时跳过在线检查
+        if LOCAL_MODEL_DIR and os.path.isdir(LOCAL_MODEL_DIR):
+            load_kwargs["local_files_only"] = True
+
+        print(f"[Z-Image] 加载模型: {MODEL_PATH}，设备: {_device}")
+        _pipe = DiffusionPipeline.from_pretrained(MODEL_PATH, **load_kwargs)
+        _pipe.to(_device)
+        print(f"[Z-Image] 模型加载成功")
+        return _pipe, _device
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    print(f"[Z-Image] 服务启动中，端口: {_model_args.get('port', DEFAULT_PORT)}")
+    # 启动时预加载模型
+    try:
+        _get_pipe()
+    except Exception as e:
+        print(f"[Z-Image] 模型预加载失败（将在首次请求时重试）: {e}")
+    yield
+    print("[Z-Image] 服务关闭")
+
+
+app = FastAPI(title="Z-Image-Turbo Gen", lifespan=lifespan)
+
+
+@app.post("/generate")
+async def generate_image(request: Request):
+    """生成图像。
+
+    输入 JSON:
+        prompt: str            — 必填，正向提示词
+        negative_prompt: str   — 可选，负向提示词
+        height: int            — 可选，输出高度（默认 1024，须为 64 的倍数）
+        width: int             — 可选，输出宽度（默认 1024，须为 64 的倍数）
+        num_inference_steps: int — 可选，推理步数（Turbo 模型建议 9，即 8 次 DiT forward）
+        guidance_scale: float  — 可选，CFG（Turbo 模型建议 0.0）
+        seed: int              — 可选，随机种子（-1 表示随机）
+        return_format: str     — 可选，"base64"（默认）或 "url"
+
+    输出 JSON:
+        success: bool
+        image: str             — base64 编码的 PNG 或文件路径
+        format: str            — "png"
+        width: int
+        height: int
+    """
+    try:
+        body = await request.json()
+    except Exception:
+        raise HTTPException(status_code=400, detail="请求体必须为 JSON")
+
+    prompt = (body.get("prompt") or "").strip()
+    if not prompt:
+        raise HTTPException(status_code=400, detail="prompt 不能为空")
+
+    height = int(body.get("height", 1024))
+    width = int(body.get("width", 1024))
+    steps = int(body.get("num_inference_steps", 11))
+    guidance = float(body.get("guidance_scale", 0.0))
+    seed = int(body.get("seed", -1))
+    negative_prompt = (body.get("negative_prompt") or "").strip() or None
+    return_format = (body.get("return_format") or "base64").strip().lower()
+
+    # 参数校验
+    if height % 64 != 0 or width % 64 != 0:
+        raise HTTPException(status_code=400, detail="height 和 width 须为 64 的整数倍")
+    if height < 256 or width < 256 or height > 2048 or width > 2048:
+        raise HTTPException(status_code=400, detail="尺寸须在 [256, 2048] 之间")
+
+    pipe, device = _get_pipe()
+
+    generator = None
+    if seed >= 0:
+        generator = torch.Generator(device).manual_seed(seed)
+
+    # 生成
+    print(f"[Z-Image] 生成中: {width}x{height}, steps={steps}, prompt={prompt[:80]}...")
+    t0 = time.time()
+    result = pipe(
+        prompt=prompt,
+        negative_prompt=negative_prompt,
+        height=height,
+        width=width,
+        num_inference_steps=steps,
+        guidance_scale=guidance,
+        generator=generator,
+    )
+    image = result.images[0]
+    elapsed = time.time() - t0
+    print(f"[Z-Image] 生成完成，耗时 {elapsed:.1f}s")
+
+    # 编码输出
+    buf = BytesIO()
+    image.save(buf, format="PNG")
+    buf.seek(0)
+
+    if return_format == "base64":
+        img_b64 = base64.b64encode(buf.read()).decode("ascii")
+        return JSONResponse({
+            "success": True,
+            "image": img_b64,
+            "format": "png",
+            "width": width,
+            "height": height,
+            "elapsed_seconds": round(elapsed, 2),
+        })
+    else:
+        return Response(content=buf.getvalue(), media_type="image/png")
+
+
+@app.get("/health")
+async def health():
+    return {"status": "ok"}
+
+
+# ======================== 启动入口 ========================
+_model_args: dict = {}
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Z-Image-Turbo Image Generation Server")
+    parser.add_argument("--port", type=int, default=DEFAULT_PORT, help=f"服务端口（默认 {DEFAULT_PORT}）")
+    args = parser.parse_args()
+
+    _model_args["port"] = args.port
+
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=args.port)

llm_server/rag_server.py

@@ -0,0 +1,168 @@
+import os
+import torch
+import torch.nn.functional as F
+from fastapi import FastAPI
+from pydantic import BaseModel
+from typing import List
+from modelscope import AutoTokenizer, AutoModel, AutoModelForCausalLM
+from transformers import BitsAndBytesConfig
+
+# 自动获取脚本所在目录作为基础路径
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+
+# 在 load_models() 里添加量化配置
+quantization_config = BitsAndBytesConfig(load_in_8bit=True)
+# ===================== 模型路径 =====================
+EMBED_MODEL_PATH = os.path.join(BASE_DIR, "models", "Qwen", "Qwen3-Embedding-0___6B")
+RERANK_MODEL_PATH = os.path.join(BASE_DIR, "models", "Qwen", "Qwen3-Reranker-4B")
+
+# ===================== 初始化 FastAPI =====================
+app = FastAPI(title="RAG Backend with 8-bit Qwen3 Models")
+
+# ===================== 全局变量（模型、分词器） =====================
+embed_tokenizer = None
+embed_model = None
+rerank_tokenizer = None
+rerank_model = None
+
+# ===================== 工具函数 =====================
+def last_token_pool(last_hidden_states, attention_mask):
+    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
+    if left_padding:
+        return last_hidden_states[:, -1]
+    else:
+        sequence_lengths = attention_mask.sum(dim=1) - 1
+        batch_size = last_hidden_states.shape[0]
+        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]
+
+def get_detailed_instruct(task_description: str, query: str) -> str:
+    return f'Instruct: {task_description}\nQuery:{query}'
+
+def format_rerank_instruction(instruction, query, doc):
+    if instruction is None:
+        instruction = 'Given a web search query, retrieve relevant passages that answer the query'
+    return f"<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}"
+
+# ===================== 启动时加载模型（8‑bit） =====================
+@app.on_event("startup")
+def load_models():
+    global embed_tokenizer, embed_model, rerank_tokenizer, rerank_model
+
+    # ---------- 加载 Embedding 模型 ----------
+    print("Loading Embedding model with 16-bit quantization...")
+    embed_tokenizer = AutoTokenizer.from_pretrained(EMBED_MODEL_PATH, padding_side='left')
+    embed_model = AutoModel.from_pretrained(
+        EMBED_MODEL_PATH,
+        #load_in_8bit=True,                # 启用 8-bit 量化
+        device_map="auto",                # 自动分配到 GPU
+        dtype=torch.float16         # 计算精度保持 fp16
+    )
+    embed_model.eval()
+    print("Embedding model loaded.")
+
+    # ---------- 加载 Reranker 模型 ----------
+    print("Loading Reranker model with 8-bit quantization...")
+    rerank_tokenizer = AutoTokenizer.from_pretrained(RERANK_MODEL_PATH, padding_side='left')
+    rerank_model = AutoModelForCausalLM.from_pretrained(
+        RERANK_MODEL_PATH,
+        quantization_config=quantization_config,
+        device_map="auto",
+        dtype=torch.float16
+    )
+    rerank_model.eval()
+
+    # 预计算 Reranker 所需的固定 tokens
+    global token_true_id, token_false_id, prefix_tokens, suffix_tokens, max_rerank_length
+    token_false_id = rerank_tokenizer.convert_tokens_to_ids("no")
+    token_true_id  = rerank_tokenizer.convert_tokens_to_ids("yes")
+    max_rerank_length = 32768
+
+    prefix = "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n"
+    suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
+    prefix_tokens = rerank_tokenizer.encode(prefix, add_special_tokens=False)
+    suffix_tokens = rerank_tokenizer.encode(suffix, add_special_tokens=False)
+
+    print("Reranker model loaded.")
+
+# ===================== API 请求体定义 =====================
+class EmbedRequest(BaseModel):
+    task_description: str = "Given a web search query, retrieve relevant passages that answer the query"
+    queries: List[str]
+    documents: List[str]           # 可选，若只查询文本可只传 queries
+
+class EmbedResponse(BaseModel):
+    query_embeddings: List[List[float]]
+    doc_embeddings: List[List[float]]   # 当传入 documents 时返回
+
+class RerankRequest(BaseModel):
+    task_description: str = "Given a web search query, retrieve relevant passages that answer the query"
+    query: str
+    documents: List[str]
+
+class RerankResponse(BaseModel):
+    scores: List[float]
+
+# ===================== Embedding 端点 =====================
+@app.post("/embed", response_model=EmbedResponse)
+async def get_embeddings(req: EmbedRequest):
+    # 为 query 加上指令，document 不加指令
+    instructed_queries = [get_detailed_instruct(req.task_description, q) for q in req.queries]
+    # 所有文本拼成一个 batch：先是 queries，后是 documents（如果有）
+    input_texts = instructed_queries + req.documents
+
+    # 分词
+    batch_dict = embed_tokenizer(
+        input_texts,
+        padding=True,
+        truncation=True,
+        return_tensors="pt",
+    ).to(embed_model.device)
+
+    with torch.no_grad():
+        outputs = embed_model(**batch_dict)
+        embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
+        embeddings = F.normalize(embeddings, p=2, dim=1)
+
+    # 拆分 query 和 document 的嵌入
+    nq = len(instructed_queries)
+    query_embs = embeddings[:nq].cpu().tolist()
+    doc_embs = embeddings[nq:].cpu().tolist() if req.documents else []
+
+    return EmbedResponse(query_embeddings=query_embs, doc_embeddings=doc_embs)
+
+# ===================== Reranker 端点 =====================
+@app.post("/rerank", response_model=RerankResponse)
+async def rerank_documents(req: RerankRequest):
+    # 构造带格式的输入对
+    pairs = [
+        format_rerank_instruction(req.task_description, req.query, doc)
+        for doc in req.documents
+    ]
+
+    # 处理输入，添加 prefix/suffix tokens 并 pad
+    inputs = rerank_tokenizer(
+        pairs,
+        padding=False,
+        truncation='longest_first',
+        return_attention_mask=False,
+        max_length=max_rerank_length - len(prefix_tokens) - len(suffix_tokens)
+    )
+    for i, ele in enumerate(inputs['input_ids']):
+        inputs['input_ids'][i] = prefix_tokens + ele + suffix_tokens
+    inputs = rerank_tokenizer.pad(inputs, padding=True, return_tensors="pt", max_length=max_rerank_length)
+    inputs = {k: v.to(rerank_model.device) for k, v in inputs.items()}
+
+    with torch.no_grad():
+        batch_scores = rerank_model(**inputs).logits[:, -1, :]
+        true_vector = batch_scores[:, token_true_id]
+        false_vector = batch_scores[:, token_false_id]
+        scores = torch.stack([false_vector, true_vector], dim=1)
+        scores = torch.nn.functional.log_softmax(scores, dim=1)
+        scores = scores[:, 1].exp().cpu().tolist()
+
+    return RerankResponse(scores=scores)
+
+# ===================== 健康检查 =====================
+@app.get("/health")
+def health():
+    return {"status": "ok"}

llm_server/start_asr.sh

@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# 自动获取脚本所在目录
+SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VENV_DIR="${SCRIPT_DIR}/myenv"
+MODEL_DIR="${SCRIPT_DIR}/models/Qwen/Qwen3-ASR-1.7B"
+HOST_PORT="${HOST_PORT:-8901}"
+
+# 激活虚拟环境
+source "${VENV_DIR}/bin/activate"
+
+# 检查端口占用
+if lsof -i "tcp:${HOST_PORT}" -t >/dev/null 2>&1; then
+  echo "Port ${HOST_PORT} already in use" >&2
+  exit 1
+fi
+
+echo "Starting vLLM server on port ${HOST_PORT} ..."
+echo "Model path: ${MODEL_DIR}"
+
+# 进入脚本目录
+cd "${SCRIPT_DIR}"
+
+# 设置环境变量
+export VLLM_USE_MODELSCOPE=true
+export MODELSCOPE_DOWNLOAD_MODE=local
+export MODELSCOPE_HUB_ENABLE_FILE_CHECK=false
+export MODELSCOPE_CACHE="${SCRIPT_DIR}/models"
+export HF_HUB_OFFLINE=1
+export TRANSFORMERS_OFFLINE=1
+
+# 固定使用 GPU 0（多 GPU 服务器上所有模型共用一张卡）
+export CUDA_VISIBLE_DEVICES=3
+
+# 启动 vLLM 服务
+vllm serve "${MODEL_DIR}" \
+  --port "${HOST_PORT}" \
+  --host 0.0.0.0 \
+  --quantization fp8 \
+  --dtype auto \
+  --kv-cache-dtype fp8 \
+  --max-model-len 16384 \
+  --allowed-local-media-path / \
+  --tensor-parallel-size 1 \
+  --gpu-memory-utilization 0.15 \
+  --max-num-batched-tokens 4096

llm_server/start_image_gen.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# ---------- 配置 ----------
+HOST_PORT="${HOST_PORT:-8904}"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# ---------- Python 虚拟环境 ----------
+if [ -f "${SCRIPT_DIR}/../myenv/bin/activate" ]; then
+  source "${SCRIPT_DIR}/../myenv/bin/activate"
+elif [ -f "${SCRIPT_DIR}/../../myenv/bin/activate" ]; then
+  source "${SCRIPT_DIR}/../../myenv/bin/activate"
+fi
+
+# ---------- 可选 CUDA 设备隔离 ----------
+CUDA_DEV="${CUDA_VISIBLE_DEVICES:-}"
+if [ -n "${CUDA_DEV}" ]; then
+  export CUDA_VISIBLE_DEVICES="${CUDA_DEV}"
+  echo "[Z-Image] CUDA_VISIBLE_DEVICES=${CUDA_DEV}"
+fi
+
+echo "[Z-Image] 启动图像生成服务 (端口=${HOST_PORT})"
+cd "${SCRIPT_DIR}"
+exec python image_gen_server.py --port "${HOST_PORT}"

llm_server/start_llm.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VENV_DIR="${SCRIPT_DIR}/myenv"
+HOST_PORT="${HOST_PORT:-8900}"
+MODEL_DIR="${SCRIPT_DIR}/models/Qwen/Qwen3___6-35B-A3B-FP8"
+
+# 激活虚拟环境
+source "${VENV_DIR}/bin/activate"
+
+# 检查端口占用
+if lsof -i "tcp:${HOST_PORT}" -t >/dev/null 2>&1; then
+  echo "Port ${HOST_PORT} already in use" >&2
+  exit 1
+fi
+
+echo "Starting vLLM server on port ${HOST_PORT} ..."
+echo "Model path: ${MODEL_DIR}"
+
+# 设置环境变量
+export VLLM_USE_MODELSCOPE=true
+export MODELSCOPE_DOWNLOAD_MODE=local
+export MODELSCOPE_HUB_ENABLE_FILE_CHECK=false
+export MODELSCOPE_CACHE="${SCRIPT_DIR}/models"
+export HF_HUB_OFFLINE=1
+export TRANSFORMERS_OFFLINE=1
+export CUDA_VISIBLE_DEVICES=3
+
+# 启动 vLLM 服务
+vllm serve "${MODEL_DIR}" \
+  --port "${HOST_PORT}" \
+  --host 0.0.0.0 \
+  --tensor-parallel-size 1 \
+  --max-model-len 262144 \
+  --reasoning-parser qwen3 \
+  --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder \
+  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
+  --enable-prefix-caching \
+  --max_num_batched_tokens 4096 \
+  --max_num_seqs 8 \
+  --kv-cache-dtype fp8_e4m3 \
+  --chat-template "${SCRIPT_DIR}/chat_template.jinja" \
+  --safetensors-load-strategy eager \
+  --served-model-name Qwen/Qwen3.6-35B-A3B-FP8

llm_server/start_rag_server.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# 自动获取脚本所在目录
+SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VENV_DIR="${VENV_DIR:-${SCRIPT_DIR}/myenv}"
+HOST_PORT="${HOST_PORT:-8903}"
+
+# 激活虚拟环境
+source "${VENV_DIR}/bin/activate"
+
+# 检查端口占用
+if lsof -i "tcp:${HOST_PORT}" -t >/dev/null 2>&1; then
+  echo "Port ${HOST_PORT} already in use" >&2
+  exit 1
+fi
+
+echo "Starting Qwen3 RAG server on port ${HOST_PORT} ..."
+
+# 进入脚本目录
+cd "${SCRIPT_DIR}"
+
+# 离线模式设置（确保已下载模型，不从远程拉取）
+export MODELSCOPE_DOWNLOAD_MODE=local
+export MODELSCOPE_HUB_ENABLE_FILE_CHECK=false
+export HF_HUB_OFFLINE=1
+export TRANSFORMERS_OFFLINE=1
+
+# 指定 modelscope 的缓存目录（如果你的模型保存在此路径下）
+# 根据你的下载路径调整，或将其设置为环境变量覆盖
+export MODELSCOPE_CACHE="${MODELSCOPE_CACHE:-${SCRIPT_DIR}/models}"
+
+# 固定使用 GPU 0（多 GPU 服务器上所有模型共用一张卡）
+export CUDA_VISIBLE_DEVICES=3
+
+# 启动 FastAPI 服务
+uvicorn rag_server:app \
+  --host 0.0.0.0 \
+  --port "${HOST_PORT}" \
+  --workers 1

llm_server/start_streaming_tts.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# ---------- 配置 ----------
+HOST_PORT="${HOST_PORT:-8902}"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# VoxCPM 模型路径（可覆盖）
+VOXCPM_MODEL_PATH="${VOXCPM_MODEL_PATH:-${SCRIPT_DIR}/models/VoxCPM1__5}"
+
+# ---------- Python 虚拟环境 ----------
+if [ -f "${SCRIPT_DIR}/../myenv/bin/activate" ]; then
+  source "${SCRIPT_DIR}/../myenv/bin/activate"
+elif [ -f "${SCRIPT_DIR}/../../myenv/bin/activate" ]; then
+  source "${SCRIPT_DIR}/../../myenv/bin/activate"
+fi
+
+# ---------- 可选 CUDA 设备隔离 ----------
+CUDA_DEV="${CUDA_VISIBLE_DEVICES:-3}"
+export CUDA_VISIBLE_DEVICES="${CUDA_DEV}"
+
+echo "[VoxCPM TTS] 启动流式 TTS 服务 (端口=${HOST_PORT}, CUDA=${CUDA_DEV})"
+echo "[VoxCPM TTS] 模型路径: ${VOXCPM_MODEL_PATH}"
+cd "${SCRIPT_DIR}"
+exec python streaming_tts_server.py \
+  --port "${HOST_PORT}" \
+  --model-path "${VOXCPM_MODEL_PATH}"

llm_server/streaming_tts_server.py

@@ -0,0 +1,325 @@
+#!/usr/bin/env python3
+"""
+VoxCPM 流式 TTS 服务 —— 按句分块合成，SSE 流式推送 PCM 音频。
+
+使用 OpenBMB VoxCPM 模型（Whispera 同款），无需 Genie TTS。
+配合前端 Web Audio API 实现边合成边播放，无需保存音频文件。
+
+启动方式:
+    python streaming_tts_server.py --port 8902 --model-path ./models/VoxCPM
+"""
+
+import os
+import sys
+import json
+import base64
+import time
+import threading
+import argparse
+import warnings
+from contextlib import asynccontextmanager
+from pathlib import Path
+
+# 抑制 PyTorch weight_norm 弃用警告（来自 voxcpm 内部）
+warnings.filterwarnings("ignore", message=".*weight_norm.*deprecated.*")
+
+# ======================== 配置区域 ========================
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+
+# 启用 TF32 加速（Ampere+ GPU）
+import torch
+torch.set_float32_matmul_precision("high")
+
+# 确保 torchaudio 已安装（VoxCPM 参考音频需要）
+try:
+    import torchaudio  # noqa: F401
+except ImportError:
+    pass  # 无参考音频时可选
+
+# 默认参数（可通过命令行覆盖）
+DEFAULT_MODEL_PATH = os.path.join(BASE_DIR, "models", "VoxCPM1__5")
+DEFAULT_PORT = 8902
+
+# 参考音频（用于统一音色，同一人物说的一段话 + 对应文本）
+# 如果配置了这对参数，所有合成都会克隆该音色，各句音色保持一致
+DEFAULT_PROMPT_WAV = os.path.join(BASE_DIR, "reference_audio.wav")
+DEFAULT_PROMPT_TEXT = "日子缓缓前行，不必急于追赶，照顾好自己的心情，认真过好当下的每一刻就足够了。"
+
+# 音频质量参数
+DEFAULT_CFG_VALUE = 1.5          # 音色贴合度 (1.0~3.0)，越高越贴近参考音频但可能不自然
+DEFAULT_INFERENCE_STEPS = 15     # 推理步数 (4~30)，越高音质越好但越慢
+# =========================================================
+
+# 请求级取消标志
+_cancel_flags: dict[str, threading.Event] = {}
+_cancel_lock = threading.Lock()
+
+# 当前活跃请求 ID（新请求到达时自动取消旧请求）
+_current_request_id: str | None = None
+_current_req_lock = threading.Lock()
+
+# 模型锁（VoxCPM 只允许单线程使用，新请求取消旧请求后等待锁释放）
+_model_lock = threading.Lock()
+
+
+def set_cancel(request_id: str) -> None:
+    with _cancel_lock:
+        evt = _cancel_flags.get(request_id)
+        if evt is None:
+            evt = threading.Event()
+            _cancel_flags[request_id] = evt
+        evt.set()
+
+
+def is_cancelled(request_id: str) -> bool:
+    with _cancel_lock:
+        evt = _cancel_flags.get(request_id)
+        return evt is not None and evt.is_set()
+
+
+def cancel_current_and_set(new_request_id: str) -> str | None:
+    """取消当前活跃请求并注册新请求 ID，返回被取消的旧请求 ID。"""
+    global _current_request_id
+    with _current_req_lock:
+        old = _current_request_id
+        _current_request_id = new_request_id
+    if old and old != new_request_id:
+        set_cancel(old)
+        # 若旧请求持有模型锁，强制释放以让新请求立即获取
+        try:
+            if _model_lock.locked():
+                _model_lock.release()
+        except RuntimeError:
+            pass
+        print(f"[VoxCPM TTS] 打断旧请求 {old}，开始新请求 {new_request_id}")
+    return old
+
+
+# Lazy-loaded model singleton
+_model = None
+_singleton_lock = threading.Lock()
+_model_sample_rate = 24000
+
+
+def _get_model(model_path: str):
+    """延迟加载 VoxCPM 模型（线程安全）。"""
+    global _model, _model_sample_rate
+    if _model is not None:
+        return _model, _model_sample_rate
+
+    with _singleton_lock:
+        if _model is not None:
+            return _model, _model_sample_rate
+
+        if "voxcpm" not in sys.modules:
+            try:
+                from voxcpm import VoxCPM  # type: ignore
+            except ImportError:
+                raise ImportError(
+                    "无法导入 voxcpm 模块。请确保 Whispera-main 项目的 "
+                    "voxcpm-tts-streaming-module/src 在 sys.path 中，"
+                    "或 pip install 了 voxcpm 包。"
+                )
+
+        from voxcpm import VoxCPM  # type: ignore
+        mp = _resolve_model_path(model_path)
+        print(f"[VoxCPM TTS] 加载模型: {mp}")
+        _model = VoxCPM(
+            voxcpm_model_path=mp,
+            enable_denoiser=False,
+            optimize=True,
+        )
+        _model_sample_rate = int(_model.tts_model.sample_rate)
+        print(f"[VoxCPM TTS] 模型加载成功，采样率: {_model_sample_rate} Hz")
+        return _model, _model_sample_rate
+
+
+def _resolve_model_path(model_path: str) -> str:
+    """解析模型路径：支持多种命名格式。"""
+    base = Path(model_path).expanduser().resolve()
+    if not base.exists():
+        raise FileNotFoundError(f"VoxCPM 模型路径不存在: {base}")
+
+    # 1. 直接就是模型目录（有 config.json）
+    if (base / "config.json").is_file():
+        return str(base)
+
+    # 2. 检查常见子目录名
+    for sub in ["openbmb__VoxCPM2", "openbmb__VoxCPM1.5",
+                "VoxCPM2", "VoxCPM1.5", "VoxCPM1__5"]:
+        candidate = base / sub
+        if candidate.is_dir() and (candidate / "config.json").is_file():
+            return str(candidate)
+
+    raise FileNotFoundError(
+        f"在 {base} 下未找到 VoxCPM 模型（需要包含 config.json 的子目录，"
+        f"如 openbmb__VoxCPM1.5/、VoxCPM1__5/ 等）"
+    )
+
+
+def _build_sse(data: dict) -> str:
+    return f"data: {json.dumps(data, ensure_ascii=False)}\n\n"
+
+
+def _stream_audio_events(text: str, request_id: str, model_path: str):
+    """生成 SSE 音频流事件（同步，在独立线程中运行）。"""
+    import numpy as np
+
+    try:
+        model, sample_rate = _get_model(model_path)
+    except Exception as e:
+        yield _build_sse({"type": "audio.error", "request_id": request_id, "message": str(e)})
+        return
+
+    # 开始
+    yield _build_sse({
+        "type": "audio.start",
+        "request_id": request_id,
+        "sample_rate": sample_rate,
+        "audio_format": "pcm_f32le",
+    })
+
+    chunk_index = 0
+    total_samples = 0
+
+    # 获取模型锁（前一请求已被 cancel_current_and_set 取消，锁应已释放）
+    # 短超时 + 重试，避免用户等待
+    for attempt in range(3):
+        if _model_lock.acquire(timeout=3):
+            break
+        if attempt < 2:
+            print(f"[VoxCPM TTS] 等待模型锁释放... (尝试 {attempt + 1}/3)")
+    else:
+        yield _build_sse({"type": "audio.error", "request_id": request_id,
+                          "message": "等待模型超时，请重试"})
+        return
+    try:
+        gen = model.generate_streaming(
+            text=text,
+            prompt_wav_path=DEFAULT_PROMPT_WAV or None,
+            prompt_text=DEFAULT_PROMPT_TEXT or None,
+            cfg_value=DEFAULT_CFG_VALUE,
+            inference_timesteps=DEFAULT_INFERENCE_STEPS,
+            retry_badcase=False,
+        )
+
+        for audio_chunk in gen:
+            if is_cancelled(request_id):
+                yield _build_sse({"type": "audio.cancelled", "request_id": request_id})
+                return
+
+            arr = np.asarray(audio_chunk, dtype=np.float32).ravel()
+            pcm_bytes = np.ascontiguousarray(arr, dtype=np.float32).tobytes()
+            num_samples = arr.size
+            total_samples += num_samples
+            b64_data = base64.b64encode(pcm_bytes).decode("ascii")
+
+            yield _build_sse({
+                "type": "audio.chunk",
+                "request_id": request_id,
+                "index": chunk_index,
+                "num_samples": num_samples,
+                "sample_rate": sample_rate,
+                "data": b64_data,
+            })
+            chunk_index += 1
+    finally:
+        _model_lock.release()
+
+    yield _build_sse({
+        "type": "audio.done",
+        "request_id": request_id,
+        "total_samples": total_samples,
+        "interrupted": False,
+    })
+
+
+# ======================== FastAPI ========================
+
+from fastapi import FastAPI, Request
+from fastapi.responses import StreamingResponse
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    mp = _model_args.get("model_path", DEFAULT_MODEL_PATH)
+    print(f"[VoxCPM TTS] 服务启动中，模型路径: {mp}")
+    print(f"[VoxCPM TTS] 调用方式: POST /stream_tts body: {{'text': '...', 'request_id': '...'}}")
+    # 启动时预加载模型（避免首次请求等待）
+    try:
+        _get_model(mp)
+    except Exception as e:
+        print(f"[VoxCPM TTS] 模型预加载失败: {e}")
+    yield
+    print("[VoxCPM TTS] 服务关闭")
+
+
+app = FastAPI(title="VoxCPM Streaming TTS", lifespan=lifespan)
+
+
+@app.post("/stream_tts")
+async def stream_tts(request: Request):
+    """流式 TTS 端点：接收文本，SSE 流式返回 PCM f32le 音频块。"""
+    try:
+        body = await request.json()
+    except Exception:
+        form = await request.form()
+        body = dict(form)
+
+    text = (body.get("text") or "").strip()
+    request_id = body.get("request_id") or f"tts-{int(time.time() * 1000)}"
+
+    if not text:
+        return StreamingResponse(
+            iter([_build_sse({"type": "audio.error", "request_id": request_id, "message": "文本为空"})]),
+            media_type="text/event-stream",
+        )
+
+    # 新请求到达：自动取消当前活跃的旧请求
+    cancel_current_and_set(request_id)
+
+    return StreamingResponse(
+        _stream_audio_events(text, request_id, _model_args.get("model_path", DEFAULT_MODEL_PATH)),
+        media_type="text/event-stream",
+        headers={
+            "Cache-Control": "no-cache",
+            "Connection": "keep-alive",
+            "X-Accel-Buffering": "no",
+        },
+    )
+
+
+@app.post("/cancel_tts")
+async def cancel_tts(request: Request):
+    """取消指定 request_id 的 TTS 合成。"""
+    try:
+        body = await request.json()
+    except Exception:
+        form = await request.form()
+        body = dict(form)
+
+    request_id = body.get("request_id", "")
+    if request_id:
+        set_cancel(request_id)
+        return {"status": "cancelled", "request_id": request_id}
+    return {"status": "error", "message": "缺少 request_id"}
+
+
+@app.get("/health")
+async def health():
+    return {"status": "ok"}
+
+
+# ======================== 启动入口 ========================
+_model_args: dict = {}
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="VoxCPM Streaming TTS Server")
+    parser.add_argument("--port", type=int, default=DEFAULT_PORT, help=f"服务端口（默认 {DEFAULT_PORT}）")
+    parser.add_argument("--model-path", type=str, default=DEFAULT_MODEL_PATH, help="VoxCPM 模型目录")
+    args = parser.parse_args()
+
+    _model_args["model_path"] = args.model_path
+
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=args.port)

llm_server/test_asr.py

@@ -0,0 +1,132 @@
+"""
+Test script for Qwen3-Omni-30B-A3B-Captioner via vLLM Chat Completions API.
+
+Note:
+    This model is an audio captioner, not a pure ASR. It generates
+    detailed descriptions of audio content (speech, environment, music, etc.)
+    without any text prompt. Only the audio file is sent.
+    It is recommended to keep audio length <= 30 seconds for best detail.
+"""
+
+import requests
+import argparse
+import os
+import sys
+import base64
+import mimetypes
+
+DEFAULT_URL = "http://127.0.0.1:8901/v1/chat/completions"
+
+# ----------------------------------------------------------------------
+# Optional: check audio duration (requires soundfile)
+# ----------------------------------------------------------------------
+def get_audio_duration_sec(file_path: str) -> float:
+    """Return audio duration in seconds, or -1.0 if unable to read."""
+    try:
+        import soundfile as sf
+        info = sf.info(file_path)
+        return info.duration
+    except Exception:
+        try:
+            # Fallback using librosa (if available)
+            import librosa
+            return librosa.get_duration(path=file_path)
+        except Exception:
+            return -1.0
+
+# ----------------------------------------------------------------------
+def file_to_url(file_path: str, use_base64: bool = False) -> str:
+    """Convert local file path to a URL accepted by vLLM.
+    
+    When the server is remote, use base64 data URL instead of file://
+    since the remote server cannot access local filesystem paths.
+    """
+    if use_base64:
+        mime_type, _ = mimetypes.guess_type(file_path)
+        if mime_type is None:
+            mime_type = "audio/wav"
+        with open(file_path, "rb") as f:
+            b64_data = base64.b64encode(f.read()).decode("utf-8")
+        return f"data:{mime_type};base64,{b64_data}"
+    else:
+        abs_path = os.path.abspath(file_path)
+        return "file://" + abs_path.replace("\\", "/")
+
+def caption_audio(audio_path: str, server_url: str = DEFAULT_URL, use_base64: bool = False) -> str:
+    """
+    Send audio to the Qwen3-Omni-30B-A3B-Captioner and return its description.
+    """
+    if not os.path.exists(audio_path):
+        raise FileNotFoundError(f"Audio file not found: {audio_path}")
+
+    # Check duration (warning only, not blocking)
+    duration = get_audio_duration_sec(audio_path)
+    if duration > 30:
+        print(
+            f"⚠️  Audio length is {duration:.1f}s (>30s). "
+            "The model may lose fine detail. Consider trimming to ≤30s.",
+            file=sys.stderr
+        )
+
+    audio_url = file_to_url(audio_path, use_base64=use_base64)
+
+    # The model expects NO text prompt. Only the audio input is required.
+    payload = {
+        "messages": [
+            {
+                "role": "user",
+                "content": [
+                    {"type": "audio_url", "audio_url": {"url": audio_url}}
+                ]
+            }
+        ]
+    }
+
+    headers = {"Content-Type": "application/json"}
+
+    try:
+        resp = requests.post(server_url, headers=headers, json=payload, timeout=120)
+        resp.raise_for_status()
+        result = resp.json()
+        # Extract assistant reply
+        text = result["choices"][0]["message"]["content"]
+        return text.strip()
+    except requests.exceptions.RequestException as e:
+        print(f"Request failed: {e}", file=sys.stderr)
+        if hasattr(e, 'response') and e.response is not None:
+            print(f"Server response: {e.response.text}", file=sys.stderr)
+        raise
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Test Qwen3-Omni-Captioner (audio captioning, NOT pure ASR)"
+    )
+    parser.add_argument("--audio", "-a", required=True,
+                        help="Path to input audio file (mp3, wav, etc.)")
+    parser.add_argument("--url", "-u", default=DEFAULT_URL,
+                        help="vLLM chat completions endpoint")
+    parser.add_argument("--output", "-o", default=None,
+                        help="Output text file (if not provided, print to stdout)")
+    parser.add_argument("--verbose", "-v", action="store_true",
+                        help="Print detailed request/response info")
+    parser.add_argument("--base64", "-b", action="store_true",
+                        help="Send audio as base64 data URL (use when server is remote)")
+
+    args = parser.parse_args()
+
+    if args.verbose:
+        print(f"Audio file: {args.audio}")
+        print(f"Server URL: {args.url}")
+        print(f"Base64 mode: {args.base64}")
+
+    caption_text = caption_audio(args.audio, args.url, use_base64=args.base64)
+
+    if args.output:
+        with open(args.output, "w", encoding="utf-8") as f:
+            f.write(caption_text)
+        print(f"Caption saved to: {args.output}")
+    else:
+        print(caption_text)
+
+if __name__ == "__main__":
+    main()

llm_server/test_llm.py

@@ -0,0 +1,145 @@
+"""
+测试 Qwen3.6-35B-A3B-FP8 LLM 服务（OpenAI 兼容 API）
+
+使用方法:
+  python test_llm.py                     # 默认地址 http://127.0.0.1:8900
+  python test_llm.py --url http://xxx:8900
+  python test_llm.py --stream            # 流式输出
+"""
+
+import argparse
+import json
+import requests
+
+
+def test_non_stream(base_url: str, model: str):
+    """非流式请求：发送消息，打印完整返回结构"""
+    payload = {
+        "model": model,
+        "messages": [
+            {"role": "user", "content": "你好，请用一句话介绍你自己。"}
+        ],
+        "stream": False,
+    }
+
+    print("\n" + "=" * 80)
+    print("  请求体 (Request)")
+    print("=" * 80)
+    print(json.dumps(payload, indent=2, ensure_ascii=False))
+
+    resp = requests.post(
+        f"{base_url}/v1/chat/completions",
+        json=payload,
+        headers={"Authorization": "Bearer <YOUR_API_KEY>"},
+    )
+
+    print("\n" + "=" * 80)
+    print(f"  响应 (Response)  HTTP {resp.status_code}")
+    print("=" * 80)
+
+    if resp.status_code == 200:
+        data = resp.json()
+        print(json.dumps(data, indent=2, ensure_ascii=False))
+
+        # 提取关键信息
+        print("\n" + "-" * 80)
+        print("  关键信息提取")
+        print("-" * 80)
+        choice = data.get("choices", [{}])[0]
+        message = choice.get("message", {})
+        print(f"  model          : {data.get('model')}")
+        print(f"  finish_reason  : {choice.get('finish_reason')}")
+        print(f"  role           : {message.get('role')}")
+        print(f"  content        : {message.get('content', '')[:200]}...")
+        if message.get("tool_calls"):
+            print(f"  tool_calls     : {json.dumps(message['tool_calls'], ensure_ascii=False)}")
+        usage = data.get("usage", {})
+        if usage:
+            print(f"  prompt_tokens  : {usage.get('prompt_tokens')}")
+            print(f"  completion_tokens: {usage.get('completion_tokens')}")
+            print(f"  total_tokens   : {usage.get('total_tokens')}")
+    else:
+        print(f"Error: {resp.text}")
+
+
+def test_stream(base_url: str, model: str):
+    """流式请求：逐块打印输出"""
+    payload = {
+        "model": model,
+        "messages": [
+            {"role": "user", "content": "你好，请用一句话介绍你自己。"}
+        ],
+        "stream": True,
+    }
+
+    print("\n" + "=" * 80)
+    print("  流式请求 (Stream)")
+    print("=" * 80)
+    print(json.dumps(payload, indent=2, ensure_ascii=False))
+
+    resp = requests.post(
+        f"{base_url}/v1/chat/completions",
+        json=payload,
+        headers={"Authorization": "Bearer <YOUR_API_KEY>"},
+        stream=True,
+    )
+
+    print("\n" + "=" * 80)
+    print(f"  流式响应  HTTP {resp.status_code}")
+    print("=" * 80)
+
+    if resp.status_code != 200:
+        print(f"Error: {resp.text}")
+        return
+
+    full_content = ""
+    for line in resp.iter_lines(decode_unicode=True):
+        if not line:
+            continue
+        if line.startswith("data: "):
+            data_str = line[6:]
+            if data_str == "[DONE]":
+                print("\n\n[DONE]")
+                break
+            try:
+                chunk = json.loads(data_str)
+                delta = chunk.get("choices", [{}])[0].get("delta", {})
+                content = delta.get("content", "")
+                if content:
+                    print(content, end="", flush=True)
+                    full_content += content
+                # 打印每个 chunk 的完整结构（首次和最后一次）
+                if not full_content.strip() or content:
+                    pass  # 避免刷屏，取消注释下面这行可查看每个 chunk
+                    # print(f"\n[chunk] {json.dumps(chunk, ensure_ascii=False)}")
+            except json.JSONDecodeError:
+                print(f"\n[无法解析] {data_str}")
+
+    print("\n\n" + "-" * 80)
+    print(f"  完整内容: {full_content}")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="测试 Qwen3.6 LLM 服务")
+    parser.add_argument("--url", default="http://127.0.0.1:8900",
+                        help="LLM 服务地址 (默认: http://127.0.0.1:8900)")
+    parser.add_argument("--model", default="Qwen/Qwen3.6-35B-A3B-FP8",
+                        help="模型名称")
+    parser.add_argument("--stream", action="store_true",
+                        help="使用流式输出")
+    args = parser.parse_args()
+
+    print("=" * 80)
+    print("  Qwen3.6-35B-A3B-FP8 LLM 测试")
+    print("=" * 80)
+    print(f"  服务地址: {args.url}")
+    print(f"  模型    : {args.model}")
+
+    if args.stream:
+        test_stream(args.url, args.model)
+    else:
+        test_non_stream(args.url, args.model)
+
+
+if __name__ == "__main__":
+    main()

llm_server/test_rag.py

@@ -0,0 +1,82 @@
+import requests
+import json
+
+# 服务地址
+BASE_URL = "http://0.0.0.0:8903"
+
+def print_separator(title):
+    print("\n" + "=" * 80)
+    print(f"  {title}")
+    print("=" * 80)
+
+def test_health():
+    print_separator("Health Check")
+    resp = requests.get(f"{BASE_URL}/health")
+    print(f"Status: {resp.status_code}")
+    print(f"Response: {resp.json()}")
+
+def test_embed():
+    print_separator("Embedding Test")
+    payload = {
+        "task_description": "Given a web search query, retrieve relevant passages that answer the query",
+        "queries": [
+            "What is the capital of France?",
+            "How to train a cat?"
+        ],
+        "documents": [
+            "Paris is the capital of France.",
+            "Cats can be trained using positive reinforcement."
+        ]
+    }
+    print(">>> Request Body:")
+    print(json.dumps(payload, indent=2, ensure_ascii=False))
+
+    resp = requests.post(f"{BASE_URL}/embed", json=payload)
+    print(f"<<< Status: {resp.status_code}")
+    if resp.status_code == 200:
+        data = resp.json()
+        # 只打印向量长度和前面几个值，避免刷屏
+        print(f"<<< Response Keys: {list(data.keys())}")
+        print(f"Number of query embeddings: {len(data['query_embeddings'])}")
+        print(f"Dimension of each query embedding: {len(data['query_embeddings'][0])}")
+        print(f"First query embedding (first 5 values): {data['query_embeddings'][0][:5]}")
+        print(f"Number of doc embeddings: {len(data['doc_embeddings'])}")
+        if data['doc_embeddings']:
+            print(f"Dimension of each doc embedding: {len(data['doc_embeddings'][0])}")
+            print(f"First doc embedding (first 5 values): {data['doc_embeddings'][0][:5]}")
+        else:
+            print("No document embeddings returned.")
+        # 如果需要完整输出，取消下面注释（会打印大量浮点数）
+        # print(f"<<< Full Response:\n{json.dumps(data, indent=2)}")
+    else:
+        print(f"Error: {resp.text}")
+
+def test_rerank():
+    print_separator("Reranker Test")
+    payload = {
+        "task_description": "Given a web search query, retrieve relevant passages that answer the query",
+        "query": "What is the capital of France?",
+        "documents": [
+            "Paris is the capital of France.",
+            "Cats are often difficult to train.",
+            "Lyon is a city in France."
+        ]
+    }
+    print(">>> Request Body:")
+    print(json.dumps(payload, indent=2, ensure_ascii=False))
+
+    resp = requests.post(f"{BASE_URL}/rerank", json=payload)
+    print(f"<<< Status: {resp.status_code}")
+    if resp.status_code == 200:
+        data = resp.json()
+        print(f"<<< Response Keys: {list(data.keys())}")
+        print(f"Scores: {data['scores']}")
+        # 打印完整响应
+        # print(f"<<< Full Response:\n{json.dumps(data, indent=2)}")
+    else:
+        print(f"Error: {resp.text}")
+
+if __name__ == "__main__":
+    test_health()
+    test_embed()
+    test_rerank()

llm_server/test_tts.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+"""
+VoxCPM 流式 TTS 测试脚本。
+
+用法:
+    python test_tts.py                        # 默认测试
+    python test_tts.py --text "你好世界"       # 自定义文本
+    python test_tts.py --url http://x.x.x.x:8902  # 远程服务器
+    python test_tts.py --save test_output.wav # 保存为 WAV 文件
+"""
+
+import argparse
+import base64
+import json
+import struct
+import sys
+import time
+
+import requests
+
+
+def main():
+    parser = argparse.ArgumentParser(description="VoxCPM 流式 TTS 测试")
+    parser.add_argument("--url", default="http://0.0.0.0:8902", help="TTS 服务地址")
+    parser.add_argument("--text", default="你好，这是一个流式语音合成测试。今天天气不错，适合出去散步。",
+                        help="测试文本")
+    parser.add_argument("--save", default="", help="保存 WAV 文件路径（不指定则不保存）")
+    parser.add_argument("--play", action="store_true", help="本地播放（需要 pyaudio）")
+    args = parser.parse_args()
+
+    base_url = args.url.rstrip("/")
+    request_id = f"test-{int(time.time() * 1000)}"
+
+    # 1. 健康检查
+    print(f"→ 健康检查 {base_url}/health ...")
+    try:
+        r = requests.get(f"{base_url}/health", timeout=5)
+        r.raise_for_status()
+        print(f"  状态: {r.json()}")
+    except Exception as e:
+        print(f"  ❌ 无法连接 {base_url}: {e}")
+        sys.exit(1)
+
+    # 2. 流式 TTS 请求
+    print(f"→ 流式 TTS 请求: \"{args.text}\"")
+    t_start = time.perf_counter()
+    sample_rate = 0
+    all_pcm = bytearray()
+    total_samples = 0
+    total_chunks = 0
+
+    try:
+        resp = requests.post(
+            f"{base_url}/stream_tts",
+            json={"text": args.text, "request_id": request_id},
+            stream=True,
+            timeout=300,
+        )
+        resp.raise_for_status()
+
+        for line in resp.iter_lines(decode_unicode=True):
+            if not line or not line.startswith("data: "):
+                continue
+
+            payload = json.loads(line[len("data: "):])
+            event_type = payload.get("type", "")
+
+            if event_type == "audio.start":
+                sample_rate = payload.get("sample_rate", 0)
+                num_segments = payload.get("num_segments", 0)
+                print(f"  audio.start  | 采样率={sample_rate}Hz 句数={num_segments}")
+
+            elif event_type == "audio.chunk":
+                b64_data = payload.get("data", "")
+                idx = payload.get("index", 0)
+                num_s = payload.get("num_samples", 0)
+                if b64_data:
+                    pcm = base64.b64decode(b64_data)
+                    all_pcm.extend(pcm)
+                    total_samples += num_s
+                    total_chunks += 1
+                # 进度提示
+                if idx % 5 == 0 and b64_data:
+                    dur = total_samples / sample_rate if sample_rate else 0
+                    print(f"  audio.chunk  | #{idx} 累计={dur:.1f}s")
+
+            elif event_type == "audio.done":
+                total = payload.get("total_segments", 0)
+                print(f"  audio.done   | 共{total}句 {total_samples}样本 {total_chunks}块")
+
+            elif event_type == "audio.error":
+                print(f"  ❌ audio.error: {payload.get('message', '')}")
+                sys.exit(1)
+
+            elif event_type == "audio.cancelled":
+                print(f"  ⚠ audio.cancelled")
+                break
+
+            elif event_type == "done":
+                pass  # 流结束
+
+    except requests.exceptions.ConnectionError as e:
+        print(f"  ❌ 连接失败: {e}")
+        sys.exit(1)
+    except Exception as e:
+        print(f"  ❌ 错误: {e}")
+        sys.exit(1)
+
+    t_elapsed = time.perf_counter() - t_start
+    duration = total_samples / sample_rate if sample_rate else 0
+    rtf = t_elapsed / duration if duration > 0 else 0
+    print(f"\n{'='*50}")
+    print(f"  总耗时:     {t_elapsed:.2f}s")
+    print(f"  音频时长:   {duration:.2f}s")
+    print(f"  RTF:        {rtf:.2f} (越小越好，<1 表示比实时快)")
+    print(f"  采样率:     {sample_rate} Hz")
+    print(f"  总样本数:   {total_samples}")
+    print(f"  PCM 大小:   {len(all_pcm)} bytes")
+    print(f"{'='*50}")
+
+    if not all_pcm:
+        print("  ⚠ 未收到任何音频数据")
+        sys.exit(1)
+
+    # 3. 保存 WAV
+    if args.save:
+        _save_wav(args.save, all_pcm, sample_rate)
+        print(f"→ 已保存: {args.save}")
+
+    # 4. 本地播放
+    if args.play:
+        _play_pcm(all_pcm, sample_rate)
+
+
+def _save_wav(filepath: str, pcm_f32: bytearray, sample_rate: int):
+    """将 PCM f32le 转为 16-bit WAV 保存。"""
+    import io
+    import wave
+    import numpy as np
+
+    arr = np.frombuffer(pcm_f32, dtype=np.float32)
+    # 转为 int16
+    arr_i16 = (arr * 32767).clip(-32768, 32767).astype(np.int16)
+
+    with wave.open(filepath, "wb") as wf:
+        wf.setnchannels(1)
+        wf.setsampwidth(2)
+        wf.setframerate(sample_rate)
+        wf.writeframes(arr_i16.tobytes())
+
+
+def _play_pcm(pcm_f32: bytearray, sample_rate: int):
+    """通过 pyaudio 播放 PCM 音频。"""
+    try:
+        import pyaudio
+        import numpy as np
+    except ImportError:
+        print("  ⚠ 需要安装 pyaudio: pip install pyaudio")
+        return
+
+    arr = np.frombuffer(pcm_f32, dtype=np.float32)
+    arr_i16 = (arr * 32767).clip(-32768, 32767).astype(np.int16)
+
+    p = pyaudio.PyAudio()
+    stream = p.open(
+        format=pyaudio.paInt16,
+        channels=1,
+        rate=sample_rate,
+        output=True,
+    )
+    print("→ 播放中...")
+    stream.write(arr_i16.tobytes())
+    stream.stop_stream()
+    stream.close()
+    p.terminate()
+    print("→ 播放完毕")
+
+
+if __name__ == "__main__":
+    main()