|
@@ -1,234 +1,125 @@
|
|
|
-<br>
|
|
|
|
|
|
|
+# MASS-Base
|
|
|
|
|
|
|
|
-<p align="center">
|
|
|
|
|
- <img alt="GPUStack" src="https://raw.githubusercontent.com/gpustack/gpustack/main/docs/assets/gpustack-logo.png" width="300px"/>
|
|
|
|
|
-</p>
|
|
|
|
|
-<br>
|
|
|
|
|
|
|
+MASS-Base 是一个开源的模型服务(Model-as-a-Service)基础平台,用于高效管理和调度 AI 模型推理服务。它支持多种推理引擎(vLLM、SGLang、TensorRT-LLM 等),可跨多节点进行性能优化与资源编排。
|
|
|
|
|
|
|
|
-<p align="center">
|
|
|
|
|
- <a href="https://docs.gpustack.ai" target="_blank">
|
|
|
|
|
- <img alt="Documentation" src="https://img.shields.io/badge/Docs-GPUStack-blue?logo=readthedocs&logoColor=white"></a>
|
|
|
|
|
- <a href="./LICENSE" target="_blank">
|
|
|
|
|
- <img alt="License" src="https://img.shields.io/github/license/gpustack/gpustack?logo=github&logoColor=white&label=License&color=blue"></a>
|
|
|
|
|
- <a href="./docs/assets/wechat-group-qrcode.jpg" target="_blank">
|
|
|
|
|
- <img alt="WeChat" src="https://img.shields.io/badge/WeChat-GPUStack-blue?logo=wechat&logoColor=white"></a>
|
|
|
|
|
- <a href="https://discord.gg/VXYJzuaqwD" target="_blank">
|
|
|
|
|
- <img alt="Discord" src="https://img.shields.io/badge/Discord-GPUStack-blue?logo=discord&logoColor=white"></a>
|
|
|
|
|
- <a href="https://twitter.com/intent/follow?screen_name=gpustack_ai" target="_blank">
|
|
|
|
|
- <img alt="Follow on X(Twitter)" src="https://img.shields.io/twitter/follow/gpustack_ai?logo=X"></a>
|
|
|
|
|
-</p>
|
|
|
|
|
-<br>
|
|
|
|
|
|
|
+## 核心特性
|
|
|
|
|
|
|
|
-<p align="center">
|
|
|
|
|
- <a href="./README.md">English</a> |
|
|
|
|
|
- <a href="./README_CN.md">简体中文</a> |
|
|
|
|
|
- <a href="./README_JP.md">日本語</a>
|
|
|
|
|
-</p>
|
|
|
|
|
|
|
+- **多集群管理**:统一管理多个环境中的计算节点,支持本地服务器和云平台。
|
|
|
|
|
+- **可插拔推理引擎**:自动配置 vLLM、SGLang、TensorRT-LLM 等高性能推理引擎,也支持自定义引擎接入。
|
|
|
|
|
+- **开箱即用的模型部署**:新模型发布即可快速部署。
|
|
|
|
|
+- **性能优化配置**:内置低延迟与高吞吐预调优模式,支持扩展 KV Cache(如 LMCache、HiCache)以降低 TTFT,并内置投机解码(EAGLE3、MTP、N-grams)支持。
|
|
|
|
|
+- **企业级运维能力**:支持自动故障恢复、负载均衡、监控、认证与访问控制。
|
|
|
|
|
|
|
|
-<br>
|
|
|
|
|
|
|
+## 架构
|
|
|
|
|
|
|
|
-## Overview
|
|
|
|
|
|
|
+MASS-Base 由以下核心组件构成:
|
|
|
|
|
|
|
|
-GPUStack is an open-source GPU cluster manager designed for efficient AI model deployment. It configures and orchestrates inference engines — vLLM, SGLang, TensorRT-LLM, or your own — to optimize performance across GPU clusters. Its core features include:
|
|
|
|
|
-- **Multi-Cluster GPU Management.** Manages GPU clusters across multiple environments. This includes on-premises servers, Kubernetes clusters, and cloud providers.
|
|
|
|
|
-- **Pluggable Inference Engines.** Automatically configures high-performance inference engines such as vLLM, SGLang, and TensorRT-LLM. You can also add custom inference engines as needed.
|
|
|
|
|
-- **Day 0 Model Support.** GPUStack's pluggable engine architecture enables you to deploy new models on the day they are released.
|
|
|
|
|
-- **Performance-Optimized Configurations.** Offers pre-tuned modes for low latency or high throughput. GPUStack supports extended KV cache systems like LMCache and HiCache to reduce TTFT. It also includes built-in support for speculative decoding methods such as EAGLE3, MTP, and N-grams.
|
|
|
|
|
-- **Enterprise-Grade Operations.** Offers support for automated failure recovery, load balancing, monitoring, authentication, and access control.
|
|
|
|
|
|
|
+- **API Server**:基于 FastAPI 构建的 RESTful 接口层,处理认证与授权。
|
|
|
|
|
+- **Scheduler**:负责将模型实例调度分配到工作节点。
|
|
|
|
|
+- **Controllers**:管理系统资源状态,处理模型实例的扩缩容。
|
|
|
|
|
+- **Worker**:检测 GPU 设备,管理模型实例的生命周期并导出性能指标。
|
|
|
|
|
+- **AI Gateway**:基于 Higress 构建,负责 API 路由与负载均衡。
|
|
|
|
|
+- **SQL Database**:默认使用嵌入式 PostgreSQL,也支持外部 PostgreSQL 或 MySQL。
|
|
|
|
|
|
|
|
-## Architecture
|
|
|
|
|
|
|
+
|
|
|
|
|
|
|
|
-GPUStack enables development teams, IT organizations, and service providers to deliver Model-as-a-Service at scale. It supports industry-standard APIs for LLM, voice, image, and video models. The platform includes built-in user authentication and access control, real-time monitoring of GPU performance and utilization, and detailed metering of token usage and API request rates.
|
|
|
|
|
|
|
+## 快速开始
|
|
|
|
|
|
|
|
-The figure below illustrates how a single GPUStack server can manage multiple GPU clusters across both on-premises and cloud environments. The GPUStack scheduler allocates GPUs to maximize resource utilization and selects the appropriate inference engines for optimal performance. Administrators also gain full visibility into system health and metrics through integrated Grafana and Prometheus dashboards.
|
|
|
|
|
|
|
+### 前置要求
|
|
|
|
|
|
|
|
-
|
|
|
|
|
|
|
+1. 至少一台 Linux 节点(支持 NVIDIA GPU、AMD GPU、Ascend NPU、Hygon DCU、MThreads GPU、Iluvatar GPU、MetaX GPU、Cambricon MLU、T-Head PPU 等加速器)。
|
|
|
|
|
+2. 工作节点需安装驱动、[Docker](https://docs.docker.com/engine/install/) 和 [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)。
|
|
|
|
|
+3. 服务端可运行在无 GPU 的 CPU 节点上,需安装 Docker。
|
|
|
|
|
|
|
|
-## Optimized Inference Performance
|
|
|
|
|
-
|
|
|
|
|
-GPUStack's automated engine selection and parameter optimization deliver strong inference performance out of the box. The following figure shows throughput improvements over default vLLM configurations:
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-For detailed benchmarking methods and results, visit our [Inference Performance Lab](https://docs.gpustack.ai/latest/performance-lab/overview/).
|
|
|
|
|
-
|
|
|
|
|
-## Supported Accelerators
|
|
|
|
|
-
|
|
|
|
|
-GPUStack supports a wide range of accelerators for AI inference:
|
|
|
|
|
-
|
|
|
|
|
-- **NVIDIA GPU**
|
|
|
|
|
-- **AMD GPU**
|
|
|
|
|
-- **Ascend NPU**
|
|
|
|
|
-- **Hygon DCU**
|
|
|
|
|
-- **MThreads GPU**
|
|
|
|
|
-- **Iluvatar GPU**
|
|
|
|
|
-- **MetaX GPU**
|
|
|
|
|
-- **Cambricon MLU**
|
|
|
|
|
-- **T-Head PPU**
|
|
|
|
|
-
|
|
|
|
|
-For detailed requirements and setup instructions, see the [Installation Requirements](https://docs.gpustack.ai/latest/installation/requirements/) documentation.
|
|
|
|
|
-
|
|
|
|
|
-## Quick Start
|
|
|
|
|
-
|
|
|
|
|
-### Prerequisites
|
|
|
|
|
-
|
|
|
|
|
-1. A node with at least one NVIDIA GPU. For other GPU types, please check the guidelines in the GPUStack UI when adding a worker, or refer to the [Installation documentation](https://docs.gpustack.ai/latest/installation/requirements/) for more details.
|
|
|
|
|
-2. Ensure the NVIDIA driver, [Docker](https://docs.docker.com/engine/install/) and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) are installed on the worker node.
|
|
|
|
|
-3. (Optional) A CPU node for hosting the GPUStack server. The GPUStack server does not require a GPU and can run on a CPU-only machine. [Docker](https://docs.docker.com/engine/install/) must be installed. Docker Desktop (for Windows and macOS) is also supported. If no dedicated CPU node is available, the GPUStack server can be installed on the same machine as a GPU worker node.
|
|
|
|
|
-4. Only Linux is supported for GPUStack worker nodes. If you use Windows, consider using WSL2 and avoid using Docker Desktop. macOS is not supported for GPUStack worker nodes.
|
|
|
|
|
-
|
|
|
|
|
-### Install GPUStack
|
|
|
|
|
-
|
|
|
|
|
-Run the following command to install and start the GPUStack server using Docker:
|
|
|
|
|
|
|
+### 安装服务端
|
|
|
|
|
|
|
|
```bash
|
|
```bash
|
|
|
-sudo docker run -d --name gpustack \
|
|
|
|
|
|
|
+sudo docker run -d --name mass-base \
|
|
|
--restart unless-stopped \
|
|
--restart unless-stopped \
|
|
|
-p 80:80 \
|
|
-p 80:80 \
|
|
|
- --volume gpustack-data:/var/lib/gpustack \
|
|
|
|
|
- gpustack/gpustack
|
|
|
|
|
|
|
+ --volume mass-base-data:/var/lib/mass-base \
|
|
|
|
|
+ mass-base/mass-base
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
-<details>
|
|
|
|
|
-<summary>Alternative: Use Quay Container Registry Mirror</summary>
|
|
|
|
|
-
|
|
|
|
|
-If you cannot pull images from `Docker Hub` or the download is very slow, you can use our `Quay.io` mirror by pointing your registry to `quay.io`:
|
|
|
|
|
|
|
+启动后查看日志:
|
|
|
|
|
|
|
|
```bash
|
|
```bash
|
|
|
-sudo docker run -d --name gpustack \
|
|
|
|
|
- --restart unless-stopped \
|
|
|
|
|
- -p 80:80 \
|
|
|
|
|
- --volume gpustack-data:/var/lib/gpustack \
|
|
|
|
|
- quay.io/gpustack/gpustack \
|
|
|
|
|
- --system-default-container-registry quay.io
|
|
|
|
|
|
|
+sudo docker logs -f mass-base
|
|
|
```
|
|
```
|
|
|
-</details>
|
|
|
|
|
|
|
|
|
|
-Check the GPUStack startup logs:
|
|
|
|
|
|
|
+获取默认管理员密码:
|
|
|
|
|
|
|
|
```bash
|
|
```bash
|
|
|
-sudo docker logs -f gpustack
|
|
|
|
|
|
|
+sudo docker exec mass-base cat /var/lib/mass-base/initial_admin_password
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
-After GPUStack starts, run the following command to get the default admin password:
|
|
|
|
|
-
|
|
|
|
|
-```bash
|
|
|
|
|
-sudo docker exec gpustack cat /var/lib/gpustack/initial_admin_password
|
|
|
|
|
-```
|
|
|
|
|
-
|
|
|
|
|
-Open your browser and navigate to `http://your_host_ip` to access the GPUStack UI. Use the default username `admin` and the password you retrieved above to log in.
|
|
|
|
|
-
|
|
|
|
|
-### Set Up a GPU Cluster
|
|
|
|
|
-
|
|
|
|
|
-1. On the GPUStack UI, navigate to the `Clusters` page.
|
|
|
|
|
-
|
|
|
|
|
-2. Click the `Add Cluster` button.
|
|
|
|
|
|
|
+在浏览器中访问 `http://your_host_ip`,使用用户名 `admin` 和获取到的密码登录。
|
|
|
|
|
|
|
|
-3. Select `Docker` as the cluster provider.
|
|
|
|
|
|
|
+### 部署模型
|
|
|
|
|
|
|
|
-4. Fill in the `Name` and `Description` fields for the new cluster, then click the `Save` button.
|
|
|
|
|
|
|
+1. 在 MASS-Base UI 中进入 **Catalog** 页面。
|
|
|
|
|
+2. 选择可用模型,通过兼容性检查后点击 **Save** 部署。
|
|
|
|
|
+3. 部署状态变为 **Running** 后即可通过 UI Playground 或 API 调用。
|
|
|
|
|
|
|
|
-5. Follow the UI guidelines to configure the new worker node. You will need to run a Docker command on the worker node to connect it to the GPUStack server. The command will look similar to the following:
|
|
|
|
|
|
|
+### 使用 API
|
|
|
|
|
|
|
|
- ```bash
|
|
|
|
|
- sudo docker run -d --name gpustack-worker \
|
|
|
|
|
- --restart=unless-stopped \
|
|
|
|
|
- --privileged \
|
|
|
|
|
- --network=host \
|
|
|
|
|
- --volume /var/run/docker.sock:/var/run/docker.sock \
|
|
|
|
|
- --volume gpustack-data:/var/lib/gpustack \
|
|
|
|
|
- --runtime nvidia \
|
|
|
|
|
- gpustack/gpustack \
|
|
|
|
|
- --server-url http://your_gpustack_server_url \
|
|
|
|
|
- --token your_worker_token \
|
|
|
|
|
- --advertise-address 192.168.1.2
|
|
|
|
|
- ```
|
|
|
|
|
-
|
|
|
|
|
-6. Execute the command on the worker node to connect it to the GPUStack server.
|
|
|
|
|
-
|
|
|
|
|
-7. After the worker node connects successfully, it will appear on the `Workers` page in the GPUStack UI.
|
|
|
|
|
-
|
|
|
|
|
-### Deploy a Model
|
|
|
|
|
-
|
|
|
|
|
-1. Navigate to the `Catalog` page in the GPUStack UI.
|
|
|
|
|
-
|
|
|
|
|
-2. Select the `Qwen3 0.6B` model from the list of available models.
|
|
|
|
|
-
|
|
|
|
|
-3. After the deployment compatibility checks pass, click the `Save` button to deploy the model.
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-4. GPUStack will start downloading the model files and deploying the model. When the deployment status shows `Running`, the model has been deployed successfully.
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-5. Click `Playground - Chat` in the navigation menu, check that the model `qwen3-0.6b` is selected from the top-right `Model` dropdown. Now you can chat with the model in the UI playground.
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-
|
|
|
|
|
-### Use the model via API
|
|
|
|
|
-
|
|
|
|
|
-1. Hover over the user avatar and navigate to the `API Keys` page, then click the `New API Key` button.
|
|
|
|
|
-
|
|
|
|
|
-2. Fill in the `Name` and click the `Save` button.
|
|
|
|
|
-
|
|
|
|
|
-3. Copy the generated API key and save it somewhere safe. Please note that you can only see it once on creation.
|
|
|
|
|
-
|
|
|
|
|
-4. You can now use the API key to access the OpenAI-compatible API endpoints provided by GPUStack. For example, use curl as the following:
|
|
|
|
|
|
|
+1. 在 UI 中进入 **API Keys** 页面,创建新的 API Key。
|
|
|
|
|
+2. 使用 API Key 调用 OpenAI 兼容接口:
|
|
|
|
|
|
|
|
```bash
|
|
```bash
|
|
|
-# Replace `your_api_key` and `your_gpustack_server_url`
|
|
|
|
|
-# with your actual API key and GPUStack server URL.
|
|
|
|
|
-export GPUSTACK_API_KEY=your_api_key
|
|
|
|
|
-curl http://your_gpustack_server_url/v1/chat/completions \
|
|
|
|
|
|
|
+export MASS_API_KEY=your_api_key
|
|
|
|
|
+curl http://your_mass_base_server_url/v1/chat/completions \
|
|
|
-H "Content-Type: application/json" \
|
|
-H "Content-Type: application/json" \
|
|
|
- -H "Authorization: Bearer $GPUSTACK_API_KEY" \
|
|
|
|
|
|
|
+ -H "Authorization: Bearer $MASS_API_KEY" \
|
|
|
-d '{
|
|
-d '{
|
|
|
- "model": "qwen3-0.6b",
|
|
|
|
|
|
|
+ "model": "your-model-name",
|
|
|
"messages": [
|
|
"messages": [
|
|
|
- {
|
|
|
|
|
- "role": "system",
|
|
|
|
|
- "content": "You are a helpful assistant."
|
|
|
|
|
- },
|
|
|
|
|
- {
|
|
|
|
|
- "role": "user",
|
|
|
|
|
- "content": "Tell me a joke."
|
|
|
|
|
- }
|
|
|
|
|
|
|
+ { "role": "system", "content": "You are a helpful assistant." },
|
|
|
|
|
+ { "role": "user", "content": "Tell me a joke." }
|
|
|
],
|
|
],
|
|
|
"stream": true
|
|
"stream": true
|
|
|
}'
|
|
}'
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
-## Documentation
|
|
|
|
|
|
|
+## 构建
|
|
|
|
|
|
|
|
-Please see the [official docs site](https://docs.gpustack.ai) for complete documentation.
|
|
|
|
|
|
|
+1. 安装 Python 3.10 ~ 3.12。
|
|
|
|
|
|
|
|
-## Build
|
|
|
|
|
|
|
+2. 执行构建:
|
|
|
|
|
|
|
|
-1. Install Python (version 3.10 to 3.12).
|
|
|
|
|
|
|
+```bash
|
|
|
|
|
+make build
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+构建产物位于 `dist` 目录。
|
|
|
|
|
+
|
|
|
|
|
+## 开发
|
|
|
|
|
|
|
|
-2. Run `make build`.
|
|
|
|
|
|
|
+```bash
|
|
|
|
|
+# 安装开发依赖
|
|
|
|
|
+make install
|
|
|
|
|
+
|
|
|
|
|
+# 本地开发启动(需先运行数据库)
|
|
|
|
|
+uv run gpustack start \
|
|
|
|
|
+ --database-url postgresql://postgres:mysecretpassword@localhost:5432/postgres \
|
|
|
|
|
+ --gateway-mode disabled \
|
|
|
|
|
+ --api-port 80
|
|
|
|
|
+```
|
|
|
|
|
|
|
|
-You can find the built wheel package in `dist` directory.
|
|
|
|
|
|
|
+更多开发指南请参考 [Development Guide](docs/development.md)。
|
|
|
|
|
|
|
|
-## Contributing
|
|
|
|
|
|
|
+## 文档
|
|
|
|
|
|
|
|
-Please read the [Contributing Guide](./docs/contributing.md) if you're interested in contributing to GPUStack.
|
|
|
|
|
|
|
+完整文档请访问 [官方文档站点](https://docs.gpustack.ai)。
|
|
|
|
|
|
|
|
-## Join Community
|
|
|
|
|
|
|
+## 加入社区
|
|
|
|
|
|
|
|
-Any issues or have suggestions, feel free to join our [Community](https://discord.gg/VXYJzuaqwD) for support.
|
|
|
|
|
|
|
+有任何问题或建议,欢迎加入我们的 [Discord 社区](https://discord.gg/VXYJzuaqwD) 获取支持。
|
|
|
|
|
|
|
|
## License
|
|
## License
|
|
|
|
|
|
|
|
-Copyright (c) 2024-2026 The GPUStack authors
|
|
|
|
|
|
|
+Copyright (c) 2024-2026 The MASS-Base authors
|
|
|
|
|
|
|
|
-Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
|
-you may not use this file except in compliance with the License.
|
|
|
|
|
-You may obtain a copy of the License at [LICENSE](./LICENSE) file for details.
|
|
|
|
|
|
|
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at [LICENSE](./LICENSE).
|
|
|
|
|
|
|
|
-Unless required by applicable law or agreed to in writing, software
|
|
|
|
|
-distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
|
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
|
-See the License for the specific language governing permissions and
|
|
|
|
|
-limitations under the License.
|
|
|
|
|
|
|
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
|