architecture.md 2.1 KB
Architecture
The diagram below provides a high-level view of the MASS-Base architecture.
The diagram below details the internal components and their interactions.
Server
The MASS-Base server consists of the following components:
- API Server: Provides a RESTful interface for clients to interact with the system. It handles authentication and authorization.
- Scheduler: Responsible for assigning model instances to workers.
- Controllers: Manages the state of resources in the system. For example, they handle the rollout and scaling of model instances to match the desired number of replicas.
Worker
The MASS-Base worker consists of the following components:
- MASS-Base Runtime: Detects GPU devices and interacts with the container runtime to deploy model instances.
- Serving Manager: Manages the lifecycle of model instances on the worker.
- Metric Exporter: Exports metrics about the model instances and their performance.
AI Gateway
The AI Gateway handles incoming API requests from clients. It routes requests to the appropriate model instances based on the requested model. MASS-Base uses Higress for API routing and load balancing.
SQL Database
The MASS-Base server connects to a SQL database as the datastore. MASS-Base uses an Embedded PostgreSQL by default, but you can configure it to use an external PostgreSQL or MySQL as well.
Inference Server
Inference servers are the backends that perform the inference tasks. MASS-Base supports vLLM, SGLang, Ascend MindIE and VoxBox as the built-in inference server. You can also add custom inference backends.
Ray
Ray is a distributed computing framework that MASS-Base utilizes to run distributed vLLM. MASS-Base bootstraps Ray cluster on-demand to run distributed vLLM across multiple workers.