Troubleshooting

View MASS-Base Logs

You can view MASS-Base logs with the following commands for the default setup:

docker logs -f gpustack

Enable Debug Mode

You can enable the DEBUG mode by setting the --debug flag when running MASS-Base:

sudo docker run -d --name gpustack \
    ...
    gpustack/gpustack \
+    --debug
    ...

You can also enable MASS-Base's debug mode at runtime by running the following command inside the server container:

gpustack reload-config --set debug=true

Configure Log Level

You can configure log level of the MASS-Base server at runtime by running the following command inside the server container:

curl -X PUT http://localhost/debug/log_level -d "debug"

The same applies to MASS-Base workers:

curl -X PUT http://localhost:10150/debug/log_level -d "debug"

The available log levels are: trace, debug, info, warning, error, critical.

Reset Admin Password

In case you forgot the admin password, you can reset it by running the following command inside the server container:

gpustack reset-admin-password

If you changed the default port using --port when starting MASS-Base, specify the MASS-Base URL using the --server-url parameter. It must be run locally on the server and accessed via localhost:

gpustack reset-admin-password --server-url http://localhost:9090

Assist in Accelerators Detection Diagnosis

After successfully deploying the MASS-Base Worker as described in the installation guide,
if the Worker fails to detect any devices,
please enter the corresponding Worker container, run the following command, and report the results to MASS-Base.

time GPUSTACK_RUNTIME_LOG_LEVEL=debug GPUSTACK_RUNTIME_LOG_EXCEPTION=1 gpustack-runtime detect --format json

Assist in Model Deployment Diagnosis

If you experience issues after deploying a model, PLEASE enter the corresponding Worker container, run the following command, and report the results to MASS-Base.

gpustack-runtime inspect <model instance name>

troubleshooting.md 2.2 KB Historie Surový