Massed Compute

Launch GPU VM

Create a VM with GPU support and NVIDIA drivers pre-installed:

# Launch via Massed Compute dashboard or API
# Product: gpu_1x_l40 (or gpu_1x_l40_spot)
# Image: 184 (Ubuntu Server 24.04 w/ Drivers)
# SSH Key: attach your public key
# Instance name: sglang-openai-api

Wait until the VM is running and copy the SSH details.

Verify GPU Access

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader'

You should see the GPU name and VRAM. This run printed an L40 at 46068 MiB.

Bootstrap SGLang

Run the setup on the VM. Image 184 ships NVIDIA drivers, not a system nvcc. SGLang’s JIT path needs CUDA_HOME pointed at the CUDA 13 toolkit inside the venv.

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail

SGLANG_MODEL="${SGLANG_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}"
SGLANG_PORT="${SGLANG_PORT:-30000}"

sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
  python3-venv python3-pip git curl

mkdir -p ~/sglang-openai
python3 -m venv ~/sglang-openai/venv
. ~/sglang-openai/venv/bin/activate

python -m pip install -U pip setuptools wheel uv
uv pip install --prerelease=allow 'sglang==0.5.17'

PYVER="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
CU="$HOME/sglang-openai/venv/lib/python${PYVER}/site-packages/nvidia/cu13"
test -x "$CU/bin/nvcc"
ln -sfn lib "$CU/lib64"
ln -sfn libcudart.so.13 "$CU/lib/libcudart.so"
echo "$CU/lib" | sudo tee /etc/ld.so.conf.d/sglang-cu13.conf
sudo ldconfig

sudo tee /etc/systemd/system/sglang-openai.service >/dev/null <<UNIT
[Unit]
Description=SGLang OpenAI-compatible API
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=${USER}
Group=${USER}
WorkingDirectory=${HOME}/sglang-openai
Environment=HOME=${HOME}
Environment=CUDA_HOME=${CU}
Environment=PATH=${CU}/bin:${HOME}/sglang-openai/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
ExecStart=${HOME}/sglang-openai/venv/bin/python -m sglang.launch_server --model-path ${SGLANG_MODEL} --host 127.0.0.1 --port ${SGLANG_PORT} --attention-backend triton --disable-cuda-graph
Restart=on-failure
RestartSec=15
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now sglang-openai
EOF

--host 127.0.0.1 keeps the API on the VM until you add a firewall rule and auth. --attention-backend triton --disable-cuda-graph is the combo that served on image 184 without a system CUDA toolkit. Newer SGLang builds prefer --cuda-graph-backend-decode=disabled --cuda-graph-backend-prefill=disabled instead of --disable-cuda-graph.

To use a 7B–8B model later, set SGLANG_MODEL before the script, or edit ExecStart and restart the unit. Gated Llama weights need HF_TOKEN in the systemd Environment= lines.

Verify Service Status

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'systemctl is-active sglang-openai'

Expected output: active

Test Models Endpoint

First start downloads the model. Poll /v1/models until it answers:

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euo pipefail
for _ in $(seq 1 90); do
  if curl -sf http://127.0.0.1:30000/v1/models >/tmp/sglang-models.json; then
    cat /tmp/sglang-models.json
    echo
    echo SGLANG_MODELS_OK
    exit 0
  fi
  sleep 10
done
journalctl -u sglang-openai -n 160 --no-pager
exit 1
EOF

You should see Qwen/Qwen2.5-0.5B-Instruct in the JSON.

Run Chat Completion Test

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euo pipefail
curl -sf http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"Reply with the single word pong."}],"max_tokens":16,"temperature":0}'
echo
EOF

The assistant message should be a short Pong (or Pong!). Same request shape as OpenAI and as the vLLM OpenAI API post — only the host port is 30000.

Confirm GPU Utilization

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'nvidia-smi --query-gpu=name,memory.used,utilization.gpu --format=csv,noheader'

This L40 sat at about 39799 MiB used after load. That is not 0.5B of weights. SGLang’s default mem-fraction-static (~0.85) reserves most of the card for the KV cache. For a smoke model, add --mem-fraction-static 0.3 to ExecStart if you want leftover VRAM. For a real 7B–8B serve, leaving the default is usually what you want.