Massed Compute

This guide launches a Massed Compute multi-GPU VM, pulls the NGC Dynamo SGLang runtime, and hits /v1/chat/completions the same way you would hit OpenAI. You get a /v1 endpoint on port 8000. The walkthrough uses ungated Qwen/Qwen3-0.6B so the API comes up without a Hugging Face token. This was validated on two NVIDIA RTX PRO 6000 Blackwell (96 GB) on August 31, 2026 — first aggregated (frontend + one worker on GPU 0, GPU 1 idle), then disaggregated (prefill on GPU 0, decode on GPU 1, NIXL/UCX). gpu_2x_a100 and gpu_2x_h100 were out of stock at launch.

If you already run SGLang or TensorRT-LLM on Massed Compute, this is the Dynamo path — same request shape, extra processes. Do not copy those pip/venv guides. The install here is Docker.

Architecture

Client
  → dynamo.frontend    127.0.0.1:8000   OpenAI /v1
       file discovery  /tmp/dynamo_store_kv
       TCP request plane (not NATS; no etcd on this smoke)

Aggregated (1 GPU)
  → dynamo.sglang worker     GPU 0     health :8081

Disaggregated (2 GPUs, this guide)
  → dynamo.sglang --disaggregation-mode prefill   GPU 0   bootstrap :12345   health :8081
  → dynamo.sglang --disaggregation-mode decode    GPU 1                      health :8082
       NIXL KV transfer (UCX)

Kubernetes, Grove, the SLA planner, and multi-node RDMA are real Dynamo features. They were not run on this VM. This post is the single-node container path NVIDIA documents in the repo README.

Massed Compute VM Pricing

Lead with the 2× Blackwell this guide was tested on. 2× A100 and 2× H100 were out of stock at launch.

Pricing fetched from the Massed Compute inventory API on August 31, 2026.

Step-by-Step Deployment

Image 184 logs in as Ubuntu (capital U). If SSH offers extra keys and then fails, add -o IdentitiesOnly=yes. Image 184 already has Docker and the NVIDIA Container Toolkit. Ubuntu is not in the docker group, so the bootstrap uses sudo docker. The NGC image’s default user is dynamo (uid 1000) and cannot write a host-mounted Hugging Face cache owned by Ubuntu — run the container as root (--user 0).

# Launch via Massed Compute dashboard or API
# Product: gpu_2x_pro_6000_blackwell
# Image: 184 (Ubuntu Server 24.04 w/ Drivers)
# SSH Key: attach your public key
# Instance name: dynamo-distributed

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'nvidia-smi --query-gpu=index,name,memory.total,driver_version --format=csv,noheader'

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
sudo ufw --force reset
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
echo y | sudo ufw enable
sudo ufw status verbose
EOF

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail

DYNAMO_IMAGE="${DYNAMO_IMAGE:-nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.1}"
DYNAMO_MODEL="${DYNAMO_MODEL:-Qwen/Qwen3-0.6B}"

mkdir -p "${HOME}/.cache/huggingface" "${HOME}/dynamo-logs"
chmod a+rwX "${HOME}/.cache/huggingface" "${HOME}/dynamo-logs"

sudo docker pull "${DYNAMO_IMAGE}"

sudo tee /usr/local/bin/dynamo-agg.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -euo pipefail
python3 -m dynamo.frontend --http-host 127.0.0.1 --http-port 8000 --discovery-backend file &
export CUDA_VISIBLE_DEVICES=0 DYN_SYSTEM_PORT=8081
exec python3 -m dynamo.sglang \
  --model-path "${DYNAMO_MODEL}" \
  --served-model-name "${DYNAMO_MODEL}" \
  --discovery-backend file \
  --tp 1 \
  --trust-remote-code \
  --page-size 16 \
  --disable-piecewise-cuda-graph
SCRIPT
sudo sed -i "s|\${DYNAMO_MODEL}|${DYNAMO_MODEL}|g" /usr/local/bin/dynamo-agg.sh
sudo chmod 755 /usr/local/bin/dynamo-agg.sh

sudo tee /etc/systemd/system/dynamo-openai.service >/dev/null <<UNIT
[Unit]
Description=NVIDIA Dynamo OpenAI-compatible API (aggregated)
After=docker.service network-online.target
Requires=docker.service
Wants=network-online.target

[Service]
Type=simple
Restart=on-failure
RestartSec=15
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker rm -f dynamo-openai
ExecStart=/usr/bin/docker run --rm --name dynamo-openai --user 0 --gpus all --network host --ipc=host --shm-size=16g -e HF_HOME=/root/.cache/huggingface -v ${HOME}/.cache/huggingface:/root/.cache/huggingface -v ${HOME}/dynamo-logs:/logs -v /usr/local/bin/dynamo-agg.sh:/dynamo-agg.sh:ro ${DYNAMO_IMAGE} bash /dynamo-agg.sh
ExecStop=/usr/bin/docker stop dynamo-openai

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now dynamo-openai
EOF

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'systemctl is-active dynamo-openai'

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euo pipefail
for _ in $(seq 1 90); do
  if curl -sf http://127.0.0.1:8000/v1/models | grep -q Qwen; then
    curl -sS http://127.0.0.1:8000/v1/models
    echo
    echo DYNAMO_MODELS_OK
    exit 0
  fi
  sleep 10
done
sudo docker logs dynamo-openai --tail 160
exit 1
EOF

{"object":"list","data":[{"id":"Qwen/Qwen3-0.6B","object":"model","created":1788179940,"owned_by":"nvidia","context_window":40960}]}

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euo pipefail
curl -sf http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Reply with the single word Pong."}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'
echo
EOF

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader'

Disaggregated Prefill / Decode (2 GPUs)

Stop the aggregated unit, then start frontend + prefill worker on GPU 0 + decode worker on GPU 1. NIXL moves KV cache over UCX. NVIDIA’s examples/backends/sglang/launch/disagg.sh is the same idea.

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
sudo systemctl stop dynamo-openai

DYNAMO_IMAGE="${DYNAMO_IMAGE:-nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.1}"
MODEL="${MODEL:-Qwen/Qwen3-0.6B}"

sudo docker rm -f dynamo-openai >/dev/null 2>&1 || true
sudo docker run -d --name dynamo-openai --user 0 \
  --gpus all --network host --ipc=host --shm-size=16g \
  -e HF_HOME=/root/.cache/huggingface \
  -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
  -v "${HOME}/dynamo-logs:/logs" \
  "${DYNAMO_IMAGE}" sleep infinity

sudo docker exec dynamo-openai bash -c "nohup python3 -m dynamo.frontend --http-host 127.0.0.1 --http-port 8000 --discovery-backend file >/logs/frontend-disagg.log 2>&1 &"
sudo docker exec dynamo-openai bash -c "export DYN_SYSTEM_PORT=8081; nohup python3 -m dynamo.sglang --model-path ${MODEL} --served-model-name ${MODEL} --discovery-backend file --tp 1 --trust-remote-code --page-size 16 --disable-piecewise-cuda-graph --disaggregation-mode prefill --disaggregation-bootstrap-port 12345 --host 0.0.0.0 --port 40000 --disaggregation-transfer-backend nixl >/logs/prefill.log 2>&1 &"
sudo docker exec dynamo-openai bash -c "export CUDA_VISIBLE_DEVICES=1 DYN_SYSTEM_PORT=8082; nohup python3 -m dynamo.sglang --model-path ${MODEL} --served-model-name ${MODEL} --discovery-backend file --tp 1 --trust-remote-code --page-size 16 --disable-piecewise-cuda-graph --disaggregation-mode decode --disaggregation-bootstrap-port 12345 --host 0.0.0.0 --disaggregation-transfer-backend nixl >/logs/decode.log 2>&1 &"
EOF

Poll /v1/models again. Both health ports should be ready:

curl -sf http://127.0.0.1:8081/health   # prefill
curl -sf http://127.0.0.1:8082/health   # decode

This run: prefill 85881 MiB on GPU 0, decode 89155 MiB on GPU 1. Logs printed NIXL KVManager initialized with backend: UCX on both workers, CommonKVBootstrapServer on :12345, and Decode worker handler initialized (disaggregated decode mode). First chat after /v1 was 3.80 s; warm Pong was 34 ms. Completions still returned Paris.

That is the distributed Dynamo smoke: two processes, two GPUs, KV moving. It is not a 70B multi-node Grove cluster.

Where Dynamo Fits vs SGLang, vLLM, and TensorRT-LLM

Use this post for Dynamo. Link out for the engines — those guides already exist.

• Dynamo (this guide): NGC container + dynamo.frontend + dynamo.sglang workers. OpenAI /v1 on port 8000. Pick this when you want disaggregated prefill/decode, KV-aware routing, or a path toward multi-node serving.
• SGLang alone: high-throughput self-hosted serving, radix / prefix cache, port 30000. See Deploy SGLang with an OpenAI-Compatible API on GPU Cloud. Dynamo’s worker is SGLang; the extra process is the frontend/router.
• vLLM: the existing Massed Compute production OpenAI-API path. Dynamo also ships a vllm-runtime image; it was not run here. See Deploy vLLM with OpenAI API on GPU Cloud.
• TensorRT-LLM: NVIDIA’s engine container + trtllm-serve. Dynamo ships tensorrtllm-runtime; it was not run here. See Deploy TensorRT-LLM on NVIDIA GPUs.

This is not a bake-off. The numbers below are Qwen3-0.6B on two Blackwell cards, not a vs-SGLang-native comparison. If you are serving one model on one GPU, the engine-only posts are enough. Dynamo starts to matter when you split prefill and decode or route across workers.

Concurrency, TTFT, and Throughput

Same 2× RTX PRO 6000 Blackwell, Qwen/Qwen3-0.6B, Dynamo 1.4.1 + SGLang 0.5.16, localhost, August 31, 2026.

Report the median TTFT, not the first stream. Both 8-way rows used thinking off so the token counts match. Disagg is slower on this 0.6B because NIXL transfer dominates a tiny model — that is expected, not a vs-SGLang-native bake-off. Same lesson as the verl 0.5B two-GPU run: prove the launcher here, size the SKU for the 7B+ job.

Eight-way smoke (same body as the table):

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euo pipefail
for i in $(seq 1 8); do
  curl -sf http://127.0.0.1:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Reply with the single word Pong."}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}' \
    >/tmp/dynamo-c$i.json &
done
wait
echo DYNAMO_CONCURRENCY_OK
EOF

Troubleshooting

SSH fails or keeps asking for a password. Image 184’s user is Ubuntu, not ubuntu. Use -o IdentitiesOnly=yes. Refresh host keys with ssh-keygen -R YOUR_VM_IP if the instance was relaunched on a reused IP.

permission denied talking to Docker. Image 184 ships Docker, but Ubuntu is not in the docker group. Use sudo docker ….

/logs/frontend-agg.log: Permission denied inside the container. The image user is dynamo (uid 1000). Host Ubuntu is a different uid. Pass --user 0 or chmod a+rwX the Hugging Face cache and log dirs.

NGC pull is slow. Tag 1.4.1 is about 27.6 GB. Pin that tag. Do not use floating latest in a systemd unit.

/v1/models is {"data":[]}. Frontend is up; the worker is still downloading weights or capturing CUDA graphs. Poll. GPU 0 jumping to ~86 GB while models is empty is expected during graph capture.

First /v1/chat/completions after /v1/models is ready takes several seconds. Expected. This restart: 4.15 s aggregated, 3.80 s disagg. Warm repeats were tens of milliseconds. Do not treat the first chat as TTFT.

Chat is a long <think> block instead of Pong. Qwen3 default thinking. Pass "chat_template_kwargs":{"enable_thinking":false} or --dyn-default-thinking-mode disabled on the worker.

Only GPU 0 is busy on the 2-GPU job. You are still on the aggregated unit (CUDA_VISIBLE_DEVICES=0). Disagg needs a prefill process and CUDA_VISIBLE_DEVICES=1 on decode.

Disagg workers cannot connect. Bootstrap is --disaggregation-bootstrap-port (12345 here) and --host 0.0.0.0 on the workers. --network host is the easy single-node path. Logs should show NIXL KVManager initialized with backend: UCX.

Laptop curl to http://YOUR_VM_IP:8000 hangs. Expected while UFW is SSH-only and HTTP is 127.0.0.1. Test from the VM. This run’s public :8000 timed out from outside, which is what you want.

Nothing on port 30000. Dynamo frontend default is 8000. Bare SGLang uses 30000.

etcd / NATS missing. Expected. This smoke uses --discovery-backend file. Production Kubernetes uses the Dynamo operator, not this file backend.

Out of memory. Smaller model, --mem-fraction-static 0.3, shorter context, or a larger SKU. Default KV reservation filled ~86 GB of a 96 GB card even for 0.6B.

Quick Setup Reference

# 1. Launch gpu_2x_pro_6000_blackwell, image 184
# 2. Verify GPU + UFW 22
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'nvidia-smi'

# 3. Bootstrap Dynamo aggregated (script in step 3)

# 4. Wait for /v1, then chat
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'curl -sf http://127.0.0.1:8000/v1/models'
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'curl -sf http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d "{...}"'

# 5. Optional: stop aggregated, start disagg (step after 7)

# 6. Monitor
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'journalctl -u dynamo-openai -f'

Frequently Asked Questions

Recipe tested on August 31, 2026.