Massed Compute

You get a /v1 endpoint on port 8000 (same as vLLM; SGLang defaults to 30000). The walkthrough uses ungated TinyLlama/TinyLlama-1.1B-Chat-v1.0 so the API comes up without a Hugging Face token. This was validated on an NVIDIA H100 PCIe (80 GB) on August 25, 2026. TensorRT-LLM 1.2.1 served that model on the PyTorch backend (the 1.2.1 default). There is no separate trtllm-build step in this path.

If you already run SGLang or vLLM on Massed Compute, this is the NVIDIA container path — same OpenAI request shape, different runtime. Do not copy those pip/venv guides; the install here is Docker.

Massed Compute VM Pricing

Lead with the H100 this guide was tested on. Blackwell is listed for FP4 work we did not run here.

Pricing fetched from the Massed Compute inventory API on August 25, 2026.

Step-by-Step Deployment

Image 184 logs in as Ubuntu (capital U). If SSH offers extra keys and then fails, add -o IdentitiesOnly=yes. Image 184 already has Docker and the NVIDIA Container Toolkit. Ubuntu is not in the docker group, so the bootstrap uses sudo docker. systemd runs the container as root, which is how this run stayed up.

# Launch via Massed Compute dashboard or API
# Product: gpu_1x_h100
# Image: 184 (Ubuntu Server 24.04 w/ Drivers)
# SSH Key: attach your public key
# Instance name: tensorrt-llm-openai

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader'

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail

TRTLLM_IMAGE="${TRTLLM_IMAGE:-nvcr.io/nvidia/tensorrt-llm/release:1.2.1}"
TRTLLM_MODEL="${TRTLLM_MODEL:-TinyLlama/TinyLlama-1.1B-Chat-v1.0}"
TRTLLM_PORT="${TRTLLM_PORT:-8000}"

mkdir -p "${HOME}/.cache/huggingface" "${HOME}/trtllm-openai"

sudo docker pull "${TRTLLM_IMAGE}"

sudo tee /etc/systemd/system/trtllm-openai.service >/dev/null <<UNIT
[Unit]
Description=TensorRT-LLM OpenAI-compatible API
After=docker.service network-online.target
Requires=docker.service
Wants=network-online.target

[Service]
Type=simple
Restart=on-failure
RestartSec=15
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker rm -f trtllm-openai
ExecStart=/usr/bin/docker run --rm --name trtllm-openai --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -p 127.0.0.1:${TRTLLM_PORT}:8000 -v ${HOME}/.cache/huggingface:/root/.cache/huggingface ${TRTLLM_IMAGE} trtllm-serve ${TRTLLM_MODEL} --host 0.0.0.0 --port 8000
ExecStop=/usr/bin/docker stop trtllm-openai

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now trtllm-openai
EOF

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'systemctl is-active trtllm-openai'

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euo pipefail
for _ in $(seq 1 90); do
  if curl -sf http://127.0.0.1:8000/v1/models >/tmp/trtllm-models.json; then
    cat /tmp/trtllm-models.json
    echo
    echo TRTLLM_MODELS_OK
    exit 0
  fi
  sleep 10
done
journalctl -u trtllm-openai -n 160 --no-pager
exit 1
EOF

{"object":"list","data":[{"id":"TinyLlama/TinyLlama-1.1B-Chat-v1.0","object":"model","created":1787660705,"owned_by":"tensorrt_llm"}]}

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euo pipefail
curl -sf http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"TinyLlama/TinyLlama-1.1B-Chat-v1.0","messages":[{"role":"user","content":"Reply with the single word pong."}],"max_tokens":16,"temperature":0}'
echo
EOF

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'nvidia-smi --query-gpu=name,memory.used,utilization.gpu --format=csv,noheader'

Model Conversion and Engine Build

On TensorRT-LLM 1.2.1, conversion and trtllm-build are not required and not available. NVIDIA’s 1.2 release notes remove the TensorRT engine backend: trtllm-build, trtllm-refit, trtllm-prune, --backend tensorrt, and the per-model convert_checkpoint.py scripts are gone. PyTorch is the only execution backend. Point trtllm-serve at a Hugging Face repo or a TensorRT Model Optimizer checkpoint. That is what this walkthrough did.

Older Hugging Face cards (for example NVIDIA’s Llama 3.1 8B Instruct FP8 notes written against TensorRT-LLM 0.13) still show convert_checkpoint.py then trtllm-build. Those commands fail in this NGC 1.2.1 container. Do not paste them into the systemd unit above.

When you want a quantized checkpoint on 1.2.1, skip the engine build and serve the published Model Optimizer weights:

# Inside the same NGC container / systemd ExecStart model argument.
# Gated Llama: add docker run -e HF_TOKEN=...  Not run on this VM.
trtllm-serve nvidia/Llama-3.1-8B-Instruct-FP8 --host 0.0.0.0 --port 8000

NVIDIA’s ungated FP8 example is nvidia/Qwen3-8B-FP8 (model card lists Blackwell / B200 test hardware). Confirm the card matches your GPU before you swap it in.

Where TensorRT-LLM Fits vs vLLM, SGLang, and TGI

Use this post for TensorRT-LLM. Link out for the other engines — those guides already exist.

• TensorRT-LLM (this guide): NVIDIA NGC container + trtllm-serve. PyTorch backend in 1.2.1. OpenAI /v1 on port 8000. Pick this when you want NVIDIA’s runtime on Hopper/Blackwell.
• SGLang: high-throughput self-hosted serving, radix / prefix cache, port 30000. See Deploy SGLang with an OpenAI-Compatible API on GPU Cloud.
• vLLM: the existing Massed Compute production OpenAI-API path. See Deploy vLLM with OpenAI API on GPU Cloud.
• TGI: Hugging Face–native serving. See Build an LLM Inference API with TGI.

This is not a bake-off. The numbers below are TinyLlama on one H100, not a vLLM or SGLang comparison. Pick TensorRT-LLM when you want NVIDIA’s Hopper/Blackwell runtime and Model Optimizer checkpoints. Pick SGLang for prefix cache / high QPS. Pick vLLM when you want the OpenAI-API path Massed Compute already runs in production.

Supported GPUs: H100, Blackwell, and FP8/FP4

TensorRT-LLM 1.2.1 is NVIDIA’s inference stack for Hopper and Blackwell. This post was validated on one H100 PCIe 80 GB.

NVIDIA’s 1.2 notes list Llama 3.1 8B Instruct at FP16 / FP8 / NVFP4 among validated models. NVFP4 is the Blackwell-oriented path. H100 product context: Llama 3.1 Benchmark by GPU Type. Blackwell product context: RTX PRO 6000 Blackwell.

Precision, VRAM, and Multi-GPU

H100 (Hopper) has FP8. RTX PRO 6000 Blackwell has FP4. This VM ran TinyLlama, not an FP8 or FP4 checkpoint.

gpu_2x_h100 was $5.46/hr with 1 free on August 25, 2026. GPU pricing for current stock. Tensor parallelism on trtllm-serve is --tp_size, not --tp. Stay on one GPU (this walkthrough) until a single card OOMs or you need a 70B-class checkpoint. Then launch gpu_2x_h100 (or Blackwell 2×) and set --tp_size 2 to match the GPU count. Pipeline parallel is --pp_size in NVIDIA’s serve CLI; it was not used here.

To try NVIDIA’s H100-documented FP8 8B (gated) after this smoke is green:

# Requires a Hugging Face token with Llama 3.1 access.
# Edit ExecStart: TinyLlama/... → nvidia/Llama-3.1-8B-Instruct-FP8
# Add: -e HF_TOKEN=... on the docker run line, then:
sudo systemctl daemon-reload
sudo systemctl restart trtllm-openai

Concurrency, TTFT, and Throughput

Same H100 PCIe, TinyLlama-1.1B-Chat-v1.0, TensorRT-LLM 1.2.1 PyTorch backend, localhost, August 25, 2026.

Report the median TTFT. The 50.7 ms first stream is warmup, not the number to quote.

This is a smoke on TinyLlama, not an FP8 vs BF16 bake-off. NVIDIA’s published before/after for a Hopper FP8 8B (not this VM): on H100 they report ~1.3× throughput for nvidia/Llama-3.1-8B-Instruct-FP8 vs FP16 (MMLU 68.3 vs 68.6) with TensorRT-LLM 0.13 on 8× H100, batch 1024, in-flight batching — see the Llama-3.1-8B-Instruct-FP8 model card. That is NVIDIA’s number, a different TRT-LLM version, and eight GPUs. Do not quote it as this Massed Compute run.

Eight-way smoke from the VM:

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euo pipefail
for i in $(seq 1 8); do
  curl -sf http://127.0.0.1:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"TinyLlama/TinyLlama-1.1B-Chat-v1.0","messages":[{"role":"user","content":"Reply with one short sentence."}],"max_tokens":32,"temperature":0}' \
    >/tmp/trtllm-c$i.json &
done
wait
echo TRTLLM_CONCURRENCY_OK
EOF

Troubleshooting

SSH fails or keeps asking for a password. Image 184’s user is Ubuntu, not ubuntu. Use -o IdentitiesOnly=yes. Refresh host keys with ssh-keygen -R YOUR_VM_IP if the instance was relaunched on a reused IP.

permission denied talking to Docker. Image 184 ships Docker, but Ubuntu is not in the docker group. Use sudo docker …, or sudo usermod -aG docker Ubuntu and start a new SSH session. systemd does not need that group.

NGC pull is slow. Tag 1.2.1 is about 16 GB. Pin that tag. Do not use floating latest in a systemd unit.

Container says CUDA Forward Compatibility ENABLED. Expected on image 184 (driver 580) with this NGC image (CUDA 13.1 / 590). This H100 served in that mode.

systemctl is-active is active but curl fails. Docker is up; weights and autotune are still loading. Poll /v1/models. Read journalctl -u trtllm-openai -f.

Tiny model using ~73 GB. Default paged KV cache. Add --kv_cache_free_gpu_memory_fraction 0.3 for smoke tests.

FP8 checkpoint fails on L40 / A6000. Hopper FP8 wants H100/H200. Ada can run TensorRT-LLM; do not assume a Hopper FP8 checkpoint will load there. FP4 wants Blackwell. Neither was run on this VM.

Gated model 401. TinyLlama does not need a token. Llama 3.1 FP8 does: put HF_TOKEN on the docker run -e line (systemd Environment= alone does not enter the container).

Nothing on port 30000. trtllm-serve default is 8000. SGLang uses 30000.

Laptop curl to http://YOUR_VM_IP:8000 hangs. Expected while the host publish is 127.0.0.1. Test from the VM.

trtllm-build: command not found or convert_checkpoint.py missing. Expected on 1.2.1. The TensorRT engine path was removed. Serve a Hugging Face or Model Optimizer checkpoint with trtllm-serve.

Out of memory. Smaller model, shorter context, lower --kv_cache_free_gpu_memory_fraction, an FP8 checkpoint NVIDIA lists for your GPU, a larger SKU, or --tp_size.

Quick Setup Reference

# 1. Launch gpu_1x_h100, image 184
# 2. Verify GPU
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'nvidia-smi'

# 3. Bootstrap TensorRT-LLM (script in step 3)

# 4. Wait for /v1, then chat
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'curl -sf http://127.0.0.1:8000/v1/models'
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'curl -sf http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d "{...}"'

# 5. Monitor
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'journalctl -u trtllm-openai -f'

Frequently Asked Questions

Recipe tested on August 25, 2026.