verl (Volcano Engine Reinforcement Learning) is a HybridFlow trainer: an FSDP actor and a vLLM rollout share the same GPUs, then a rule-based reward scores completions. This guide launches a Massed Compute multi-GPU VM, installs verl with uv, converts GSM8K to parquet, and runs GRPO on an ungated instruct model until a two-rank FSDP checkpoint is on disk.
The walkthrough uses Qwen/Qwen2.5-0.5B-Instruct and openai/gsm8k (7,473 train / 1,319 test rows after examples/data_preprocess/gsm8k.py). That is a smoke you can finish in minutes, not a production 7B or 70B RL run. Swap the model path once global_step_* /actor/ is green. This was validated on two NVIDIA A100 PCIe (80 GB) on August 27, 2026 — first with CUDA_VISIBLE_DEVICES=0 and n_gpus_per_node=1 (10 steps), then on both cards with n_gpus_per_node=2 (20 steps). algorithm.adv_estimator=grpo, actor_rollout_ref.rollout.name=vllm. That is verl multi-GPU post-training on a Massed Compute VM, not a single-GPU notebook.
If you already follow Fine-Tune LLMs with Axolotl on Multi-GPU VMs, that post is YAML SFT / QLoRA. Fine-Tune LLMs Faster with Unsloth on GPU Cloud is Unsloth’s FastLanguageModel path. This post is python3 -m verl.trainer.main_ppo with GRPO. Do not copy those install commands. PPO with a critic, Megatron-LM, and DeepSpeed ZeRO are named here, not walked through.
| Component | Version | Purpose |
|---|---|---|
| Ubuntu Server | 24.04 LTS | Image 184: NVIDIA driver 580.126.16, CUDA 13.0 |
| verl | 0.10.0.dev0 (662507f) |
HybridFlow trainer (verl.trainer.main_ppo) |
| PyTorch | 2.11.0+cu130 | CUDA 13.0 wheels from manage_envs.py sync vllm fsdp |
| vLLM | 0.24.0 | Rollout generation (actor_rollout_ref.rollout.name=vllm) |
| Transformers | 5.5.3 | Qwen2 tokenizer + causal LM |
| Ray | 2.55.1 | Single-controller workers (GPU:2 on this host) |
| flash-attn | 2.8.3 | Prebuilt cu130 wheel from the verl wheelhouse |
| FSDP | v1, world_size: 2 |
Actor shards; ref param-offload |
| Model (validated) | Qwen/Qwen2.5-0.5B-Instruct | Ungated instruct model, GRPO |
| Dataset | openai/gsm8k | 7,473 / 1,319 parquet rows, rule reward #### |
| Resource | This walkthrough | Notes |
|---|---|---|
| GPU | 2× A100 PCIe 80 GB (gpu_2x_a100) |
Validated. 0.5B GRPO peaked 27 GB on one card / ~20 GB on both. 7B GRPO is why you want 80 GB × 2 |
| System RAM | 150 GB | SKU ships 150 GiB; Ray object store used tens of GB |
| vCPU | 30 | SKU ships 30 vCPU |
| Storage | 1024 GB | Hugging Face cache plus FSDP checkpoints (~6.1 GB per step with optimizer) |
| Network | 1 Gbps | First uv sync pulls CUDA 13 wheels; first train pulls Qwen weights |
Why this job wants more than one GPU
SFT (Axolotl / Unsloth) is one training process. verl’s HybridFlow keeps three roles on the node at once:
- Rollout — vLLM samples a group of completions per prompt (
rollout.n=4here). - Actor — FSDP updates the policy from GRPO advantages.
- Reference — FSDP forward for KL (
use_kl_loss=True). GRPO does not train a critic.
Those roles are colocated. The 1-GPU baseline peaked at 27233 MiB on GPU 0 and 0 MiB on GPU 1. The 2-GPU run peaked at 20053 / 20055 MiB with 100% util on both — not “GPU 0 trains, GPU 1 idle.” Same global batch (8 prompts × 4 samples). A 7B–8B GRPO job with longer responses and larger rollout.n is why the SKU is two 80 GB cards, not a 24 GB box. This 0.5B smoke is slower per step on two GPUs (~21 s vs ~10.5 s after warmup) because FSDP + vLLM sync dominates a tiny model. That is the same lesson as the Axolotl 0.5B DDP run: prove the launcher here, size the SKU for the 7B job.
PPO with a value head adds a critic replica. Skip that until you need GAE. This smoke set algorithm.adv_estimator=grpo; verl logged Disabled critic as algorithm.adv_estimator != gae.
Massed Compute VM Pricing
Lead with the 2× A100 this guide was tested on. 2× H100 was out of stock at launch; 8× A100 SXM is listed for jobs that do not fit two cards.
Pricing fetched from the Massed Compute inventory API on August 27, 2026.
| SKU | Description | vCPU | RAM | Storage | Price | Capacity |
|---|---|---|---|---|---|---|
gpu_1x_a100 |
1x A100 (80GB) | 16 | 96 GiB | 512 GB | $1.35/hr | 3 |
gpu_2x_l40 |
2x L40 (48GB) | 26 | 144 GiB | 1250 GB | $1.72/hr | 10 |
gpu_2x_a100 |
2x A100 (80GB) | 30 | 150 GiB | 1024 GB | $2.70/hr | 1 |
gpu_1x_h100 |
1x H100 (80GB) | 20 | 128 GiB | 1250 GB | $2.73/hr | 1 |
gpu_2x_h100 |
2x H100 (80GB) | 40 | 256 GiB | 2500 GB | $5.46/hr | 0 |
gpu_8x_A100_SXM4 |
8x A100 SXM4 (80GB) | 126 | 800 GiB | 5000 GB | $11.04/hr | 1 |
gpu_2x_a100. Use on-demand if the job cannot restart. A100 context: NVIDIA A100 GPU Best Use Cases. Hopper / H100 context: Llama 3.1 Benchmark by GPU Type. Training vs inference card sizing: The Best GPU for LLM Inference Without Overpaying. After you have a merged actor, serve it with SGLang — that is a different post.Step-by-Step Deployment
Image 184 logs in as Ubuntu (capital U). If SSH offers extra keys and then fails, add -o IdentitiesOnly=yes. Image 184 ships NVIDIA drivers, not a ready Python venv. Install python3.12-venv (or python3-venv) and git before the uv path.
Launch GPU VM
# Launch via Massed Compute dashboard or API
# Product: gpu_2x_a100
# Image: 184 (Ubuntu Server 24.04 w/ Drivers)
# SSH Key: attach your public key
# Instance name: verl-multigpu
Wait until the VM is running and copy the SSH details. gpu_2x_h100 was out of stock the day we launched; the commands are the same on two 80 GB cards.
Verify GPU Access
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
'nvidia-smi -L; nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader'
This run printed two lines: NVIDIA A100 80GB PCIe, 81920 MiB, 580.126.16. nvidia-smi -L must show 2 GPUs before you treat this as a multi-GPU host.
Lock the firewall to SSH only. Ray’s dashboard binds 127.0.0.1:8265 — do not open it to the public internet.
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
echo y | sudo ufw enable
sudo ufw status
EOF
Install verl (uv, vLLM + FSDP)
Official path: clone verl-project/verl and sync extras vllm and fsdp. That combination is one inference engine plus one training backend — do not sync vllm and sglang together. Pin whatever manage_envs.py actually installs. This VM printed verl 0.10.0.dev0, torch 2.11.0+cu130, vLLM 0.24.0, Transformers 5.5.3, Ray 2.55.1, flash-attn 2.8.3. Docs: verl installation.
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3.12-venv curl git build-essential
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
cd ~
git clone --depth 1 https://github.com/verl-project/verl.git
cd ~/verl
python3 manage_envs.py sync vllm fsdp
source .venv/bin/activate
python - <<'PY'
import importlib.metadata as md, torch
print("torch", torch.__version__, "cuda", torch.version.cuda, "gpus", torch.cuda.device_count())
for pkg in ["verl", "vllm", "transformers", "ray"]:
print(pkg, md.version(pkg))
PY
EOF
This install finished in about a minute on a clean image 184 box (wheels, not a from-source flash-attn build). If uv sync cannot find flash-attn / apex, you are off the supported platform: Linux x86_64, CPython 3.12, cu130.
Prepare GSM8K parquet
verl’s reward looks for a #### numeric answer. examples/data_preprocess/gsm8k.py writes Hugging Face openai/gsm8k to parquet with prompt, reward_model.ground_truth, and data_source.
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
export PATH="$HOME/.local/bin:$PATH"
cd ~/verl
source .venv/bin/activate
python3 examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k
python3 - <<'PY'
import pandas as pd
tr = pd.read_parquet("/home/Ubuntu/data/gsm8k/train.parquet")
te = pd.read_parquet("/home/Ubuntu/data/gsm8k/test.parquet")
print("train", len(tr), "test", len(te), "cols", list(tr.columns))
PY
EOF
This run: 7,473 train / 1,319 test, columns data_source, prompt, ability, reward_model, extra_info. Files: train.parquet 6.3 MB, test.parquet 1.2 MB.
Reward function + GRPO topology
The GSM8K scorer is a regex, not a learned RM. On this VM:
gold 1.0 # completion ends with #### 5
bad 0 # "I think the answer is five."
| Knob | This smoke | Why |
|---|---|---|
algorithm.adv_estimator=grpo |
required | Group-relative advantages; critic off |
actor_rollout_ref.rollout.n |
4 | Completions per prompt (must be > 1) |
data.train_batch_size |
8 | Prompts per step → 8 × 4 = 32 trajectories |
actor.ppo_mini_batch_size |
8 | Must divide train_batch_size * n |
data.max_response_length |
512 | Short 256-token answers rarely emit #### |
data.dataloader_num_workers |
0 | Avoids Ray teardown killing parquet workers |
rollout.gpu_memory_utilization |
0.25 (1 GPU) / 0.20 (2 GPU) | vLLM reservation. 0.4 on 80 GB reserves ~32 GB and inflates nvidia-smi |
trainer.n_gpus_per_node |
1 then 2 | Baseline, then both A100s |
trainer.total_training_steps |
10 then 20 | Smoke. Full GSM8K is total_epochs |
trainer.save_freq |
equal to step count | One 6.1 GB checkpoint, not one per step |
NCCL_P2P_DISABLE |
1 | PCIe pair, no NVLink |
PPO-with-critic would set algorithm.adv_estimator=gae and critic.model.path=…. Not this run.
Single-GPU baseline, then two GPUs
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
export PATH="$HOME/.local/bin:$PATH"
export WANDB_MODE=disabled
export NCCL_P2P_DISABLE=1
export PYTHONUNBUFFERED=1
cd ~/verl
source .venv/bin/activate
mkdir -p ~/verl-run/1gpu/ckpts ~/verl-run/2gpu/ckpts
# 1-GPU baseline — GPU 1 must stay idle
CUDA_VISIBLE_DEVICES=0 python3 -m verl.trainer.main_ppo \
algorithm.adv_estimator=grpo \
algorithm.use_kl_in_reward=False \
data.train_files=$HOME/data/gsm8k/train.parquet \
data.val_files=$HOME/data/gsm8k/test.parquet \
data.train_batch_size=8 \
data.max_prompt_length=512 \
data.max_response_length=512 \
data.filter_overlong_prompts=True \
data.truncation=error \
data.dataloader_num_workers=0 \
actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
actor_rollout_ref.model.use_remove_padding=True \
actor_rollout_ref.model.enable_gradient_checkpointing=True \
actor_rollout_ref.actor.optim.lr=1e-6 \
actor_rollout_ref.actor.ppo_mini_batch_size=8 \
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=2 \
actor_rollout_ref.actor.use_kl_loss=True \
actor_rollout_ref.actor.kl_loss_coef=0.001 \
actor_rollout_ref.actor.kl_loss_type=low_var_kl \
actor_rollout_ref.actor.entropy_coeff=0 \
actor_rollout_ref.rollout.name=vllm \
actor_rollout_ref.rollout.n=4 \
actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
actor_rollout_ref.rollout.gpu_memory_utilization=0.25 \
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=4 \
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
actor_rollout_ref.ref.fsdp_config.param_offload=True \
trainer.critic_warmup=0 \
trainer.logger='["console"]' \
trainer.project_name=verl_smoke \
trainer.experiment_name=qwen05b_grpo_1gpu \
trainer.n_gpus_per_node=1 \
trainer.nnodes=1 \
trainer.save_freq=10 \
trainer.test_freq=-1 \
trainer.val_before_train=False \
trainer.total_training_steps=10 \
trainer.default_local_dir=$HOME/verl-run/1gpu/ckpts
EOF
Same Hydra list for two GPUs, except CUDA_VISIBLE_DEVICES unset, n_gpus_per_node=2, gpu_memory_utilization=0.20, total_training_steps=20, save_freq=20, default_local_dir=$HOME/verl-run/2gpu/ckpts.
You should see Total training steps: 10 then 20, and step:N lines with timing_s/gen, actor/kl_loss, and critic/score/mean (the GRPO rule score — there is no critic). 1-GPU process wall 267 s. 2-GPU 566 s. After warmup, step wall ~10.5 s vs ~21 s. Same batch on two cards is not 2× faster on 0.5B.
critic/score/mean is the fraction of the 32 trajectories that matched #### <answer>. 1-GPU: 1 / 10 steps > 0, max 0.03125 (1/32). 2-GPU: 12 / 20 steps > 0, max 0.09375 (3/32). That is a noisy batch hit rate, not a GSM8K leaderboard.
Checkpoints on disk
verl wrote one FSDP checkpoint per run:
| Run | fsdp_config.json |
Shards |
|---|---|---|
| 1-GPU step 10 | {"FSDP_version": 1, "world_size": 1} |
model_world_size_1_rank_0.pt |
| 2-GPU step 20 | {"FSDP_version": 1, "world_size": 2} |
model_world_size_2_rank_{0,1}.pt (1,260,496,363 B each) |
Each global_step_* directory was 6.1 GB (model + optimizer). actor/huggingface/ is tokenizer + config.json, not a merged 16-bit model. Merge later with python3 -m verl.model_merger merge --backend fsdp --local_dir …/actor --target_dir …/huggingface if a server needs one Hugging Face folder.
GPU util
nvidia-smi sampled every 5 s:
| When | GPU 0 | GPU 1 |
|---|---|---|
| Idle | 0 / 81920 MiB | 0 / 81920 MiB |
| 1-GPU GRPO peak | 27233 MiB, 100% util | 0 MiB |
| 2-GPU GRPO peak | 20053 MiB, 100% util | 20055 MiB, 100% util |
Used rose on the cards that were supposed to work. gpu_memory_utilization=0.20 on 80 GB reserves ~16 GB for vLLM — that is most of the 20 GB peak, not 0.5B weights. An earlier probe with utilization 0.4 peaked at ~35.5 GB/card; do not size 7B from that number. Size the SKU from nvidia-smi, not from actor/perf/max_memory_allocated_gb.
SFT vs GRPO vs PPO
| Path | What it is | Use this post? |
|---|---|---|
| verl GRPO | Group sampling, no critic, FSDP + vLLM | Yes — this guide |
| verl PPO | GAE + critic replica | Named; not run |
| Axolotl | YAML SFT / QLoRA, torchrun DDP | Axolotl multi-GPU |
| Unsloth | FastLanguageModel SFT |
Unsloth GPU cloud |
| PEFT + bitsandbytes | Generic Hugging Face QLoRA | Existing QLoRA guide |
| DeepSpeed ZeRO | Later cluster post | Do not copy that workflow here |
| LLaMA-Factory | Later cluster post | Named only |
This VM did not run Axolotl, Unsloth, or PPO+critic side-by-side. Do not quote a speedup from this article.
VRAM / GPU tier
| Job | This article | SKU |
|---|---|---|
| 0.5B GRPO, n=4, seq 512 | Measured: 27 GB on 1× A100; ~20 GB each on 2× A100 | 48 GB can fit this smoke; we used 2× 80 GB to show FSDP |
| 7B–8B GRPO | Not run | Start on 2× A100 80 GB (this SKU) or 2× H100 |
70B / long CoT / large rollout.n |
Not run | 8× A100 SXM or 2–8× H100 — not this smoke |
| PPO + critic | Not run | Extra HBM for the value model; GRPO first |
CTA: launch gpu_2x_a100 for 7B-class GRPO and this hybrid layout. Use gpu_8x_A100_SXM4 when one node of two 80 GB cards is not enough. This smoke used gpu_2x_a100 so n_gpus_per_node=2 is real.
Measured result
Same 2× A100 PCIe 80 GB, verl 0.10.0.dev0, vLLM 0.24.0, Qwen2.5-0.5B-Instruct GRPO, openai/gsm8k (full parquet), August 27, 2026.
| Metric | 1 GPU (CUDA_VISIBLE_DEVICES=0) |
2 GPU (n_gpus_per_node=2) |
|---|---|---|
| Steps | 10 | 20 |
| Process wall | 267 s | 566 s |
| Step wall after warmup | ~10.5 s | ~21 s |
timing_s/gen (vLLM) |
~2.08 s | ~2.08 s |
| Throughput (after warmup) | ~1100–1465 tok/s | ~230–350 tok/s |
| Rule score > 0 | 1 / 10 steps (max 0.03125) | 12 / 20 steps (max 0.09375) |
Peak nvidia-smi |
27233 / 81920 MiB on GPU 0 only | 20053 / 20055 MiB on both |
| FSDP | world_size: 1 |
world_size: 2 |
| Checkpoint | ~/verl-run/1gpu/ckpts/global_step_10/ 6.1 GB |
~/verl-run/2gpu/ckpts/global_step_20/ 6.1 GB |
This is a 0.5B smoke, not an 8B overnight GRPO run and not a vs-Axolotl bake-off. Two GPUs hosted rollout and actor at the same time. Two GPUs did not shorten the 0.5B step — they proved world_size: 2. Do not quote a GSM8K test score from critic/score/mean.
Troubleshooting
SSH fails or keeps asking for a password. Image 184’s user is Ubuntu, not ubuntu. Use -o IdentitiesOnly=yes. Refresh host keys with ssh-keygen -R YOUR_VM_IP if the instance was relaunched on a reused IP.
ensurepip is not available. Image 184 needs sudo apt-get install -y python3.12-venv before any venv. This run used uv’s copy of 3.12.
No solution found for flash-attn / apex. The uv lock is cu130 / torch 2.11 / CPython 3.12 on Linux x86_64. Do not mix vllm and sglang extras.
Only GPU 0 is busy on the 2-GPU job. n_gpus_per_node is still 1, or CUDA_VISIBLE_DEVICES=0 is left set. The 1-GPU baseline is supposed to idle GPU 1. Ray on the 2-GPU job should log GPU,2.
OOM during vLLM init. Lower actor_rollout_ref.rollout.gpu_memory_utilization (this smoke used 0.20–0.25). 0.4 on an 80 GB card reserved ~32 GB and made nvidia-smi look like 0.5B needed 35 GB. Do not set utilization to 0.9 on a colocated actor.
OOM during actor update. Set ppo_micro_batch_size_per_gpu=1, enable actor.fsdp_config.param_offload=True, or add GPUs. DeepSpeed is not the lever in this post.
P2P support not detected / NCCL hang. This PCIe pair used NCCL_P2P_DISABLE=1. Do not force P2P on cards that are not NVLinked.
Disabled critic as algorithm.adv_estimator != gae. Expected for GRPO. If you wanted PPO, you launched the wrong estimator.
critic/score/mean: 0.0. The rule reward did not see #### <number> matches. Use max_response_length=512 (256 was too short on this 0.5B). Two steps is not enough — this smoke used 10 / 20. Still not a GSM8K test-set eval.
DataLoader worker is killed by signal: Killed after the last step. Seen with default dataloader_num_workers=8. Set data.dataloader_num_workers=0. The 10-step and 20-step runs with that flag exited clean.
Gated Llama 401. This guide uses ungated Qwen2.5 Instruct. Llama 3.x needs HF_TOKEN before the trainer downloads weights.
Ray dashboard on :8265. Local only. UFW should stay SSH-22. Do not ufw allow 8265.
RL Post-Training with verl on Multi-GPU VMs
Launch NVIDIA A100 instances for verl GRPO post-training. Get two 80GB cards, FSDP + vLLM on the node, and per-second billing.
Think it. Build it. Scale it.
Quick Setup Reference
# 1. Launch gpu_2x_a100, image 184
# 2. Verify two GPUs
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'nvidia-smi -L'
# 3. uv + verl (vLLM rollout, FSDP train)
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
sudo apt-get update -qq && sudo apt-get install -y python3.12-venv curl git build-essential
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
git clone --depth 1 https://github.com/verl-project/verl.git
cd ~/verl && python3 manage_envs.py sync vllm fsdp
EOF
# 4. GSM8K parquet
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
'source ~/verl/.venv/bin/activate && cd ~/verl && python3 examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k'
# 5. 1-GPU then 2-GPU GRPO (see step 6)
# CUDA_VISIBLE_DEVICES=0 n_gpus_per_node=1 total_training_steps=10
# n_gpus_per_node=2 total_training_steps=20 gpu_memory_utilization=0.20
# 6. Confirm FSDP shards
ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
'cat ~/verl-run/2gpu/ckpts/global_step_20/actor/fsdp_config.json; ls -lh ~/verl-run/2gpu/ckpts/global_step_20/actor/model_world_size_2_rank_*.pt'
Frequently Asked Questions
01What did this guide actually run?
gpu_2x_a100, image 184, verl 0.10.0.dev0, vLLM 0.24.0, Qwen/Qwen2.5-0.5B-Instruct GRPO, openai/gsm8k parquet, rollout.n=4. 10 steps on one A100 (wall 267 s, peak 27233 MiB, GPU 1 idle) then 20 steps on both (wall 566 s, peak ~20 GB both, FSDP world_size: 2). Rule score hit on 12 of 20 two-GPU steps (max 0.09375). Not a GSM8K test-set number.
02Is GRPO the same as RLHF with a reward model?
No. This smoke uses a regex on GSM8K (#### answer). A learned reward model is a different worker. PPO+critic is also different — not run here.
03Is this faster than Axolotl SFT?
Not measured here. Use Fine-Tune LLMs with Axolotl on Multi-GPU VMs for SFT. This post is RL post-training.
04Single GPU or multi-GPU?
The 0.5B smoke fits one 80 GB card (measured). verl’s HybridFlow still wants several GPUs once the model or rollout.n grows. 7B GRPO: 2× A100 (this SKU). 70B-class: 8× A100 SXM or Hopper. This 0.5B smoke is not a reason to rent eight cards — two GPUs were slower per step at the same batch.
05Should I merge the FSDP shards?
Not for the smoke. Keep model_world_size_2_rank_*.pt. verl.model_merger when a server needs one Hugging Face folder.
06Do I need DeepSpeed?
Not for this FSDP smoke. Megatron-LM is the other verl train extra (sync vllm megatron). DeepSpeed is a later cluster post.
07Do I need a Hugging Face token?
Not for Qwen2.5-0.5B-Instruct or openai/gsm8k. Gated Llama weights need HF_TOKEN.
08What’s the difference between spot and on-demand?
Spot is cheaper and can be interrupted. This run used on-demand gpu_2x_a100. Use on-demand if a long GRPO epoch cannot restart.
Recipe tested on August 27, 2026.
