Massed Compute

Launch GPU VM

# Launch via Massed Compute dashboard or API
# Product: gpu_2x_h100
# Image: 184 (Ubuntu Server 24.04 w/ Drivers)
# SSH Key: attach your public key
# Instance name: axolotl-multigpu

Wait until the VM is running and copy the SSH details.

Verify GPU Access

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'nvidia-smi -L; nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader'

This run printed two lines: NVIDIA H100 PCIe, 81559 MiB, 580.126.16. nvidia-smi -L must show 2 GPUs before you treat this as a multi-GPU host.

Lock the firewall to SSH only. No public training UI is required.

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
echo y | sudo ufw enable
sudo ufw status
EOF

Install Axolotl

Pin whatever uv pip actually installs. This VM printed Axolotl 0.18.0, torch 2.12.1+cu130, PEFT 0.19.1, Transformers 5.14.1, bitsandbytes 0.49.1. Official docs: Axolotl installation. This run installed axolotl without the DeepSpeed extra — DeepSpeed configs are a later post.

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3.12-venv curl
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
export UV_TORCH_BACKEND=cu130
uv venv --python 3.12 ~/axolotl_env
source ~/axolotl_env/bin/activate
uv pip install -U pip packaging setuptools wheel ninja
uv pip install torch==2.12.1 torchvision
uv pip install --no-build-isolation axolotl
python - <<'PY'
import axolotl, torch, peft, transformers, bitsandbytes as bnb
import importlib.metadata as md
print("axolotl", md.version("axolotl"))
print("torch", torch.__version__, "cuda", torch.version.cuda, "gpus", torch.cuda.device_count())
print("peft", peft.__version__)
print("transformers", transformers.__version__)
print("bnb", bnb.__version__)
PY
axolotl --version || true
EOF

Config-file structure

Axolotl is one YAML. Model, dataset, adapter, and hyperparameters live in that file. type: alpaca matches yahma/alpaca-cleaned rows (instruction / input / output). split: train[:1024] keeps the smoke short. Full-epoch training is num_epochs: 1 with max_steps unset — not this smoke.

base_model: Qwen/Qwen2.5-0.5B-Instruct
load_in_4bit: true
adapter: qlora
datasets:
  - path: yahma/alpaca-cleaned
    type: alpaca
    split: train[:1024]
val_set_size: 0.0
output_dir: ./outputs/qlora-1gpu
sequence_len: 512
sample_packing: false
lora_r: 16
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_mlp_kernel: false
lora_qkv_kernel: false
lora_o_kernel: false
lora_embedding_kernel: false
micro_batch_size: 2
gradient_accumulation_steps: 4
max_steps: 60
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.0002
bf16: true
gradient_checkpointing: false
logging_steps: 1
warmup_steps: 5
use_wandb: false
attn_implementation: eager

load_in_4bit: true + adapter: qlora is QLoRA. r=16, lora_alpha=16. Target modules on this run: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. Trainable 8,798,208 of 502,830,976 (1.75%). sequence_len: 512 for the smoke (11 of 1,024 rows dropped as longer than 512).

Default attention (SDPA) plus Axolotl’s auto LoRA kernels produced NaN grad_norm and loss 0 after step 1 on this torch 2.12.1 / bitsandbytes 0.49.1 stack. The YAML above sets attn_implementation: eager and turns the LoRA kernels off. That is the config that trained.

Single-GPU baseline

Same YAML. Pin one card. --launcher python skips Accelerate so you are not accidentally DDP on a hidden second GPU.

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
export PATH="$HOME/.local/bin:$PATH"
source ~/axolotl_env/bin/activate
mkdir -p ~/axolotl-run
# put qlora.yml from step 4 on the VM
cd ~/axolotl-run
CUDA_VISIBLE_DEVICES=0 AXOLOTL_DO_NOT_TRACK=1 WANDB_DISABLED=true \
  axolotl train qlora.yml --launcher python --no-use-wandb --max-steps 60
EOF

This run: wall 75.75 s (train_runtime 59.69 s after dataset already cached). Loss 2.287 → 0.886 (mean 1.419). nvidia-smi peak 6335 MiB on GPU 0, 0 MiB on GPU 1.

Multi-GPU launch

Same YAML. Change output_dir so you do not overwrite the 1-GPU adapter. Axolotl’s documented launcher:

axolotl train qlora-2gpu.yml --launcher torchrun -- --nproc_per_node=2

Arguments after -- go to torchrun. This host has no GPU P2P; Axolotl logged P2P support not detected, setting NCCL_P2P_DISABLE=1 and still DDP’d (world_size: 2, ddp: true).

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
export PATH="$HOME/.local/bin:$PATH"
source ~/axolotl_env/bin/activate
cd ~/axolotl-run
sed 's|output_dir: ./outputs/qlora-1gpu|output_dir: ./outputs/qlora-2gpu|' qlora.yml > qlora-2gpu.yml
AXOLOTL_DO_NOT_TRACK=1 WANDB_DISABLED=true \
  axolotl train qlora-2gpu.yml --launcher torchrun -- --nproc_per_node=2
nvidia-smi --query-gpu=index,memory.used,utilization.gpu --format=csv
EOF

This run: wall 81.71 s (train_runtime 60.32 s). Loss 2.206 → 1.119 (mean 1.398). Peak 6707 MiB and 37% / 30% util on both GPUs. Throughput 15.92 samples/s vs 8.04 on one GPU — about 2× samples per wall-clock step, not a 2× shorter 60-step job. A 0.5B smoke does not need two H100s; the dual-card launch is the pattern you keep for 7B–70B.

FSDP and DeepSpeed ZeRO are Axolotl options for models that do not fit DDP. They are not this YAML. DeepSpeed is a later cluster post.

Checkpoints / adapter on disk

axolotl train wrote:

That is the adapter, not a merged 16-bit model. Merge is optional (axolotl merge-lora, FAQ). Paths: ~/axolotl-run/outputs/qlora-1gpu/ and ~/axolotl-run/outputs/qlora-2gpu/.

GPU util

nvidia-smi memory.used during the 60-step jobs:

Used rose on the cards that were supposed to work. 0.5B QLoRA barely taxes 80 GB. Size the SKU for the 7B–70B job you actually want, not this smoke.