Massed Compute

Launch GPU VM

# Launch via Massed Compute dashboard or API
# Product: gpu_1x_l40
# Image: 184 (Ubuntu Server 24.04 w/ Drivers)
# SSH Key: attach your public key
# Instance name: llama-factory-finetune

Wait until the VM is running and copy the SSH details.

Verify GPU Access

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader'

This run printed NVIDIA L40, 49140 MiB, 580.126.16.

Install LLaMA-Factory

Pin whatever pip actually installs. This VM printed LLaMA-Factory 0.9.6.dev0 at git 7fcf5b3, torch 2.13.0+cu130, transformers 5.8.0. Official install from source is pip install -e . plus requirements/metrics.txt. Add bitsandbytes for QLoRA (quantization_method: bnb).

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
set -euxo pipefail
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3.12-venv git
python3 -m venv ~/lf_env
~/lf_env/bin/pip install -U pip
git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git ~/LlamaFactory
cd ~/LlamaFactory
~/lf_env/bin/pip install -e .
~/lf_env/bin/pip install -r requirements/metrics.txt
~/lf_env/bin/pip install bitsandbytes
~/lf_env/bin/python - <<'PY'
import llamafactory, torch, transformers, peft, bitsandbytes
print("llamafactory", llamafactory.__version__)
print("torch", torch.__version__, "cuda", torch.version.cuda)
print("transformers", transformers.__version__)
print("peft", peft.__version__)
print("bnb", bitsandbytes.__version__)
print("gpu", torch.cuda.get_device_name(0))
PY
EOF

Upstream install notes: LLaMA-Factory Getting Started.

Register the dataset

Bundled alpaca_en_demo is a tiny local JSON. This smoke used the same public set as the Unsloth post: yahma/alpaca-cleaned. Add it to data/dataset_info.json so YAML can name dataset: alpaca_cleaned.

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'python3 -' <<'PY'
import json
from pathlib import Path
p = Path("/home/Ubuntu/LlamaFactory/data/dataset_info.json")
d = json.loads(p.read_text())
d["alpaca_cleaned"] = {
    "hf_hub_url": "yahma/alpaca-cleaned",
    "columns": {"prompt": "instruction", "query": "input", "response": "output"},
}
p.write_text(json.dumps(d, indent=2) + "\n")
print("added alpaca_cleaned")
PY

max_samples: 1024 keeps the job a smoke. The Hub repo has 51,760 train rows.

QLoRA YAML

Copy of the file this VM trained: examples/train_qlora/qwen25_05b_qlora_sft.yaml. Template for Qwen2.5 Instruct is qwen, not qwen3_nothink.

### model
model_name_or_path: Qwen/Qwen2.5-0.5B-Instruct
quantization_bit: 4
quantization_method: bnb
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 16
lora_target: all

### dataset
dataset: alpaca_cleaned
template: qwen
cutoff_len: 2048
max_samples: 1024
preprocessing_num_workers: 4
dataloader_num_workers: 0

### output
output_dir: saves/qwen25-05b/qlora/sft
logging_steps: 5
save_steps: 60
plot_loss: false
overwrite_output_dir: true
save_only_model: false
report_to: none

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 2.0e-4
num_train_epochs: 1.0
max_steps: 60
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

Effective batch is 2 × 4 = 8. lora_rank: 16 (PEFT wrote lora_alpha: 32). dataloader_num_workers: 0 avoids extra DataLoader processes on a short job. 16-bit LoRA is the same file without quantization_bit / quantization_method. DPO is a different stage: — see examples/train_lora/qwen3_lora_dpo.yaml; not this smoke.

Train

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
export PATH="$HOME/lf_env/bin:$PATH"
export WANDB_MODE=disabled
cd ~/LlamaFactory
llamafactory-cli train examples/train_qlora/qwen25_05b_qlora_sft.yaml
EOF

The trainer printed Num examples = 1,024, trainable params: 8,798,208 / 502,830,976 (1.7497%), Quantizing model to 4 bit with bitsandbytes. train_runtime 71.91 s. Mean loss 1.354. Process wall including the first Hugging Face download was ~102 s.

Save the adapter

output_dir wrote:

That is the adapter, not a merged 16-bit model. Merge is llamafactory-cli export (FAQ). Do not pass quantization_bit when merging.

Generate from the adapter

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP 'bash -s' <<'EOF'
export PATH="$HOME/lf_env/bin:$PATH"
cd ~/LlamaFactory
python3 - <<'PY'
from llamafactory.chat.chat_model import ChatModel
chat = ChatModel({
    "model_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "adapter_name_or_path": "saves/qwen25-05b/qlora/sft",
    "template": "qwen",
    "finetuning_type": "lora",
    "quantization_bit": 4,
    "quantization_method": "bnb",
    "infer_backend": "huggingface",
    "trust_remote_code": True,
})
out = chat.chat([{"role": "user", "content": "Explain LoRA in one sentence."}])[0]
print(out.response_text)
PY
EOF

This smoke returned a completion (Loaded adapter(s): saves/qwen25-05b/qlora/sft). Quality after 60 steps on 0.5B is not a product claim — it only proves generate runs on the adapter. Interactive equivalent: llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml with your paths.

Optional: LLaMA Board on localhost

ssh -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP \
  'cd ~/LlamaFactory && CUDA_VISIBLE_DEVICES="" GRADIO_SERVER_NAME=127.0.0.1 GRADIO_SERVER_PORT=7860 \
   ~/lf_env/bin/llamafactory-cli webui'

This VM answered HTTP 200 from curl http://127.0.0.1:7860/ (uvicorn / Gradio). Tunnel from your laptop: ssh -L 7860:127.0.0.1:7860 -o IdentitiesOnly=yes Ubuntu@YOUR_VM_IP.

GPU util

nvidia-smi sampled every 2 s during the 60-step job:

0.5B QLoRA does not fill a 48 GB L40. Size the SKU for the 7B–8B job you actually want, not this smoke.