Massed Compute

Connect to Target Server

SSH into the server you want to check. No VM provisioning required—this diagnostic runs on any existing Ubuntu host.

ssh YOUR_USERNAME@YOUR_VM_IP

Wait for the shell prompt. The preflight check requires no additional setup or sudo privileges.

Run Preflight Function

Paste the complete function definition and execute it. The function performs 8 checks and returns an exit code.

preflight() {
  local rc=0
  local hn ts diskroot diskhome memavail load nproc gpuok netok apps
  hn=$(hostname)
  ts=$(timedatectl show -p NTPSynchronized --value 2>/dev/null)
  diskroot=$(df --output=avail -BG / | tail -1 | tr -dc '0-9')
  diskhome=$(df --output=avail -BG "$HOME" | tail -1 | tr -dc '0-9')
  memavail=$(awk '/MemAvailable/ {printf "%d", $2/1024}' /proc/meminfo)
  load=$(awk '{print $1}' /proc/loadavg)
  nproc=$(nproc)
  echo "1. IDENTITY   PASS — host=$hn user=$(whoami) os=$(. /etc/os-release; echo $PRETTY_NAME)"
  if [ "$ts" = "yes" ]; then
    local skew
    skew=$(chronyc tracking 2>/dev/null | awk '/System time/ {print $4}' || echo "?")
    echo "2. TIME       PASS — NTP synced, skew=${skew}s tz=$(timedatectl show -p Timezone --value)"
  else
    echo "2. TIME       FAIL — NTP not synchronized"; rc=1
  fi
  if [ "$diskroot" -lt 20 ]; then echo "3. DISK /     FAIL — only ${diskroot}G free on / (need >=20G)"; rc=1
  else echo "3. DISK /     PASS — ${diskroot}G free on /"; fi
  if [ "$diskhome" -lt 50 ]; then echo "4. DISK \$HOME FAIL — only ${diskhome}G free on \$HOME (need >=50G)"; rc=1
  else echo "4. DISK \$HOME PASS — ${diskhome}G free on $HOME"; fi
  if [ "$memavail" -lt 2048 ]; then echo "5. MEM/CPU    FAIL — only ${memavail}M MemAvailable"; rc=1
  else echo "5. MEM/CPU    PASS — ${memavail}M MemAvailable, load=$load on ${nproc} CPUs"; fi
  if nvidia-smi --query-gpu=name,memory.free,utilization.gpu --format=csv,noheader >/dev/null 2>&1; then
    echo "6. GPU        PASS — $(nvidia-smi --query-gpu=name,memory.free --format=csv,noheader | head -1)"
  else
    echo "6. GPU        FAIL — nvidia-smi did not exit 0; see nvidia-smi-troubleshooting"; rc=1
  fi
  if curl -fsS --max-time 5 https://huggingface.co/ -o /dev/null 2>/dev/null; then
    echo "7. NETWORK    PASS — huggingface.co reachable"
  else
    echo "7. NETWORK    FAIL — cannot reach huggingface.co (5s timeout)"; rc=1
  fi
  apps=$(nvidia-smi --query-compute-apps=pid,process_name --format=csv,noheader 2>/dev/null)
  if [ -z "$apps" ]; then
    echo "8. WORKLOADS  PASS — no compute apps on GPU; load=$load"
  else
    foreign=$(echo "$apps" | grep -v "^$$," | head -3)
    echo "8. WORKLOADS  WARN — foreign GPU apps present: $foreign"
  fi
  return $rc
}
preflight; echo "preflight_exit=$?"

Verify Results

Check the output for preflight_exit=0 and no FAIL lines. A healthy server shows all PASS results.

1. IDENTITY   PASS — host=gpu-worker-01 user=ubuntu os=Ubuntu 24.04.4 LTS
2. TIME       PASS — NTP synced, skew=0.000123s tz=Etc/UTC
3. DISK /     PASS — 85G free on /
4. DISK $HOME PASS — 412G free on /home/ubuntu
5. MEM/CPU    PASS — 187432M MemAvailable, load=0.12 on 16 CPUs
6. GPU        PASS — NVIDIA L40, 48555 MiB
7. NETWORK    PASS — huggingface.co reachable
8. WORKLOADS  PASS — no compute apps on GPU; load=0.12
preflight_exit=0

If any check shows FAIL, address that issue before proceeding with model downloads or training runs.