Massed Compute

GPU workloads fail when nvidia-smi stops working. This guide helps you diagnose and fix common nvidia-smi failures on Ubuntu hosts with NVIDIA GPUs, including driver/library version mismatches, module loading issues, and hardware communication problems.

The diagnostic approach uses an error string lookup table to classify failures before attempting any remediation. This prevents unnecessary module reloads that can kill running CUDA workloads.

Massed Compute VM Pricing

Error String Lookup Table

Before running any commands, match the exact error output from nvidia-smi against this lookup table to determine the appropriate remediation path:

Step-by-Step Deployment

ssh YOUR_VM_IP

echo "=== nvidia-smi attempt ==="
nvidia-smi; echo "exit=$?"

echo "=== userspace presence ==="
which nvidia-smi || echo "MISSING: nvidia-smi not on PATH"
dpkg -l | grep -iE '^ii\s+(nvidia-driver|libnvidia|nvidia-utils)' | awk '{print $2, $3}'

echo "=== kernel modules ==="
lsmod | grep -E '^nvidia' || echo "MISSING: no nvidia modules loaded"

echo "=== recent kernel messages ==="
sudo dmesg -T | grep -iE 'nvidia|nvrm|fallen off the bus|Xid' | tail -50

echo "=== persistence daemon ==="
systemctl status nvidia-persistenced --no-pager | head -10 || true

echo "=== PCIe enumeration ==="
lspci -nn | grep -i nvidia || echo "MISSING: no NVIDIA device on PCIe bus"

sudo fuser -v /dev/nvidia* 2>/dev/null || true
sudo lsof /dev/nvidia* 2>/dev/null | awk 'NR>1 {print $2, $1, $9}' | sort -u
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader 2>/dev/null || true

sudo nvidia-smi -pm 1
nvidia-smi --query-gpu=persistence_mode --format=csv,noheader

# Unload in dependency order
sudo modprobe -r nvidia_uvm nvidia_modeset nvidia_drm nvidia

# Reload (takes 5-15 seconds)
sudo modprobe nvidia
sudo modprobe nvidia_uvm
sudo modprobe nvidia_modeset
sudo modprobe nvidia_drm

sudo modprobe -r nvidia_uvm
sudo modprobe nvidia_uvm

sudo systemctl restart nvidia-persistenced
systemctl status nvidia-persistenced --no-pager | head -10

nvidia-smi --query-gpu=name,driver_version --format=csv,noheader | grep -q "NVIDIA"; echo "result=$?"

lsmod | grep -E '^nvidia'

sudo dmesg -T | grep -iE 'nvidia|nvrm|Xid' | tail -10

nvidia-smi --query-gpu=driver_version --format=csv,noheader
dpkg -l | grep -E '^ii\s+nvidia-driver' | awk '{print $3}'

Troubleshooting Common Issues

Module Reload Failures

If modprobe -r fails with “Module nvidia is in use” but lsof shows no holders, the driver is wedged. A reboot is required – do not loop on remediation attempts.

Persistence Mode Failures

If nvidia-smi -pm 1 returns “Unable to set persistence mode”, the daemon isn’t running or the driver is broken. Try restarting nvidia-persistenced first, then proceed to module reload if still failing.

Recurring Failures

If nvidia-smi works for minutes then fails again after remediation, this indicates an intermittent hardware fault. Escalate to the provider – do not repeatedly attempt the same fixes.

Hardware Communication Errors

“GPU has fallen off the bus” errors in dmesg indicate hardware or thermal events that persist across power cycles. This requires provider intervention for board inspection. Reboot provides only temporary mitigation.

Memory Corruption

If ECC Uncorrectable counter shows values greater than 0, stop all GPU workloads immediately. Data written to VRAM may already be corrupt. Escalate for board replacement.

Skip All of This: Deploy with an AI Agent

This guide exists as a tested, machine-readable recipe in the Massed Compute MCP. Connect an AI agent to automatically run the diagnostic ladder, apply appropriate remediations with approval gates, and report results.

{
  "mcpServers": {
    "massed-compute": {
      "type": "http",
      "url": "https://vm.massedcompute.com/api/mcp",
      "headers": { "Authorization": "Bearer MC_TOKEN" }
    }
  }
}

Then say:

The agent matches your request against the recipe catalog, provisions the right diagnostics, runs the error lookup and verification steps above, and reports back with the result. The recipe was tested on June 10, 2026 and includes approval gates to prevent destructive operations without confirmation.

Quick Setup Reference

For rapid deployment of the diagnostic process:

1. Save error output: Copy the exact nvidia-smi failure message
2. Match lookup table: Find your error in the table to determine remediation path
3. Run diagnostics: Execute the read-only diagnostic ladder first
4. Check processes: Identify any active GPU workloads before module operations
5. Apply gated fix: Use the least invasive remediation that matches your error
6. Verify repair: Run all verification checks in sequence

Frequently Asked Questions