Massed Compute

Your cloud provider runs the hardware and the network. Everything inside your VM is yours: the users, SSH, the firewall, the packages, and the Docker containers your models run in. This guide hardens both layers, the VM and its containers, then sets up automatic security patching that won’t swap your NVIDIA driver out from under a running job.

Every command here was tested on an Ubuntu 24.04 GPU VM running Docker and the NVIDIA 580 driver in November 2026. Each step starts with a quick check, because images and snapshots differ. Look at what your VM does before you change it.

At a glance

Before You Start

• Keep a second SSH session open while you change SSH or firewall settings. If a change locks you out, the open session can undo it. Only close it after a fresh login works.
• Everything below runs as your normal login user with sudo.

Part 1: Harden the VM

Step 1: Use least-privilege users

Your login user probably has sudo, often without a password. That’s fine for administering the box, but your model servers shouldn’t run as that user, and never as root. Give each long-running service its own locked-down system user:

sudo useradd --system --create-home --home-dir /srv/llm --shell /usr/sbin/nologin llm
id llm

Then run the service as that user, for example with User=llm in its systemd unit. If it gets compromised, the attacker gets a user who can’t log in and can’t sudo.

One rule that catches people out: membership in the docker group is equivalent to root. Anyone who can talk to the Docker socket can start a privileged container and mount the host’s disk. Only add trusted admins to it, and never add service users like llm. Inside containers, least privilege comes from --user (Part 2).

Step 2: SSH keys only, no root, no passwords

First, check what your SSH server actually allows:

sudo sshd -T | grep -E '^(passwordauthentication|permitrootlogin|kbdinteractiveauthentication) '

If you see passwordauthentication yes, anyone on the internet can try to guess a password. Don’t assume the main sshd_config tells the whole story. Other files in /etc/ssh/sshd_config.d/ can override it, and cloud-init and other provisioning tools commonly drop one there.

Before you turn anything off, make sure you won’t lock yourself out:

• Don’t log in as root. Run whoami. If it says root, first create a sudo user (adduser <user> && usermod -aG sudo <user>), copy /root/.ssh/authorized_keys to /home/<user>/.ssh/ (owned by that user, mode 600), and confirm you can log in as that user.
• Test your key on its own: ssh -o IdentitiesOnly=yes -i ~/.ssh/your_key <user>@<vm-ip>. The MaxAuthTries 3 setting below counts every key your SSH agent offers, so if you carry several keys, use IdentitiesOnly and -i (or IdentityFile in ~/.ssh/config).

Then add your own drop-in file:

sudo tee /etc/ssh/sshd_config.d/00-hardening.conf >/dev/null <<'EOF'
# Named 00- so it loads first. sshd keeps the first value it reads.
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
X11Forwarding no
EOF

The 00- prefix matters. Ubuntu’s sshd_config includes sshd_config.d/*.conf at the top, in name order, and for most settings sshd uses the first value it reads. A file named 99-hardening.conf would lose to any lower-numbered file that sets the same option. Test the config, restart, and confirm:

sudo sshd -t && sudo systemctl restart ssh
sudo sshd -T | grep -E '^(passwordauthentication|permitrootlogin|maxauthtries) '

maxauthtries 3
permitrootlogin no
passwordauthentication no

Now prove it from your laptop. A password-only attempt should be refused before it ever asks for a password:

ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password <user>@<vm-ip>

<user>@<vm-ip>: Permission denied (publickey).

Step 3: Turn on the firewall (ufw)

ufw ships with Ubuntu but is inactive until you enable it. First check which port SSH listens on with sudo ss -tlnp | grep sshd. The commands below assume port 22; if yours differs, use sudo ufw limit <port>/tcp instead of sudo ufw limit OpenSSH, or the firewall will block your next login.

Deny everything incoming, allow SSH, then enable it:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw limit OpenSSH
sudo ufw --force enable
sudo ufw status verbose

Status: active
Default: deny (incoming), allow (outgoing), deny (routed)

To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           LIMIT IN    Anywhere
22/tcp (OpenSSH (v6))      LIMIT IN    Anywhere (v6)

limit allows SSH but rejects an IP that opens 6 or more connections within 30 seconds. That slows down brute-force bots, but it can also block you. A script that opens several SSH connections in a row trips the limit easily, and the next login gets Connection refused for about a minute. Rsync loops, Ansible and CI jobs all do this. If you script a lot of SSH, reuse one connection with ControlMaster in your SSH config. Or allow your own IP outright: sudo ufw allow from <your-ip> to any port 22 proto tcp.

Running Docker? ufw does not protect ports that Docker publishes. That’s the most important fact in this guide, and Part 2 covers it.

Step 4: Ban brute-force attempts with fail2ban

fail2ban watches the SSH logs and temporarily bans IPs that keep failing. Check whether it’s already running with sudo fail2ban-client status sshd. If not, install it:

sudo apt install fail2ban

Add a local config so your settings survive package updates, and add your own IP to ignoreip so you can’t ban yourself:

sudo tee /etc/fail2ban/jail.local >/dev/null <<'EOF'
[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 5
# Never ban yourself. Add your own public IP here:
ignoreip = 127.0.0.1/8 ::1

[sshd]
enabled = true
backend = systemd
EOF
sudo systemctl enable fail2ban
sudo systemctl restart fail2ban
sudo fail2ban-client status sshd

On an internet-facing VM, don’t be surprised to see bans within minutes:

Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     16
`- Actions
   |- Currently banned: 1
   |- Total banned:     3

Step 5: Close ports you don’t use

List everything that’s listening, and on which address:

sudo ss -tulpn

Anything bound to 0.0.0.0 or [::] is listening on every interface, including the public one. Common surprises are a database left on its default port, a Jupyter server, an old test web server, or a helper service you never use. With ufw on, those ports are blocked from outside, but don’t rely on one layer. Stop what you don’t need, and bind the rest to 127.0.0.1:

sudo systemctl disable --now <service-name>

Part 2: Lock Down Docker

Docker skips your firewall

This is the one that bites people. When you publish a container port with -p 8080:80, Docker routes that traffic through its own iptables rules in the nat table. Those rules run before ufw’s rules, so ufw never sees the traffic. Docker’s own docs say so.

You can see it for yourself. With ufw active and set to deny all incoming traffic except SSH, start two containers:

sudo docker run -d --name web-public -p 8080:80 nginx:1.27-alpine
sudo docker run -d --name web-local  -p 127.0.0.1:8081:80 nginx:1.27-alpine

Then, from a laptop on the internet:

$ curl -s -o /dev/null -w '%{http_code}\n' http://<vm-ip>:8080/
200        <- reachable, even though ufw says "deny (incoming)"
$ nc -zv -w 4 <vm-ip> 8081
(no answer) <- the 127.0.0.1-only container is not reachable

ufw status shows nothing but SSH, and the container on 8080 serves pages to the whole internet anyway. Every vLLM, Jupyter, ComfyUI or Open WebUI container you start with -p port:port behaves the same way. Remove the test containers with sudo docker rm -f web-public web-local.

The fix: publish to localhost, connect over SSH

Bind container ports to 127.0.0.1 so they’re only reachable from inside the VM:

docker run -d -p 127.0.0.1:8000:8000 your-inference-image

Or in Docker Compose:

services:
  vllm:
    image: vllm/vllm-openai:<pinned-tag>
    ports:
      - "127.0.0.1:8000:8000"

Then reach it from your laptop through an SSH tunnel:

ssh -N -L 8000:127.0.0.1:8000 <user>@<vm-ip>
# now open http://localhost:8000 on your laptop

The tunnel is encrypted, needs your SSH key, and opens no new ports. Your laptop gets the service, and the internet doesn’t.

If a service really must be public, put an authenticating reverse proxy with TLS in front of it, publish only the proxy’s port 443, and keep the app itself on 127.0.0.1. To restrict a published port to specific source IPs, add rules to the DOCKER-USER iptables chain, which Docker evaluates before its own rules (see Docker’s packet filtering docs).

Run containers with less privilege

A container that runs as root, with every capability, is one bug away from owning the host. Tighten what you can:

docker run --rm \
  --user 1000:1000 \
  --cap-drop ALL \
  --read-only \
  --security-opt no-new-privileges \
  ubuntu:24.04 id

uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu)

Not every image works with all of these flags, so add them one at a time. Some patterns you should avoid entirely:

• --privileged, which gives the container nearly full access to the host.
• -v /var/run/docker.sock:/var/run/docker.sock, which hands the container root on the host.
• --network host, which makes the container’s ports host ports and skips Docker’s network isolation.
• Writable mounts of your model weights. Mount them read-only: -v /data/models:/models:ro.

GPU containers only need --gpus all (or --gpus device=0). That doesn’t require --privileged:

sudo docker run --rm --gpus all ubuntu:24.04 nvidia-smi --query-gpu=name,driver_version --format=csv

Keep secrets out of images and snapshots

• Never put tokens in a Dockerfile. ENV HF_TOKEN=... or COPY .env . bakes the secret into an image layer. Deleting the file in a later layer doesn’t remove it, and docker history or a pulled copy of the image can recover it.
• Pass secrets at runtime with --env-file /etc/llm/llm.env (owned by root, mode 0600) or Compose secrets:. For systemd services, use EnvironmentFile=.
• Scrub the VM before you snapshot it. Check for tokens in ~/.bash_history, notebooks, ~/.cache/huggingface/token, ~/.aws, ~/.config, registry logins in ~/.docker/config.json and /root/.docker/config.json, and webhook or API URLs in /etc (such as /etc/patch-alert.url from Part 4). Remove old keys from ~/.ssh/authorized_keys. Anyone who launches your snapshot gets everything in it.
• Remember that docker inspect shows environment variables, including ones loaded from --env-file. Anyone with Docker access can read them, which is one more reason to keep the docker group small.
• Rotate anything that leaked. If a token ever landed in an image or a public repo, revoke it. Deleting the file isn’t enough.

Use images you can trust

• Prefer official or verified-publisher images.
• Pin a version, and ideally a digest, never just latest: nginx:1.27-alpine@sha256:.... Tags can be re-pointed. In March 2026, attackers force-pushed 76 of the 77 version tags of the trivy-action GitHub Action to credential-stealing code, and published a malicious Trivy 0.69.4 release and Docker images (CVE-2026-33634). Version tags couldn’t be trusted, while references pinned to a known-good commit or digest kept pulling the real thing.
• Scan what you run. Docker Scout (docker scout cves <image>) or Trivy will list known CVEs in an image. Pinning an image keeps it stable, but it also freezes its vulnerabilities. In a Trivy scan on November 5, 2026, the older nginx:1.27-alpine image had 63 HIGH or CRITICAL CVEs. That’s why Part 3 rebuilds images on a schedule.

Part 3: Keep the VM and Containers Patched

Check whether automatic updates are on

dpkg -l unattended-upgrades | tail -1
systemctl is-enabled apt-daily.timer apt-daily-upgrade.timer
apt-config dump | grep -E '^APT::Periodic::(Update-Package-Lists|Unattended-Upgrade) '

On a stock Ubuntu Server install, unattended-upgrades is installed and installs security updates daily. Don’t assume yours does, though. It’s often off on:

• Custom images and old snapshots.
• VMs someone set up to “never change under a running job.”

Any of these results means it’s off:

• dpkg -l shows rc (removed) instead of ii.
• systemctl says masked or disabled for either timer. The timers run the daily updates; the unattended-upgrades service only makes sure a run finishes before shutdown.
• The periodic values are 0 or missing.

If you keep a VM for more than a few days, turn it on.

Turn on unattended-upgrades

sudo apt update
sudo apt install unattended-upgrades
sudo systemctl unmask unattended-upgrades      # harmless if it wasn't masked
sudo systemctl enable --now unattended-upgrades

sudo tee /etc/apt/apt.conf.d/20auto-upgrades >/dev/null <<'EOF'
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
EOF

sudo systemctl unmask apt-daily.timer apt-daily-upgrade.timer   # harmless if they weren't masked
sudo systemctl enable --now apt-daily.timer apt-daily-upgrade.timer
systemctl list-timers 'apt-daily*'

Update-Package-Lists "1" refreshes package lists daily, and Unattended-Upgrade "1" installs updates daily. The two apt-daily timers run them, each with a random delay.

Security-only or everything?

The allowed sources live in /etc/apt/apt.conf.d/50unattended-upgrades. By default only security updates are allowed:

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}";
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";
//  "${distro_id}:${distro_codename}-updates";

Adding -updates would install every bug-fix release too. On a GPU VM, keep the default (security only) and install everything else yourself during a monthly maintenance window (see below). Security updates are small, targeted fixes. Bulk updates are where surprises come from.

Also note what’s not on that list: Docker’s repository and NVIDIA’s repositories. unattended-upgrades never touches Docker Engine, the NVIDIA Container Toolkit, or anything from NVIDIA’s CUDA repo. You update those deliberately.

Put your settings in a local override file

Don’t edit 50unattended-upgrades itself; a package update can replace it. Put your changes in a file that sorts after it:

sudo tee /etc/apt/apt.conf.d/52unattended-upgrades-local >/dev/null <<'EOF'
// Local overrides. 50unattended-upgrades stays untouched.

// Never auto-upgrade the GPU driver stack. Update it by hand in a maintenance window.
Unattended-Upgrade::Package-Blacklist {
    "nvidia-";
    "libnvidia-";
    "xserver-xorg-video-nvidia";
};

// Reboot automatically when an update needs it, but only at 04:00 server time.
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";

// Clean up old kernels and unused dependencies.
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
EOF

apt-config dump | grep -E 'Unattended-Upgrade::(Package-Blacklist|Automatic-Reboot)'

Always confirm with apt-config dump. For single values, whichever file sorts last wins, and installers and other tools sometimes leave their own files in /etc/apt/apt.conf.d/. Check the merged result, not just the file you wrote.

Decide about automatic reboots

Kernel and core-library updates need a reboot to take effect. With the settings above, the VM reboots at 04:00 only when an update requires it, and the time is in the VM’s time zone (timedatectl). A reboot kills whatever is running, so choose one:

• Inference servers: allow the 04:00 reboot, and start containers with --restart unless-stopped (or restart: unless-stopped in Compose) so they come back on their own.
• Long training runs: set Automatic-Reboot "false", checkpoint regularly, and reboot by hand between runs. The alert in Part 4 tells you when a reboot is pending.

Keep needrestart from restarting Docker

Since Ubuntu 24.04, needrestart automatically restarts services that are using outdated libraries after an upgrade. If that includes docker.service or containerd.service, every container on the VM restarts with it. Tell it to leave those two alone:

sudo tee /etc/needrestart/conf.d/50-keep-containers.conf >/dev/null <<'EOF'
# Restarting dockerd/containerd restarts every container. Do it in a maintenance window instead.
$nrconf{override_rc}{qr(^docker\.service$)} = 0;
$nrconf{override_rc}{qr(^containerd\.service$)} = 0;
EOF
sudo perl -c /etc/needrestart/conf.d/50-keep-containers.conf

Add your own inference service to that list the same way if it can’t tolerate a surprise restart.

Protect the NVIDIA driver

A driver update while your job is running, or a kernel update the driver module can’t build against, is how auto-patching breaks GPU boxes. There are two layers of protection.

1. Package holds. Check what’s held, and hold your driver packages if they aren’t already:

apt-mark showhold
sudo apt-mark hold nvidia-driver-580-open nvidia-dkms-580-open   # use your driver's package names

Hold your kernel metapackages too. This is required if Automatic-Reboot is true. New kernels arrive through the security pocket. If one installs unattended and your held driver can’t build against it, the 04:00 reboot brings the VM up with no GPU (the monthly routine below shows exactly that failure). Find the metapackages and hold them:

dpkg -l 'linux-generic*' 'linux-image-generic*' 'linux-headers-generic*' 'linux-virtual*' | awk '/^ii/{print $2}'
sudo apt-mark hold linux-generic-hwe-24.04 linux-image-generic-hwe-24.04 linux-headers-generic-hwe-24.04   # use the names printed above

unattended-upgrades respects holds, so it now skips the driver and the kernel. The log shows a message like linux-generic-hwe-24.04 upgradable but fails to be marked for upgrade, which is expected.

2. The blacklist in the override file above catches anything named nvidia-* or libnvidia-* that isn’t held.

If your driver came from NVIDIA’s own CUDA repository instead of Ubuntu’s, NVIDIA documents a version-lock pin file (Pin: version 580*) that keeps every NVIDIA package on one driver branch.

The trade-off: a held kernel doesn’t get security fixes automatically. That’s the price of a stable GPU stack, so you patch the kernel and driver by hand, on your schedule (see the monthly routine below).

Two more things to know:

• Manual apt upgrade is not protected by the blacklist. The blacklist only applies to unattended-upgrades. If NVIDIA’s CUDA repo is configured on your VM, a plain sudo apt upgrade can move the NVIDIA Container Toolkit and NVIDIA’s utility libraries to new versions, sometimes onto a newer driver branch than your driver. Read the list before you press y.
• CUDA usually lives in the container, not on the host. Your container brings its own CUDA runtime and needs only a host driver new enough to support it. nvidia-smi shows the newest CUDA version your driver supports (driver 580 shows CUDA Version: 13.0), so images built for that version or older will run. Pin your image’s CUDA tag (for example nvidia/cuda:12.8.1-runtime-ubuntu24.04) so it doesn’t drift.

Patch Docker and your containers

Containers never patch themselves. The OS inside an image stays exactly as it was when the image was built.

• Rebuild and re-pull on a schedule (weekly or monthly), and whenever your base image publishes a security fix:

bash
docker compose pull
docker compose up -d
docker image prune -f

• Rebuild your own images with docker build --pull so the newest base image gets used.
• Update Docker Engine and the NVIDIA Container Toolkit in your monthly window. They come from Docker’s and NVIDIA’s repositories, which unattended-upgrades skips. (If you installed Ubuntu’s own docker.io package instead, it does get security updates automatically, and its upgrade restarts the Docker daemon.)
• Consider "live-restore": true in /etc/docker/daemon.json. With it, containers keep running while the Docker daemon restarts for an upgrade. It’s a documented Docker option; restart Docker once after adding it.

The monthly maintenance routine

This is the part that automation deliberately leaves to you: the kernel, the driver stack, Docker, and non-security updates. Once a month, between jobs:

sudo apt update
apt list --upgradable                      # read it, especially anything linux-, nvidia or docker

apt-mark showhold > ~/holds.txt            # remember what was held
sudo apt-mark unhold $(cat ~/holds.txt)    # unhold the kernel AND the driver, together
sudo apt upgrade

dkms status                                # do not reboot until this line looks right

Unhold the kernel and the driver together. Our testing shows why. We unheld only the kernel, and apt upgrade moved the HWE kernel from 6.17 to 7.0. The held driver (580.126.16) then failed to build its module against the new kernel:

Autoinstall of module nvidia/580.126.16 for kernel 7.0.0-42-generic (x86_64)
nvidia-uvm/uvm_hmm.c:84:39: error: too few arguments to function 'zone_device_page_init'
Error! Bad return status for module build on kernel: 7.0.0-42-generic (x86_64)

/boot/vmlinuz already pointed at the new kernel, so a reboot at that point would have come up with no GPU driver at all. Unholding the driver too and running apt upgrade again moved it to 580.196.07 on the same 580 branch, and the module built for both kernels:

nvidia/580.196.07, 6.17.0-14-generic, x86_64: installed
nvidia/580.196.07, 7.0.0-42-generic, x86_64: installed

Only when dkms status shows installed for the new kernel is it safe to reboot:

sudo reboot

After the reboot, check the GPU on the host and in a container, then put the holds back:

uname -r
nvidia-smi
sudo docker run --rm --gpus all ubuntu:24.04 nvidia-smi --query-gpu=name,driver_version --format=csv
sudo apt-mark hold $(cat ~/holds.txt)

All three should report the new kernel and the same driver version, on the host and inside the container. Any container started with --restart unless-stopped comes back on its own.

If dkms status shows an error and you can’t fix it, don’t reboot. Remove the new kernel so the VM boots the one that works, but simulate the removal first, because it can also take your kernel metapackages with it:

apt remove -s linux-image-<new-version>-generic linux-modules-<new-version>-generic   # -s = simulate

Read the list, then run the same command without -s. If it removed a metapackage such as linux-generic-hwe-24.04, reinstall it once a driver that supports the new kernel is out.

If nvidia-smi prints Failed to initialize NVML: Driver/library version mismatch, the driver libraries were updated but the old kernel module is still loaded. Reboot.

On RHEL, Rocky or AlmaLinux: dnf-automatic

The same idea on RHEL-family systems:

sudo dnf install dnf-automatic

In /etc/dnf/automatic.conf, set upgrade_type = security and apply_updates = yes under [commands]. Set reboot = when-needed if you want automatic reboots (the default is never). Then:

sudo systemctl enable --now dnf-automatic.timer

To keep the NVIDIA driver from moving, use dnf versionlock (from the python3-dnf-plugin-versionlock package) or NVIDIA’s driver module streams. Fedora 41 and later use dnf5, whose dnf-automatic uses the same config file but differs in some options, so check man dnf5-automatic.

Part 4: Verify It Works

Dry run. See what would be installed, without changing anything:

sudo unattended-upgrade --dry-run -v

Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, o=UbuntuESMApps,a=noble-apps-security, o=UbuntuESM,a=noble-infra-security
Initial blacklist: nvidia- libnvidia- xserver-xorg-video-nvidia
Option --dry-run given, *not* performing real actions
Packages that will be upgraded: ... openssh-server openssl ... sudo systemd ...

Check three things in the output:

• Allowed origins shows only security sources.
• Initial blacklist shows your NVIDIA patterns.
• No nvidia packages appear in the upgrade list.

Logs. Each run writes here:

sudo tail -n 30 /var/log/unattended-upgrades/unattended-upgrades.log
grep -A3 "Start-Date" /var/log/apt/history.log | tail -n 20
journalctl -u apt-daily-upgrade.service --since "2 days ago"

A healthy run on a GPU VM with held driver and kernel packages ends with lines like these:

INFO Package nvidia-driver-580-open is marked to be held back.
INFO Package linux-image-generic-hwe-24.04 is marked to be held back.
INFO Package xserver-xorg-video-nvidia-580 is blacklisted.
WARNING Found /var/run/reboot-required, rebooting
WARNING Shutdown msg: b"Reboot scheduled for Fri 2026-11-06 04:00:00 UTC, use 'shutdown -c' to cancel."

• Held: the driver and kernel were skipped.
• Blacklisted: the unheld NVIDIA package was skipped.
• Reboot: the needed reboot was scheduled for 04:00 instead of happening mid-day. To reboot sooner yourself, run sudo shutdown -c, then sudo reboot.
• Docker: with the needrestart override above, Docker kept running through the upgrade, and the containers weren’t restarted. sudo needrestart -b lists docker.service and containerd.service as waiting for a restart, which you can do at your convenience.

Is a reboot pending?

cat /var/run/reboot-required 2>/dev/null && cat /var/run/reboot-required.pkgs
sudo needrestart -b

Is only SSH reachable? Run this from your laptop, not from the VM, with your containers running:

nc -zv -w 4 <vm-ip> 22      # should succeed
nc -zv -w 4 <vm-ip> 8000    # should fail if your app is bound to 127.0.0.1

A small alert so you don’t have to remember

This script checks for a pending reboot, unattended-upgrades errors, a run that’s more than 3 days old, stopped containers or an unresponsive Docker, and a broken nvidia-smi. If anything is wrong, it posts one line to a Slack or Discord webhook. It needs jq, which isn’t installed on every Ubuntu server, so install that first:

sudo apt install -y jq

sudo tee /usr/local/sbin/patch-alert >/dev/null <<'EOF'
#!/usr/bin/env bash
# Posts a short message to a Slack/Discord webhook when this VM needs attention.
set -u
command -v jq >/dev/null || { echo "patch-alert: jq is not installed" >&2; exit 1; }
WEBHOOK_URL="$(cat /etc/patch-alert.url 2>/dev/null)"
host="$(hostname)"
msgs=()

[ -f /var/run/reboot-required ] && \
  msgs+=("reboot pending for: $(tr '\n' ' ' < /var/run/reboot-required.pkgs 2>/dev/null)")

log=/var/log/unattended-upgrades/unattended-upgrades.log
if [ -f "$log" ] && tail -n 50 "$log" | grep -q -E 'ERROR|Traceback'; then
  msgs+=("unattended-upgrades logged an error, check $log")
fi

last=$(stat -c %Y /var/lib/apt/periodic/unattended-upgrades-stamp 2>/dev/null || echo 0)
if [ $(( $(date +%s) - last )) -gt $(( 3 * 86400 )) ]; then
  msgs+=("unattended-upgrades has not run in over 3 days")
fi

if command -v docker >/dev/null; then
  if ! down=$(docker ps -a --filter status=exited --filter status=dead --format '{{.Names}}' 2>/dev/null); then
    msgs+=("docker is not responding")
  elif [ -n "$down" ]; then
    msgs+=("stopped containers: $(echo $down)")
  fi
fi

if command -v nvidia-smi >/dev/null && ! nvidia-smi >/dev/null 2>&1; then
  msgs+=("nvidia-smi is failing (driver/library mismatch? reboot)")
fi

[ ${#msgs[@]} -eq 0 ] && exit 0
text="[$host] $(printf '%s; ' "${msgs[@]}")"
if [ -n "$WEBHOOK_URL" ]; then
  # Slack reads "text", Discord reads "content". Sending both works for either.
  payload=$(jq -n --arg t "$text" '{text: $t, content: $t}')
  curl -fsS -m 10 -H 'Content-Type: application/json' -d "$payload" "$WEBHOOK_URL" >/dev/null
else
  echo "$text"
fi
EOF
sudo chmod 0755 /usr/local/sbin/patch-alert

Store your webhook URL in a root-only file. Create the file with the right permissions first, then paste the URL in the editor, so it never lands in your shell history:

sudo install -m 600 -o root -g root /dev/null /etc/patch-alert.url
sudoedit /etc/patch-alert.url        # paste the webhook URL on one line, save

Then run the script every morning with a systemd timer:

sudo tee /etc/systemd/system/patch-alert.service >/dev/null <<'EOF'
[Unit]
Description=Alert when this VM needs patching attention

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/patch-alert
EOF

sudo tee /etc/systemd/system/patch-alert.timer >/dev/null <<'EOF'
[Unit]
Description=Run patch-alert every morning

[Timer]
OnCalendar=*-*-* 08:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now patch-alert.timer
sudo /usr/local/sbin/patch-alert   # test it now

When everything is healthy, the script prints nothing and sends nothing. To test it, run a container that exits right away (sudo docker run --name crashed ubuntu:24.04 false). Your webhook then receives:

{
  "text": "[my-gpu-vm] stopped containers: crashed; ",
  "content": "[my-gpu-vm] stopped containers: crashed; "
}

Clean up with sudo docker rm crashed.

Copy-Paste Checklist

VM

• Service users for workloads; only trusted admins in sudo and docker
• /etc/ssh/sshd_config.d/00-hardening.conf: no root, no passwords
• sudo sshd -T | grep passwordauthentication → no
• Password login refused from your laptop: Permission denied (publickey)
• sudo ufw status → active, deny incoming, only SSH allowed
• sudo fail2ban-client status sshd → jail running, your IP in ignoreip
• sudo ss -tulpn → nothing listening publicly that you don’t need

Docker

• Every published port is 127.0.0.1:host:container; access over ssh -L
• No --privileged, no docker.sock mounts, no --network host
• Containers run as non-root where possible; model weights mounted :ro
• No secrets in Dockerfiles or images; --env-file with mode 0600
• Images pinned by version or digest and scanned

Patching

• unattended-upgrades installed, apt-daily* timers enabled, 20auto-upgrades set to 1
• Security-only origins; your settings in 52unattended-upgrades-local
• NVIDIA driver held (apt-mark showhold) and in Package-Blacklist
• Kernel metapackages held if Automatic-Reboot is true
• Reboot policy chosen; containers use --restart unless-stopped
• needrestart leaves docker/containerd alone
• sudo unattended-upgrade --dry-run -v shows no nvidia packages
• patch-alert.timer enabled and tested
• Monthly: kernel and driver unheld together, dkms status checked before reboot, then nvidia-smi

Frequently Asked Questions