Shutting Down a Proxmox HA Cluster Cleanly on UPS Power

When a UPS runs out of battery, every node in an HA cluster should go down in a controlled order. Getting that right on Proxmox with NUT (Network UPS Tools) is harder than it looks.

The Problem

If every node in an HA cluster is connected to upsmon, every node receives the shutdown signal at the same time. Each one then tries to migrate its HA guests to the other nodes before powering off. Those migrations never finish, and the shutdown stalls.

The Approach

We use one orchestrator on one node. It controls the order and leaves the other nodes passive:

  1. Only one node runs the shutdown logic. The node with the UPS connection (the NUT primary) calls the orchestration script. The other nodes get SHUTDOWNCMD "/bin/true" and do not power themselves off.
  2. Freeze HA first. ha-manager crm-command disarm-ha freeze stops migrations. If this command fails, the script aborts before any node goes down.
  3. Shut down the remote nodes one at a time. The script uses the Proxmox API to power off each remote node and waits until it is offline before moving on. Each wait has a timeout, so a hung node cannot block the orchestrator.
  4. The orchestrator goes last. Once the remote nodes are down, the primary powers itself off.
  5. Re-arm HA after the cluster returns. A systemd oneshot unit waits for quorum, then runs ha-manager crm-command arm-ha. The state file is removed only if arming succeeds.

The upsmon.conf settings are:

# Orchestrator (NUT primary)
SHUTDOWNCMD "/opt/bin/nut-cluster-shutdown-22039"
HOSTSYNC 15

# Other nodes
SHUTDOWNCMD "/bin/true"

You can find the scripts here: meissnerIT/ansible-role-mit.nut-pve-cluster-shutdown

NUT  PVE