NVIDIA GPU Hangs — Power-Management Resume Failure

NVIDIA GPU Hangs — Power-Management Resume Failure

Search terms: display frozen · clock stopped · screen locked but system alive · nvidia-modeset ERROR · GPU progress · Xid 154 · GSP boot failed · gpuPowerManagementResume · GPU requires reset · RTD3 · NVreg_DynamicPowerManagement · runtime_suspended_time

Root cause found 2026-08-10 after a second occurrence. The display freezes while the machine keeps running. The GPU suspends itself when idle and occasionally fails to wake.

Symptom

The screen stops updating — an on-screen clock freezes at the moment of failure — but the system is fine underneath. SSH works, cron fires, journald keeps logging. Only the display is dead.

Recovering it requires a reboot; nothing softer clears the GPU state.

The signature

nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000c77d:0 2:2:0:4040

repeating every 5 seconds. But that is the consequence. The cause is 5 seconds earlier:

NVRM: GPU0 GSP RPC buffer contains function 47 (UNLOADING_GUEST_DRIVER) sequence 13711
NVRM: nvCheckOkFailedNoLog: Check failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062)
NVRM: gpuPowerManagementResume: GSP boot failed at resume (bootMode 0x1): 0x62
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)

gpuPowerManagementResume: GSP boot failed at resume is the line that matters. The GPU had entered a low-power state, and the GSP — the firmware processor on Ampere-and-later cards — failed to boot when asked to wake. Xid 154 is NVIDIA’s “this GPU needs a reset” code.

Confirm the GPU is in that state:

nvidia-smi --query-gpu=name,driver_version,temperature.gpu --format=csv
# a hung GPU reports:  [GPU requires reset]

Look 5-10 seconds BEFORE the obvious error. nvidia-modeset: ERROR is the display driver noticing the GPU is gone. It says nothing about why. The NVRM lines above it name the cause, and they scroll past easily because the modeset error repeats hundreds of times and dominates any grep.

Diagnosis

# 1. Confirm the machine is alive and only the display died
journalctl -b 0 -o short-iso | tail -1        # still logging?
uptime -s                                      # no reboot?

# 2. First occurrence — should match the frozen clock
journalctl -b 0 -o short-iso | grep -i "nvidia-modeset.*ERROR" | head -1

# 3. THE CAUSE — the NVRM lines just before it
journalctl -b 0 -o short-iso | grep -iE "NVRM|Xid" | head -10

# 4. How much time is the GPU spending asleep?
cat /sys/bus/pci/devices/0000:01:00.0/power/control            # "auto" = suspends
cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_suspended_time   # milliseconds

That last number is the smoking gun. On 2026-08-10 it read 170,989,647 ms — 47.5 hours out of 48 of uptime. The GPU was asleep 99% of the time, so every wake was another chance for the GSP boot to fail.

Fix (corrected 2026-08-15 — the first two attempts silently did nothing)

Read the failure analysis below before applying anything. Two earlier fixes were “applied” and had zero effect for five days, because the check performed was “does the config file exist” rather than “is the setting in effect.”

The one that actually works

/usr/lib/modprobe.d/nvidia-runtimepm.conf (shipped by Ubuntu, owned by no package — gpu-manager generates it) contains options nvidia "NVreg_DynamicPowerManagement=0x02", which enables fine-grained RTD3. To beat it, the override must use the same basename:

sudo bash -c '
echo "options nvidia \"NVreg_DynamicPowerManagement=0x00\"" > /etc/modprobe.d/nvidia-runtimepm.conf
update-initramfs -u
modprobe -c | grep "options nvidia"      # must show 0x00 and NOTHING with 0x02
'

Then reboot — module parameters are only read at load time.

Optional belt-and-braces: re-pin power/control

/usr/bin/gpu-manager writes auto to power/control about a second after the driver binds. With RTD3 disabled at the driver this is harmless, but if you want the knob to read on as well:

sudo bash -c '
cat > /usr/local/sbin/nvidia-pm-pin <<\EOF
#!/bin/sh
for d in /sys/bus/pci/devices/*/; do
  [ "$(cat "$d/vendor" 2>/dev/null)" = "0x10de" ] || continue
  case "$(cat "$d/class" 2>/dev/null)" in 0x0300*) ;; *) continue ;; esac
  [ -w "$d/power/control" ] && echo on > "$d/power/control"
done
exit 0
EOF
chmod 755 /usr/local/sbin/nvidia-pm-pin
mkdir -p /etc/systemd/system/gpu-manager.service.d
printf "[Service]\nExecStartPost=-/usr/local/sbin/nvidia-pm-pin\n" > /etc/systemd/system/gpu-manager.service.d/10-pin-runtime-pm.conf
systemctl daemon-reload'

A drop-in on gpu-manager itself — not a standalone unit. gpu-manager.service is After=sysinit.target basic.target system.slice, so ordering a unit After=multi-user.target is not an ordering edge against it at all; it would appear to work only because plymouth-quit-wait delays multi-user by ~21 s. The drop-in runs inside gpu-manager’s own job, and re-fires on every display-manager restart.

Cost of the whole fix: slightly higher idle power draw. On a machine that lives on AC, negligible.

Why the first two fixes did nothing — three traps

1. /etc/modprobe.d/ does NOT automatically beat /usr/lib/modprobe.d/

The original fix created /etc/modprobe.d/**nvidia-power**.conf. kmod sorts config files by basename across all directories, then applies every matching options line in order — last assignment wins. nvidia-power sorts before nvidia-runtimepm, so ours was applied and then immediately overwritten. Both values were literally on the insmod command line:

insmod nvidia.ko ... NVreg_DynamicPowerManagement=0x00 "NVreg_DynamicPowerManagement=0x02"
                     ^^^^ ours                          ^^^^ vendor, last, wins

/etc only overrides /usr/lib when the FILENAME matches. A different filename in /etc doesn’t override — it just loses.

2. power/control is written by a systemd service, not a udev rule

A 99- udev rule was added to beat NVIDIA’s 71-nvidia.rules. udevadm test confirmed it fires and writes on. It still read auto after every boot, because the writer is /usr/bin/gpu-manager:

/var/log/gpu-manager.log:
  Setting power control to "auto" in /sys/bus/pci/devices/0000:01:00.0/power/control

That is a systemd service (Type=oneshot, WantedBy=gdm.service). No udev rule filename can outrank a systemd service — and it re-runs on every gdm restart, which emits no PCI uevent, so the udev rule never re-fires either.

3. The initramfs was rebuilt before the config was written

initrd.img was stamped 06:21:36 and nvidia-power.conf 06:25:58 — four minutes later. Even had the filename been right, the option wasn’t in the initramfs.

Verification — the only checks that cannot lie

grep 'DynamicPowerManagement:' /proc/driver/nvidia/params
head -1 /proc/driver/nvidia/gpus/0000:01:00.0/power
cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_suspended_time
Want Why it is trustworthy
DynamicPowerManagement: 0 the driver reporting its own live config. Not a file, not a knob
Runtime D3 status: Disabled driver confirming it will not power the GPU down
runtime_suspended_time staying at 0 the outcome itself

power/control reading auto is NOT a failure once DynamicPowerManagement: 0. The driver holds the GPU awake regardless, and gpu-manager will keep writing auto at every login. Judging by that knob is what made this take a week.

Reference numbers from this machine:

BROKEN   339,916 ms suspended of 406,000 ms uptime   (84%, six minutes after boot)
BROKEN   92% of a five-day uptime
FIXED    0 ms after 525 s uptime

An optional watcher logs every state change to ~/gpu-pm-watch.log: ~/.local/bin/gpu-pm-watch.sh. A 45-second sample is not enough — an early check showed on holding for 45 s and it had reverted within the hour.

Establishing whether it’s a pattern

wtmp holds far more history than the journal, and records unclean shutdowns:

last -x | grep crash          # "crash" in the duration column = never shut down cleanly

Historical rate on this machine, computed from ~2.8 years of wtmp: 48 distinct incidents, averaging 1.24/month in 2026 (62 raw records; bursts within 2h merged, since those are troubleshooting sessions rather than independent failures).

Against that baseline, two identical signatures 48 hours apart is a clear change, not noise. Before drawing conclusions from a single event, work out what normal looks like.

Recovering a hung GPU when the display is dead

The screen is frozen, so the desktop is unusable — but the machine is fine over SSH. Do not hard power off with VMs running.

Shut guests down from the command line via VMware Tools:

vmrun -T ws list
vmrun -T ws stop "/path/to/guest.vmx" soft

Expect this to be slow, and possibly to stall. VMware’s MKS (mouse-keyboard-screen) subsystem talks to X11, and X is wedged by the hung GPU, so the VMX process can block during display teardown. On 2026-08-10 Windows completed in ~7 minutes; Kali stalled permanently at XConnection: Closing console display.

That stall is a deadlock — VMX waits on an X server that cannot recover until the GPU resets, and the GPU cannot reset without the reboot. Waiting longer does not help.

Check whether the guest actually shut down, which is what matters for data:

tail -20 "/path/to/vmware.log"

Look for Tools: Last heartbeat ... last received Ns ago, USB devices disconnecting, and AIOMGR-S : stat — that sequence is the guest OS halting and releasing hardware. Once seen, the guest filesystem is flushed and a host reboot is safe even if VMX itself never exits.

sync && sudo reboot

Notes

  • vmrun stop ... soft blocks until the VM powers off. Give it minutes, not seconds — a 16 GB Windows guest with services to close is not quick. Run it in the background or with a generous timeout.
  • This fix addresses runtime power management. Suspend-to-RAM is a separate path (NVreg_EnableS0ixPowerManagement) and is not affected.
  • Related: Settings That Outlived Their Reason — driver defaults are a setting too, and this one was never chosen deliberately.