How to work out what froze from the logs, instead of rebooting and hoping. Written 2026-08-08 after a desktop freeze that turned out to be a GPU hang — every example below is real output from that incident.
The one question that drives everything: did the system stop logging, or did it keep logging while something else died? Answer that first and you’ve eliminated most of the search space.
# 1. Was the shutdown clean?
last -x | grep crash
# 2. Which boot do I want?
journalctl --list-boots
# 3. Did logging stop, or continue? <- the highest-value check
journalctl -b -1 -o short-iso --since "04:40" --until "05:25" \
| awk '{print substr($1,12,5)}' | uniq -c
# 4. Who went quiet / who started shouting?
journalctl -b -1 -o short-iso --since "<before>" --until "<at freeze>" \
| awk '{print $4}' | sort | uniq -c | sort -rn | head
# 5. Grep the suspect — ALWAYS with a date filter
journalctl -b -1 -o short-iso --since "YYYY-MM-DD 00:00" | grep -i <suspect>
last records boots, shutdowns and logins. It prints newest first, and on this box
wtmp goes back to Oct 2023 — so running it bare scrolls the useful part out of the
terminal buffer.
The single most useful form:
last -x | grep crash
aztechgu :0 :0 Sun Aug 2 17:59 - crash (5+11:27)
aztechgu seat0 login screen Sun Aug 2 17:59 - crash (5+11:27)
crash in the duration column means the session never ended cleanly — a hard power-off,
panic, or forced reset. That word is the fastest unclean-shutdown detector on the system.
Other forms worth knowing:
| Command | Use |
|---|---|
last -x -n 15 |
Last 15 events. Quickest sanity check. |
last -x --since "-3days" --time-format=iso |
Time-filtered, unambiguous timestamps |
last -x reboot --time-format=iso | head -5 |
Boots only |
last -x | tac |
Oldest→newest (newest at the bottom) |
--since needs util-linux ≥ 2.31; this box has 2.39.3, so it works. --time-format=iso
is worth the extra typing — the default omits the year, which gets confusing fast.
A clean reboot leaves a pair of records:
reboot system boot 6.8.0-136-generi 2026-08-02T17:59:32-07:00 still running
shutdown system down 6.8.0-136-generi 2026-08-02T17:58:24-07:00 - 2026-08-02T17:59:32
An unclean one leaves a boot with no matching shutdown, and the login session shows
crash.
journalctl --list-boots
IDX BOOT ID FIRST ENTRY LAST ENTRY
-2 9a55885d2b0b4cfeb8474f0aceae6ef5 Tue 2026-07-28 21:07:11 MST Sun 2026-08-02 17:58:25 MST
-1 1fba8c9d34e64ed38cfe6b6788aadcd5 Sun 2026-08-02 17:59:32 MST Sat 2026-08-08 05:24:45 MST
0 99c0cf9f12114bb89191976dc8672c6a Sat 2026-08-08 05:26:38 MST Sat 2026-08-08 05:33:05 MST
0 is now, -1 is the boot that died, -2 the one before. Use -b -1 for everything else.
Ordering gotcha: last is newest-first, journalctl is oldest-first — the opposite.
Add -r to reverse journalctl. Mixing these up costs more time than any other habit here.
Read the LAST ENTRY column carefully. Above, the screen froze at 04:54 but boot -1
logged until 05:24:45. Thirty minutes of logging after the freeze. That single fact
proved the kernel was alive and reframed the whole investigation.
This is the highest-value check in the whole procedure. Count log lines per minute across the incident:
journalctl -b -1 --no-pager -o short-iso \
--since "2026-08-08 04:40:00" --until "2026-08-08 05:25:00" \
| awk '{print substr($1,12,5)}' | uniq -c
substr($1,12,5) pulls HH:MM out of an ISO timestamp like
2026-08-08T04:54:39-07:00 — position 12, five characters.
04:52 229
04:53 219
04:54 229 <- last normal minute
04:55 171
04:56 135
04:57 119
05:00 125 <- new steady state, ~half the old rate
Three possible shapes, three different diagnoses:
| Shape | Means | Look at |
|---|---|---|
| Lines stop dead | Full kernel lockup or power loss | Hardware, thermals, PSU, last message before silence |
| Lines continue unchanged | Nothing actually hung — check your assumption | Was it just slow? An app, not the system? |
| Lines drop to a lower rate | Partial hang — one subsystem died | Whatever was producing the missing volume |
The drop from ~230 to ~110 said something generating roughly 100 lines/minute had stopped, while everything else carried on.
Compare the top log sources on each side of the inflection point:
# before
journalctl -b -1 --no-pager -o short-iso \
--since "2026-08-08 04:30:00" --until "2026-08-08 04:54:30" \
| awk '{print $4}' | sed 's/\[[0-9]*\]:$//;s/:$//' | sort | uniq -c | sort -rn | head -8
# after
journalctl -b -1 --no-pager -o short-iso \
--since "2026-08-08 04:56:00" --until "2026-08-08 05:24:50" \
| awk '{print $4}' | sed 's/\[[0-9]*\]:$//;s/:$//' | sort | uniq -c | sort -rn | head -8
BEFORE AFTER
2930 audit 1762 nvme
1553 nvme 627 <warn>
534 <warn> 585 pcieport
516 pcieport 345 nvidia-modeset <- absent before
293 kauditd_printk_skb 39 audit
nvidia-modeset appears only on the right. That’s the suspect.
Note on $4: with -o short-iso the format is
TIMESTAMP HOST IDENTIFIER[PID]: message, so $4 is the first word of the message,
not the identifier. For kernel messages that is usually the subsystem (nvme, pcieport,
nvidia-modeset), which is exactly what you want. For userspace it’s the first word of
prose, which is noisier. Use -o short and $5 if you want the unit name instead.
journalctl -b -1 --no-pager -o short-iso --since "2026-08-08 00:00:00" | grep -i nvidia
04:54:39 kernel: nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000c77d:0 2:2:0:4040
04:54:44 kernel: nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000c77d:0 2:2:0:4040
04:54:49 kernel: nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000c77d:0 2:2:0:4040
First at 04:54:39, every 5 seconds, 362 times, never recovering. Zero before that.
The mistake to avoid. The first attempt was
grep -i nvidia | head -12with no date filter. A boot spanning six days starts with normal driver-load messages from day one, soheadreturned Aug 2 initialisation lines and none of the Aug 8 errors — evidence that looked relevant and wasn’t. Filter by date before you truncate, every time.
Error while waiting for GPU progress means the driver submitted work and the GPU never
reported finishing. The GPU hung. The desktop froze because nothing could draw a frame —
the clock stopped because the screen stopped being redrawn, not because the machine
stopped. Everything else (journald, cron, network) kept running, which is exactly what
the Step 3 line-count showed.
That also explains why a restart fixes it and nothing is broken afterward: a GPU hang wedges the display pipeline but corrupts nothing on disk.
| Log line | Means |
|---|---|
Error while waiting for GPU progress |
GPU hang — display dead, system alive |
BUG: soft lockup - CPU#N stuck for Xs |
A CPU spun in kernel space; usually a driver |
blocked for more than 120 seconds |
Task stuck in uninterruptible I/O — storage or network FS |
Out of memory: Killed process |
OOM killer fired; check what it took |
PCIe Bus Error: severity=Correctable |
Link errors, auto-recovered — marginal link, no data loss |
PCIe Bus Error: severity=Uncorrected |
Serious. Data path failure |
mce: Machine check events logged |
CPU/memory hardware error — check mcelog |
| (nothing at all) | Power loss, hard reset, or a lockup so complete journald never flushed |
Search terms: PCIe Bus Error · severity=Correctable · RxErr · AER: Correctable error message received · nvme · pcieport · huge journal · 2.5 GB journal · SN570
Hundreds of thousands of these, continuously:
pcieport 10000:e0:1d.4: AER: Correctable error message received from 10000:e2:00.0
nvme 10000:e2:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
nvme 10000:e2:00.0: device [15b7:5025] error status/mask=00000001/0000e000
nvme 10000:e2:00.0: [ 0] RxErr (First)
Counted over a 6-day boot: 606,401 nvme + 201,668 pcieport lines — about 25 AER events a minute — and the journal had grown to 2.5 GB. Each event writes 4 lines.
for d in /sys/class/nvme/nvme*; do
n=$(basename $d)
addr=$(readlink -f $d/device | grep -oE "[0-9a-f]{4,5}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-9]" | tail -1)
echo "$n $addr $(cat $d/model)"
done
10000:e2:00.0 = WD Blue SN570 2TB — the drive holding the VMs and all backups.
D=/sys/bus/pci/devices/10000:e2:00.0
cat $D/current_link_speed $D/current_link_width # vs max_link_speed / max_link_width
cat $D/aer_dev_correctable $D/aer_dev_fatal $D/aer_dev_nonfatal
Result: link at 8.0 GT/s × 4 = its maximum (a failing link downtrains to Gen1 or x1), and
zero fatal, zero non-fatal in 2.8 years. Only RxErr. So: noisy, not failing. Nothing had
ever failed to get through.
aer_dev_correctable is a live counter, so you can measure a rate instead of guessing.
Vary one condition at a time:
| Condition | New AER errors |
|---|---|
| Idle, VMs off, 60 s | 0 |
| Sustained: 6 GB sequential read in 2 s | 0 |
| Intermittent: 40 × 4K reads, 0.7 s apart, 30 s | 38 |
D=/sys/bus/pci/devices/10000:e2:00.0
before=$(grep TOTAL_ERR_COR $D/aer_dev_correctable | awk '{print $2}')
for i in $(seq 1 40); do
dd if=<big-file-on-that-drive> of=/dev/null bs=4k count=1 skip=$((RANDOM*i)) 2>/dev/null
sleep 0.7
done
after=$(grep TOTAL_ERR_COR $D/aer_dev_correctable | awk '{print $2}')
echo "new errors: $((after-before))"
The transferable lesson. Idle was clean. Heavy load was clean. Only the pattern with gaps in it triggered the fault. When something is load-correlated, don’t just turn the volume up — vary the shape of the load. Sustained work keeps a link awake; total idle keeps it asleep; bursty work makes it transition constantly, and the transition is where things break. Same logic applies to CPU C-states, USB autosuspend, and disk spin-down.
PCIe ASPM L1 was enabled on the device. The link parked in low-power during each idle gap and threw a receiver error on every wake. The VMs’ bursty small I/O drove it ~25 times a minute.
cat /sys/bus/pci/devices/10000:e2:00.0/link/l1_aspm # 1 = enabled
Runtime, reversible, no reboot:
echo 0 | sudo tee /sys/bus/pci/devices/10000:e2:00.0/link/l1_aspm
Re-running the identical 40-read test: 38 errors → 0.
Permanent, at /etc/udev/rules.d/60-wd-sn570-aspm.rules:
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x15b7", ATTR{device}=="0x5025", ATTR{link/l1_aspm}="0"
Match on vendor:device, never the PCI address — NVMe enumeration on this box is not stable
across reboots (nvme0n1p1 and nvme1n1p1 swapped between the 2026-08-02 and 2026-08-08
boots). A hard-coded address would silently stop applying. This matches exactly one device;
the Samsung 980 PRO root drive is deliberately left alone.
Then reclaim the space:
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
Disabling L1 ASPM governs the PCIe link, not the drive. The SSD’s own runtime PM and APST
(the controller’s internal idle states) are untouched — confirm with
cat /sys/class/nvme/nvme0/device/power/control. The link PHY stays energised, costing a
fraction of a watt. No effect on NAND wear, which comes from writes. There is no
“performance mode” here; it is on/off for link napping.
Correctable means every error recovered, so no data was ever lost. But a link retraining constantly under bursty I/O is a credible mechanism for a drive appearing to drop off the bus mid-write — which is a plausible origin for the 509-day-old ext4 orphan-inode error cleared from this same drive on 2026-08-02 (see the System Update Runbook). Assumption, not proof — a March 2025 event can’t be retroactively attributed. But the mechanism is now measured rather than hypothesised.
journalctl --list-boots
shows the real window — on this box that was 2026-07-28 onward, so anything older is gone.journalctl -k limits output to kernel messages, which cuts a lot of userspace noise when
hunting hardware problems.kdump) or
watch for hardware causes — thermals, power, RAM. Nothing in the journal means journald
never got the chance to write.