Search terms: false green · monitoring lies · all clear but broken · silent failure · check passes when thing is missing · proxy metric · meta-unit · unreadable treated as zero · alert fatigue · cumulative counter
Twelve times in two weeks, a status reported healthy for something it wasn’t measuring. Not one bug twelve times — one shape, in twelve unrelated systems, across Linux, Windows, a security product, a UPS, and the system updater itself. This page is that shape, how to spot it, and what to do instead.
The individual fixes live on their own pages. This is the pattern, which outlives all of them.
A check that cannot fail is not a check. It is a green light wired to nothing.
Every instance below has the same structure: something is absent, unreadable, or unmeasured, and the absence gets scored as good rather than as unknown.
| What it claimed | What it actually measured | How it was caught |
|---|---|---|
| “Drive health: all clear” | zero drives — running as non-root, smartctl returned nothing, the device list was empty |
noticing the summary body had no rows in it |
| (nothing at all) | a healthy run and a job that never fired looked identical — success wrote no log line | the weekly digest failing to arrive; ~20 min lost on 2026-08-11 |
| Dashboard returns HTTP 200 | the web server was alive; the feature behind it was dead | trying to use the feature |
| “Plex is serving fine” | the deb install was inspected; a dormant snap was actually serving :32400 | resolving the port owner by cgroup instead of by guess |
systemctl is-active postgresql → active |
a meta-unit. Every real cluster (postgresql@VER-CLUSTER) was down |
GVM refusing to start |
power/control → on |
a knob a userspace service rewrites. The driver suspended the GPU anyway | /proc/driver/nvidia/params — the driver’s own live config |
sda … PASSED |
four of its fields were ? — a SATA SSD asked hard-disk questions it structurally cannot answer, with ${x:-0} scoring the silence as zero |
reading the ? characters instead of the word PASSED |
| “all clear” on 2 drives | correct while the backup drive is detached — but still 2 while it is attached, because a USB-enclosure drive fails smartctl -i and is skipped with no comment |
the CRC state file coming back empty |
| UPS alerting “working — test messages delivered” | the script’s logic, run by hand as root. In production upsmon runs it as user nut, which could not read the webhook — four real events since 2026-08-14 alerted no one |
pulling the mains plug and getting no Discord message |
sfc /scannow → “All files … successfully repaired” |
that the transaction committed. The same line printed four lines after two Cannot repair member file failures — identical wording on success and on failure |
reading CBS.log instead of the console summary |
| HitmanPro.Alert reporting protection disabled | that the detection features were off. hmpalert.dll was still LoadedModule[4] in every process afterward — the injected DLL never unloaded |
checking the crash dump’s module list, not the product’s own UI |
cat /var/run/reboot-required → “No such file” = “no reboot needed” |
nothing — the flag is never written on a kernel install here (helper present, but no /etc/kernel/postinst.d/ hook calls it). 6.8.0-138 sat installed-and-unbooted while the check read clean |
uname -r (137) vs newest /boot/vmlinuz-* (138) — the artifacts, not the proxy |
The UPS one is the sharpest instance on this page — see the worked example below. The reboot-required one (2026-08-21) is the purest: the check read “all clear” at the exact moment a reboot was most needed, because absence-of-flag was wired to mean good. The USB-enclosure drive is still open; see the bottom.
Run these against any check you rely on.
${x:-0}, || true, 2>/dev/null
swallowing the failure. These are where absence becomes health.postgresql.service,
umbrella targets, “overall health” fields — routinely report OK while every child is down.sudo and run in production as a service
account is a different program. This is the one that hides longest, because the test is
genuinely green — it just tested something else.sfc /scannow prints “All files … successfully repaired” either way.The single highest-value step. Break the thing deliberately and confirm the check goes red. Every check added this week was validated this way:
# GPU: does the assertion reject the broken value, not just accept the good one?
printf 'DynamicPowerManagement: 2\n' > /tmp/fake
grep -qx 'DynamicPowerManagement: 0' /tmp/fake && echo "BUG" || echo "fails correctly"
# Drive monitor: delete the temperature attribute from a fixture, re-run, confirm
# the digest title changes from a tick to a warning.
A check you have only ever seen pass is a check you have never tested.
all clear (4 of 4 drives) cannot silently drop a device. all clear can.
num() { case "${1:-}" in ''|*[!0-9]*) return 1 ;; *) return 0 ;; esac; }
req() { num "$1" || UNKNOWN+=("$d: $2 — not readable"); }
Then surface it separately, and change the headline so it cannot read as clean:
⚠️ Drive health: no faults, but 1 attribute(s) unreadable
| Weak | Strong |
|---|---|
/etc/modprobe.d/*.conf exists |
/proc/driver/nvidia/params says DynamicPowerManagement: 0 |
power/control = on |
runtime_suspended_time still 0 after 8 hours |
systemctl is-active postgresql |
pg_lsclusters shows every cluster online |
| backup archive exists | archive extracts, sentinel present, entry count sane |
| image hash matches | the restored image boots |
The driver’s own /proc entry beat every file and knob around it. Wherever a component
publishes its live state, that beats anything describing what it was told to do.
log "clean: ${#SUMMARY[@]} drive(s) checked, no problems, nothing posted"
Without this line, a healthy Tuesday and a cron job that silently died are the same event.
A cumulative counter that never resets will fire forever after one historical blip, and alert fatigue kills a channel faster than no monitoring at all. Store a baseline; alert on growth.
CRC errors rose 1 → 2 since the last check <- signal
1 lifetime CRC error, unchanged since March <- noise
Media-damage counters (reallocated / pending / uncorrectable) keep absolute thresholds — any non-zero there is real damage whenever it happened. Interface counters do not.
The media server’s freshly cloned Samsung 870 EVO, in the weekly digest:
sda Samsung SSD 870 EVO 50 0.0y ?°C realloc=0 pend=? uncorr=? PASSED
Three fields unknown, and still PASSED. The cause, in smart-health-check.sh:
for dev in $(lsblk -dno NAME,TYPE | awk '$2=="disk" && $1 !~ /^nvme/ {print $1}'); do
realloc=$(a 5); pending=$(a 197); uncorr=$(a 198); temp=$(a 194)
[ "${pending:-0}" -gt 0 ] && PROBLEMS+=(...)
Two faults compounding:
nvme* took the hard-disk path.
A SATA SSD has no attribute 197 or 198 at all, and reports temperature on 190, not
194. Three of the four questions were unanswerable by construction.${pending:-0} turned “unreadable” into “0”, and 0 passes. The absence of a measurement
became a clean bill of health.The kernel already knew:
cat /sys/block/sda/queue/rotational # 0 = SSD, 1 = spinning
Fix: branch on rotational; read 177 / 179 / 187 / 190 / 241 for SATA SSDs and
5 / 194 / 197 / 198 for spinning disks; check 199 (interface CRC) on both; and route every
expected-but-unreadable attribute into a named Could not be read block that changes the
headline.
sda Samsung SSD 870 EVO 50 0.0y 37°C SSD wear=0% rsvd=0 uncorr=0 crc=0 written=0.13 TB PASSED
Two gotchas worth keeping:
199 CRC errors point at the cable, not the platters. A failing SATA connection presents
exactly like a dying drive.063, 099). Bash reads a leading zero as
octal, so $((100 - 099)) dies with “value too great for base 8”. Force base 10: $((10#$x)).The sharpest instance of this whole pattern. The UPS notification script was tested on 2026-08-13, delivered two Discord messages, and was documented as “working.” On 2026-08-18 the mains were pulled for real. No alert arrived. The same had been true of every real event since 2026-08-14 — including a six-minute loss of communication with the UPS.
The config was flawless. NOTIFYCMD was set. Every NOTIFYFLAG carried EXEC. The script was
correct. The webhook was valid. The test was the broken part.
upsmon forks and drops privileges:
ps -eo user,pid,ppid,comm | grep upsmon
root 1985 1 upsmon # privileged parent — exists ONLY to run the shutdown
nut 1988 1985 upsmon # unprivileged child — THIS runs NOTIFYCMD
So the notification script runs as nut, and could reach neither of the two files it needs:
/var/log/ups-events.log 644 root:root nut cannot write
~/.config/discord-webhooks/power.txt 600 aztechguy nut cannot read
It exited at the webhook readability test, before ever attempting the POST.
sudo NOTIFYTYPE=ONBATT /usr/local/bin/ups-notify.sh "TEST -- simulated, no actual outage"
As root, both files are reachable. The test exercised the script in a security context production never uses. It proved the script’s logic; it could not detect that the script would be unable to run at all.
The giveaway was sitting in the log the whole time. Compare the message text:
[ONBATT] TEST — simulated power failure, no actual outage <- a human typed this
[ONBATT] UPS cyberpower@localhost on battery <- upsmon's own wording
A test whose message text you wrote yourself did not come through the real caller.
The script always wrote a local log, falling back to /tmp when the real log failed.
Its author’s stated reason was “an alert that only exists if the network survives is not a
record of what happened.” That fallback is the only reason the missed events were
recoverable four days later.
The error suppression was written wrong, and that was lucky.
echo "$LINE" >> "$LOG_FILE" 2>/dev/null || echo "$LINE" >> /tmp/ups-events.log
Bash applies redirections left to right, so >> "$LOG_FILE" fails before 2>/dev/null
takes effect — the “Permission denied” leaked to stderr and landed in the journal, naming
the exact line and the exact cause. Had the suppression worked as intended, there would have
been no evidence at all. Leave it noisy.
sudo chown root:nut /var/log/ups-events.log && sudo chmod 664 /var/log/ups-events.log
sudo install -o root -g nut -m 640 ~/.config/discord-webhooks/power.txt /etc/nut/power-webhook.txt
sudo sed -i 's#^WEBHOOK_FILE=.*#WEBHOOK_FILE="/etc/nut/power-webhook.txt"#' /usr/local/bin/ups-notify.sh
Verified by restoring mains and watching the real transition fire:
2026-08-18 06:53:58 [ONLINE] UPS cyberpower@localhost on line power | charge=46% ...
2026-08-18 06:53:58 posted OK [ONLINE]
upsmon’s own message text plus posted OK is the proof. Neither was ever present before.
Run the test the way production runs it, or you have tested a different program. Same user, same privileges, same caller, same environment.
For anything invoked by a service, the minimum honest test is:
sudo -u <the-service-account> /path/to/script # not: sudo /path/to/script
And the strongest test remains the one at the bottom of this page — cause the real event. Pull the plug. The 2026-08-13 test could never have found this; the 2026-08-18 plug-pull found it in ninety seconds.
Full detail: UPS — Reading and Managing It.
A USB-enclosure drive is skipped without comment whenever it is attached.
The SanDisk Extreme Pro holding both Clonezilla images fails smartctl -i, so || continue
drops it from the loop and it never appears in the report. This is a narrow fault, not a
broad one — that drive is deliberately kept unplugged except while copying or restoring, so
a two-drive report is correct almost all of the time.
The gap is that the omission is silent when the drive is attached — which is precisely when it is working hardest, and the only window in which its health could be observed at all. A cold-stored backup drive is inherently unmonitored; that is a property of cold storage, not a bug. What is fixable is the report claiming completeness while quietly dropping a row.
Most USB bridges need an explicit transport:
sudo smartctl -d sat -i /dev/sda # most common
sudo smartctl -d sntasmedia -i /dev/sda # some ASMedia bridges
sudo smartctl --scan-open # what smartctl thinks it can reach
If a transport works, the loop should retry with it. If none does, the drive should be listed as explicitly unmonitored rather than omitted — an absent row is the same lie in a different font.
Before trusting any check, break the thing it watches and confirm it goes red. Everything on this page is a corollary.
Related: NVIDIA GPU Hangs —
where judging by power/control instead of the driver’s own state cost a week.