When a Check Doesn't Measure What It Claims

When a Check Doesn’t Measure What It Claims

Search terms: false green · monitoring lies · all clear but broken · silent failure · check passes when thing is missing · proxy metric · meta-unit · unreadable treated as zero · alert fatigue · cumulative counter

Twelve times in two weeks, a status reported healthy for something it wasn’t measuring. Not one bug twelve times — one shape, in twelve unrelated systems, across Linux, Windows, a security product, a UPS, and the system updater itself. This page is that shape, how to spot it, and what to do instead.

The individual fixes live on their own pages. This is the pattern, which outlives all of them.


The shape

A check that cannot fail is not a check. It is a green light wired to nothing.

Every instance below has the same structure: something is absent, unreadable, or unmeasured, and the absence gets scored as good rather than as unknown.


The twelve

What it claimed What it actually measured How it was caught
“Drive health: all clear” zero drives — running as non-root, smartctl returned nothing, the device list was empty noticing the summary body had no rows in it
(nothing at all) a healthy run and a job that never fired looked identical — success wrote no log line the weekly digest failing to arrive; ~20 min lost on 2026-08-11
Dashboard returns HTTP 200 the web server was alive; the feature behind it was dead trying to use the feature
“Plex is serving fine” the deb install was inspected; a dormant snap was actually serving :32400 resolving the port owner by cgroup instead of by guess
systemctl is-active postgresqlactive a meta-unit. Every real cluster (postgresql@VER-CLUSTER) was down GVM refusing to start
power/controlon a knob a userspace service rewrites. The driver suspended the GPU anyway /proc/driver/nvidia/params — the driver’s own live config
sda … PASSED four of its fields were ? — a SATA SSD asked hard-disk questions it structurally cannot answer, with ${x:-0} scoring the silence as zero reading the ? characters instead of the word PASSED
“all clear” on 2 drives correct while the backup drive is detached — but still 2 while it is attached, because a USB-enclosure drive fails smartctl -i and is skipped with no comment the CRC state file coming back empty
UPS alerting “working — test messages delivered” the script’s logic, run by hand as root. In production upsmon runs it as user nut, which could not read the webhook — four real events since 2026-08-14 alerted no one pulling the mains plug and getting no Discord message
sfc /scannow“All files … successfully repaired” that the transaction committed. The same line printed four lines after two Cannot repair member file failures — identical wording on success and on failure reading CBS.log instead of the console summary
HitmanPro.Alert reporting protection disabled that the detection features were off. hmpalert.dll was still LoadedModule[4] in every process afterward — the injected DLL never unloaded checking the crash dump’s module list, not the product’s own UI
cat /var/run/reboot-required“No such file” = “no reboot needed” nothing — the flag is never written on a kernel install here (helper present, but no /etc/kernel/postinst.d/ hook calls it). 6.8.0-138 sat installed-and-unbooted while the check read clean uname -r (137) vs newest /boot/vmlinuz-* (138) — the artifacts, not the proxy

The UPS one is the sharpest instance on this page — see the worked example below. The reboot-required one (2026-08-21) is the purest: the check read “all clear” at the exact moment a reboot was most needed, because absence-of-flag was wired to mean good. The USB-enclosure drive is still open; see the bottom.


The tells

Run these against any check you rely on.

  1. Does it pass when the subject is absent? Unplug the thing, remove the file, stop the service. If the check still goes green, it is measuring its own existence.
  2. Is there a denominator? “All clear” over what — 2 drives, 4 drives, none? A count without a total cannot be wrong, which is the problem.
  3. Is there a default that turns missing into good? ${x:-0}, || true, 2>/dev/null swallowing the failure. These are where absence becomes health.
  4. Is it a proxy? A config file is not a running setting. A knob is not an outcome. An HTTP 200 is not a working feature.
  5. Is it a meta-object? Wrappers that aggregate children — postgresql.service, umbrella targets, “overall health” fields — routinely report OK while every child is down.
  6. Is silence ambiguous? If “healthy” and “never ran” produce identical output, you have no monitoring, only a habit.
  7. Did the test run in the same context production will? Same user, same privileges, same environment, same caller. A script tested with sudo and run in production as a service account is a different program. This is the one that hides longest, because the test is genuinely green — it just tested something else.
  8. Would the success message and the failure message look different? If the closing line is identical whether the work succeeded or failed, it is reporting that the code ran, not that the work happened. sfc /scannow prints “All files … successfully repaired” either way.

The counter-moves

1. Make it fail on demand, before you trust it

The single highest-value step. Break the thing deliberately and confirm the check goes red. Every check added this week was validated this way:

# GPU: does the assertion reject the broken value, not just accept the good one?
printf 'DynamicPowerManagement: 2\n' > /tmp/fake
grep -qx 'DynamicPowerManagement: 0' /tmp/fake && echo "BUG" || echo "fails correctly"

# Drive monitor: delete the temperature attribute from a fixture, re-run, confirm
# the digest title changes from a tick to a warning.

A check you have only ever seen pass is a check you have never tested.

2. Report the denominator

all clear (4 of 4 drives) cannot silently drop a device. all clear can.

3. Missing is its own state — never a passing one

num() { case "${1:-}" in ''|*[!0-9]*) return 1 ;; *) return 0 ;; esac; }
req() { num "$1" || UNKNOWN+=("$d: $2 — not readable"); }

Then surface it separately, and change the headline so it cannot read as clean:

⚠️ Drive health: no faults, but 1 attribute(s) unreadable

4. Measure the outcome, not the setting

Weak Strong
/etc/modprobe.d/*.conf exists /proc/driver/nvidia/params says DynamicPowerManagement: 0
power/control = on runtime_suspended_time still 0 after 8 hours
systemctl is-active postgresql pg_lsclusters shows every cluster online
backup archive exists archive extracts, sentinel present, entry count sane
image hash matches the restored image boots

5. Ask the subject, not the wrapper

The driver’s own /proc entry beat every file and knob around it. Wherever a component publishes its live state, that beats anything describing what it was told to do.

6. Make success leave a trace

log "clean: ${#SUMMARY[@]} drive(s) checked, no problems, nothing posted"

Without this line, a healthy Tuesday and a cron job that silently died are the same event.

7. Alert on change, not on lifetime totals

A cumulative counter that never resets will fire forever after one historical blip, and alert fatigue kills a channel faster than no monitoring at all. Store a baseline; alert on growth.

CRC errors rose 1 → 2 since the last check    <- signal
1 lifetime CRC error, unchanged since March   <- noise

Media-damage counters (reallocated / pending / uncorrectable) keep absolute thresholds — any non-zero there is real damage whenever it happened. Interface counters do not.


Worked example: a new SSD graded on questions it was never asked

The media server’s freshly cloned Samsung 870 EVO, in the weekly digest:

sda   Samsung SSD 870 EVO 50   0.0y   ?°C  realloc=0 pend=?  uncorr=?   PASSED

Three fields unknown, and still PASSED. The cause, in smart-health-check.sh:

for dev in $(lsblk -dno NAME,TYPE | awk '$2=="disk" && $1 !~ /^nvme/ {print $1}'); do
    realloc=$(a 5); pending=$(a 197); uncorr=$(a 198); temp=$(a 194)
    [ "${pending:-0}" -gt 0 ] && PROBLEMS+=(...)

Two faults compounding:

  1. It branched on the device name. Anything not called nvme* took the hard-disk path. A SATA SSD has no attribute 197 or 198 at all, and reports temperature on 190, not 194. Three of the four questions were unanswerable by construction.
  2. ${pending:-0} turned “unreadable” into “0”, and 0 passes. The absence of a measurement became a clean bill of health.

The kernel already knew:

cat /sys/block/sda/queue/rotational    # 0 = SSD, 1 = spinning

Fix: branch on rotational; read 177 / 179 / 187 / 190 / 241 for SATA SSDs and 5 / 194 / 197 / 198 for spinning disks; check 199 (interface CRC) on both; and route every expected-but-unreadable attribute into a named Could not be read block that changes the headline.

sda   Samsung SSD 870 EVO 50  0.0y  37°C  SSD  wear=0% rsvd=0 uncorr=0 crc=0 written=0.13 TB  PASSED

Two gotchas worth keeping:

  • 199 CRC errors point at the cable, not the platters. A failing SATA connection presents exactly like a dying drive.
  • smartctl’s VALUE column carries leading zeros (063, 099). Bash reads a leading zero as octal, so $((100 - 099)) dies with “value too great for base 8”. Force base 10: $((10#$x)).

Worked example: an alert that passed its test and would have been silent in a real outage

The sharpest instance of this whole pattern. The UPS notification script was tested on 2026-08-13, delivered two Discord messages, and was documented as “working.” On 2026-08-18 the mains were pulled for real. No alert arrived. The same had been true of every real event since 2026-08-14 — including a six-minute loss of communication with the UPS.

The config was flawless. NOTIFYCMD was set. Every NOTIFYFLAG carried EXEC. The script was correct. The webhook was valid. The test was the broken part.

The mechanism

upsmon forks and drops privileges:

ps -eo user,pid,ppid,comm | grep upsmon
root  1985     1  upsmon     # privileged parent — exists ONLY to run the shutdown
nut   1988  1985  upsmon     # unprivileged child — THIS runs NOTIFYCMD

So the notification script runs as nut, and could reach neither of the two files it needs:

/var/log/ups-events.log                       644 root:root       nut cannot write
~/.config/discord-webhooks/power.txt          600 aztechguy       nut cannot read

It exited at the webhook readability test, before ever attempting the POST.

Why the test passed

sudo NOTIFYTYPE=ONBATT /usr/local/bin/ups-notify.sh "TEST -- simulated, no actual outage"

As root, both files are reachable. The test exercised the script in a security context production never uses. It proved the script’s logic; it could not detect that the script would be unable to run at all.

The giveaway was sitting in the log the whole time. Compare the message text:

[ONBATT] TEST — simulated power failure, no actual outage     <- a human typed this
[ONBATT] UPS cyberpower@localhost on battery                  <- upsmon's own wording

A test whose message text you wrote yourself did not come through the real caller.

Two design decisions that saved the diagnosis

  1. The script always wrote a local log, falling back to /tmp when the real log failed. Its author’s stated reason was “an alert that only exists if the network survives is not a record of what happened.” That fallback is the only reason the missed events were recoverable four days later.

  2. The error suppression was written wrong, and that was lucky.

    echo "$LINE" >> "$LOG_FILE" 2>/dev/null || echo "$LINE" >> /tmp/ups-events.log
    

    Bash applies redirections left to right, so >> "$LOG_FILE" fails before 2>/dev/null takes effect — the “Permission denied” leaked to stderr and landed in the journal, naming the exact line and the exact cause. Had the suppression worked as intended, there would have been no evidence at all. Leave it noisy.

The fix

sudo chown root:nut /var/log/ups-events.log && sudo chmod 664 /var/log/ups-events.log
sudo install -o root -g nut -m 640 ~/.config/discord-webhooks/power.txt /etc/nut/power-webhook.txt
sudo sed -i 's#^WEBHOOK_FILE=.*#WEBHOOK_FILE="/etc/nut/power-webhook.txt"#' /usr/local/bin/ups-notify.sh

Verified by restoring mains and watching the real transition fire:

2026-08-18 06:53:58 [ONLINE] UPS cyberpower@localhost on line power | charge=46% ...
2026-08-18 06:53:58 posted OK [ONLINE]

upsmon’s own message text plus posted OK is the proof. Neither was ever present before.

The generalizable rule

Run the test the way production runs it, or you have tested a different program. Same user, same privileges, same caller, same environment.

For anything invoked by a service, the minimum honest test is:

sudo -u <the-service-account> /path/to/script    # not: sudo /path/to/script

And the strongest test remains the one at the bottom of this page — cause the real event. Pull the plug. The 2026-08-13 test could never have found this; the 2026-08-18 plug-pull found it in ninety seconds.

Full detail: UPS — Reading and Managing It.


Still open

A USB-enclosure drive is skipped without comment whenever it is attached.

The SanDisk Extreme Pro holding both Clonezilla images fails smartctl -i, so || continue drops it from the loop and it never appears in the report. This is a narrow fault, not a broad one — that drive is deliberately kept unplugged except while copying or restoring, so a two-drive report is correct almost all of the time.

The gap is that the omission is silent when the drive is attached — which is precisely when it is working hardest, and the only window in which its health could be observed at all. A cold-stored backup drive is inherently unmonitored; that is a property of cold storage, not a bug. What is fixable is the report claiming completeness while quietly dropping a row.

Most USB bridges need an explicit transport:

sudo smartctl -d sat -i /dev/sda        # most common
sudo smartctl -d sntasmedia -i /dev/sda # some ASMedia bridges
sudo smartctl --scan-open               # what smartctl thinks it can reach

If a transport works, the loop should retry with it. If none does, the drive should be listed as explicitly unmonitored rather than omitted — an absent row is the same lie in a different font.


The one-line version

Before trusting any check, break the thing it watches and confirm it goes red. Everything on this page is a corollary.

Related: NVIDIA GPU Hangs — where judging by power/control instead of the driver’s own state cost a week.