Daily Host Status — the Infrastructure Half of the Morning Briefing

Daily Host Status

Search terms: host-status.sh · morning briefing infrastructure · UPS in briefing · daily infrastructure digest · disk headroom alert · reboot required · GPU RTD3 check · reachability check

Six lines appended to the 05:45 morning briefing, covering everything the trading side’s briefing doesn’t: UPS, drives, disk headroom, pending reboots, host reachability, and whether the GPU freeze fix is still in place.

Added 2026-08-16.

What it looks like

🖥️ Host
🔋 UPS: OL — battery 100%, load 10%, 62 min runtime, 121.0 V in
💽 Drives: laptop 2/2 ok (09:18) · media-server 4/4 ok (09:18)
📊 Disk: / 22% · 2TB 44% · media/ 13% · NAS 70%
🔄 Reboot required: no
🌐 Reachable: media-server ✅ · NAS ✅ · pi-hole ✅
🎮 GPU: RTD3 disabled, 0 ms suspended in 24h (0.0%)

Two files, deliberately

File Owns Changes
~/.local/bin/host-status.sh all infrastructure logic often — this is the one to edit
~/.local/bin/morning-briefing.py presentation only should never need editing again

The briefing gained exactly three things: a HOST_STATUS constant, a six-line section_host() that shells out, and one entry in fields. It learns nothing about UPS or SMART — it prints whatever the collector hands it.

def section_host():
    if not os.path.exists(HOST_STATUS):
        return f"⚠️ {HOST_STATUS} is missing"
    r = subprocess.run([HOST_STATUS], capture_output=True, text=True, timeout=30)
    out = r.stdout.strip()
    return out if out else f"⚠️ host-status.sh returned nothing (rc={r.returncode})"

("🖥️ Host", safe(section_host)),

Why split it: the briefing is the trading side’s file and runs live every morning. The infrastructure side changes far more often. Splitting means iterating on one never risks the other. The existing safe() wrapper means a broken collector produces a warning line rather than killing the whole briefing.

What each line actually reads

Runs as the normal user from the 05:45 user crontab, so nothing may require root.

Line Source Notes
UPS upsc cyberpower on the media server NUT is not installed on the laptop — the UPS is attached to the server. Fetched over the same SSH round trip as everything else remote
Drives last line of ~/drive-health.log, both hosts SMART needs root; the root cron job leaves the log, this reads it. Flags STALE past 36 h — at 05:45 the last run was yesterday 08:00, so ~22 h is normal
Disk df on /, the 2 TB vault, media server /, the NAS ⚠️ at ≥85 %
Reboot /var/run/reboot-required, both hosts needrestart is not installed here, so nothing else prompts
Reachable ping -c1 -W2 media server, NAS, Pi-hole
GPU /proc/driver/nvidia/params + runtime_suspended_time Never power/control — gpu-manager rewrites it to auto at every login, harmlessly

One SSH round trip fetches UPS, media-server disk, its reboot flag and its drive log together. Four separate calls would be slower and four times as many things to hang. Whole script runs in ~0.6 s with an 8 s hard SSH timeout, so a dead host delays the briefing rather than breaking it.

Verified by breaking it, not by watching it pass

Per When a Check Doesn’t Measure What It Claims, every line was validated by breaking the thing it watches:

Broken Output
media server unreachable 🔴 UPS: unknown — could not reach media-server and media-server ? / media/ ? — rows present, not dropped
drive log 3 days old laptop STALE (74h)
drive log shows faults laptop 3 PROBLEM (08:00)
disk over threshold ⚠️ / 22% per filesystem
RTD3 re-enabled 🔴 GPU: RTD3 is ENABLED (DynamicPowerManagement=2) — the freeze fix has been undone
no NVIDIA driver 🎮 GPU: no NVIDIA driver loaded

Reproduce any of them by copying the script and overriding the constant at the top:

sed 's/^MEDIA_HOST=.*/MEDIA_HOST="no-such-host"/' ~/.local/bin/host-status.sh > /tmp/t.sh && bash /tmp/t.sh

That fifth row is the point of the GPU line. If an NVIDIA driver update ever renames Ubuntu’s /usr/lib/modprobe.d/nvidia-runtimepm.conf and the basename collision returns, the briefing says so the next morning — instead of a frozen display saying so three weeks later. See NVIDIA GPU Hangs.

Testing without spamming Discord

morning-briefing.py posts when run — there is no dry-run flag. To exercise it safely, run a copy with both webhook paths pointed at files that do not exist; get_webhook() returns None and it prints to console only:

python3 - <<'PY'
s=open("/home/aztechguy/.local/bin/morning-briefing.py").read()
s=s.replace('discord-webhooks/briefing.txt','NEVER-POST-1')
s=s.replace('discord-webhooks/stocks.txt','NEVER-POST-2')
open("/tmp/test-brief.py","w").write(s)
PY
/home/aztechguy/Projects/calendar-automation/.venv/bin/python /tmp/test-brief.py

The collector alone is always safe — it only prints:

~/.local/bin/host-status.sh

Learned the hard way the same morning: smart-health-check.sh --digest posts, only --dry-run prints. Three test messages reached the live #drive-alerts channel before that registered. Check which flag prints before testing anything that owns a webhook.

Tuning

Constants at the top of host-status.sh:

MEDIA_HOST="media-server"
NAS_IP="10.0.0.15"
PIHOLE_IP="10.0.0.33"
DISK_WARN=85            # % used before a filesystem is flagged
SSH_TIMEOUT=8           # hard cap; the briefing must never hang on a dead host

Adding a seventh line means editing only host-status.sh. The briefing picks it up automatically.