System Update — Backup & Verify Runbook

System Update — Backup & Verify Runbook

How to safely apply Ubuntu/system updates on the trading machine without breaking the live Python bots. Created 2026-07-13 after a real update run (Python 3.12 security patch + others). The rule: baseline → back up → update → re-verify with the smoke test → only trust it when it reads all-green.

The full sequence (run top to bottom)

Timing comes first. Never update during market hours. The market is 6:30 AM–1:00 PM Phoenix (MST, no DST — never ET/PT), but the bot cron is */5 6-13 * * 1-5, and the 6-13 hour range fires through 1:55 PM, not 1:00. So the machine isn’t actually quiet until ~2:00 PM on a weekday — the 1 PM-hour runs are plausibly end-of-day position handling, and a reboot mid-cycle could interrupt a close. Wait until 2:00 PM on a weekday, or do it on a weekend. Run date rather than trusting the clock in your head.

# 0. BASELINE — prove the stack is green BEFORE you touch anything.
post-update-smoke-test.sh              # want: 27 passed, 0 failed (25 on the media server) — RECORD it

# 1. FRESH backups — both restore points. ';' so both run even if one hiccups.
backup-projects.sh ; backup-antigravity.sh

# 1b. VERIFY the fresh backup is restorable (not just "created"):
verify-backups.sh --always-notify      # 7z integrity test + content sentinel, one shot
#    (or by hand — see "Verify a backup is actually restorable" below)

# 2. UPDATE. Hold claude-desktop so a running remote session survives the upgrade.
sudo apt-mark hold claude-desktop
sudo apt update && sudo apt full-upgrade      # full-upgrade, NOT plain upgrade — see below
sudo apt-mark unhold claude-desktop
#    — or just use the GUI "Software Updater" if you're at the machine (preferred for a kernel batch).
#
#    Why full-upgrade: a KERNEL batch installs NEW packages (linux-image-6.8.0-NNN, its modules,
#    and the matching linux-modules-nvidia-*). Plain `apt upgrade` never installs new packages,
#    so it holds the kernel back and you'd see "The following packages have been kept back."
#    `full-upgrade` pulls them in. It can also REMOVE packages to resolve deps — on Ubuntu that's
#    normally just superseded kernels, but read the removal list before you say yes.
#    The GUI updater does the right thing here without the choice, which is why it's preferred
#    when you're sitting in front of the machine.

# 3. REBOOT if the batch includes a kernel / glibc / OpenSSL (see "Does it need a reboot?").
#    Shut the VMware guests down cleanly and PROVE they halted first:
vm-shutdown.sh && sync && sudo reboot

# ---- after reboot ----
# 4. If the KERNEL changed, VMware rebuilds vmmon/vmnet on first launch. That compile needs
#    sudo → a PHYSICAL hardware-key touch. Be present; it cannot be done remotely.

# 5. RE-VERIFY — the entire point of the exercise. Run this AFTER you are on the new kernel.
post-update-smoke-test.sh              # any check that flips PASS→FAIL vs step 0 is the update's fault

# Reboot check — do NOT trust /var/run/reboot-required on this box. A kernel install here does
# NOT create it (no hook in /etc/kernel/postinst.d/), so "No such file" is a FALSE all-clear —
# it read clean on 2026-08-21 while 6.8.0-138 sat installed and unbooted. Compare directly:
uname -r; ls -1 /boot/vmlinuz-* | sed 's#.*/vmlinuz-##' | sort -V | tail -1
#   running == newest installed  → you are on the new kernel, done.
#   they differ                  → a reboot into the newer kernel is STILL PENDING.

# 6. Bring the Kali guest back UP if it runs scheduled scans — a reboot leaves it down, and
#    the next scheduled scan silently won't fire if you forget. (Kali = the Fri/Sat scans.)

The two steps people skip, and shouldn’t:

• Step 0 and step 5 are the same command, run twice. The baseline is what makes the “after” mean anything — a lone post-update run tells you the stack is green now, but not what the update changed. Skip the baseline and you throw away the whole diagnostic; a check it silently broke reads identically to one that was never green. Record step 0’s N passed.
• Step 1 creates a fresh backup. The 7z t snippet lower down tests last night’s archive — good, but it doesn’t contain today’s work. Run the backups, then verify that archive.

Any check that flips PASS→FAIL after the update is the update’s fault, and it names the exact library or script.

Script 1 — backup-antigravity.sh

• Path: /home/aztechguy/.local/bin/backup-antigravity.sh · Cron: nightly 1:10 AM
• What it backs up: the trading code/state/credentials in ~/.gemini/antigravity/ (portfolio2_trader.py, portfolio2_state.json, fetch scripts, and the .env with Robinhood credentials) — the stuff backup-projects.sh does NOT cover. Skips the 58 MB of Gemini-CLI internals (.venv, brain, logs).
• Where it lands: encrypted 7z on the secondary drive, /media/aztechguy/<uuid>/antigravity-backup/antigravity-YYYY-MM-DD.7z, 30-day retention, same password file as the Projects backup (~/.config/backup/backup.pass).
• About that secondary drive (1.8T ext4, UUID 2a3a6a06-1836-43c3-a2ab-0e20a87ed351): as of 2026-08-02 it is in /etc/fstab and systemd mounts it ~35s after boot. Before that it was a udisks click-to-mount that only appeared at desktop login — so both backup crons (1:00 / 1:10 AM) would have silently failed on any boot where nobody had logged in yet. Never refer to this drive by /dev/ path — NVMe enumeration is not stable here; it was nvme0n1p1 before the 2026-08-02 reboot and nvme1n1p1 after, while nvme0n1p1 became the EFI partition. Use the UUID or the mount path.
• Backups all the same byte size is normal. The antigravity archives run identical lengths (e.g. 13711 bytes for days on end) because the source files are stable in size. Confirm they’re genuinely distinct with md5sum — different hashes, same size, is the expected healthy state.
• Companion: backup-projects.sh (/usr/local/bin/, → projects-backup/, nightly 1:00 AM). See Projects online backup.
• Verify a backup is actually restorable (not just “created”):

  7z t /media/aztechguy/<uuid>/projects-backup/projects-YYYY-MM-DD.7z -p"$(cat ~/.config/backup/backup.pass)"
  # want "Everything is Ok"
  

What backup-projects.sh covers (updated 2026-08-08)

Two passes into one archive:

Why two passes: both script directories end in bin, and a single 7z call fails with Duplicate filename on disk: bin, bin because they collide at the archive root. Pass 2 uses -spf2 (full path minus root) to keep them distinct. Pass 2 warns but never aborts — a failure there still leaves a valid Projects archive.

Excluded from pass 2: agy, agy.*.old, __pycache__. The two agy binaries are ~366 MB combined; including them would roughly triple the nightly archive and the MEGA upload for a redistributable you would reinstall rather than restore. With them excluded the archive grew only 144 MB → 145 MB.

Three things share the name “antigravity” — don’t confuse them:

Thing	What	Where	Backed up
backup-antigravity.sh	a script, 2 KB	~/.local/bin/	yes, since 2026-08-08
~/.gemini/antigravity/	the data it protects, 117 MB	~/.gemini/	yes, into antigravity-*.7z
agy	a 185 MB binary	~/.local/bin/	no — excluded on purpose

The script and the binary are neighbours in the same folder. Pass 2 archives that folder and then excludes the binary from it. “~/.local/bin is excluded” is wrong — one file in it is. The scripts are the irreplaceable part: post-update-smoke-test.sh encodes which venv runs which bot and which libraries each needs. That’s knowledge. agy is software you redownload.

Why this was added: until 2026-08-08 nothing outside ~/Projects and ~/.gemini/antigravity/ was backed up — verified by listing an actual archive and finding zero entries matching .local/bin or usr/local/bin. That meant post-update-smoke-test.sh, verify-backups.sh, domain-monitor.sh, morning-briefing.py, run-health-import.sh, run-buffer-automation.sh and both backup scripts themselves existed in exactly one place. A drive failure would have taken the tooling that performs and verifies the backups along with everything else.

Previous version of the script saved at ~/backup-projects.sh.bak-2026-08-08.

Check coverage after any change to it — verify the archive, never the exit code:

A=$(ls -t /media/aztechguy/<uuid>/projects-backup/*.7z | head -1)
7z l "$A" -p"$(cat ~/.config/backup/backup.pass)" | grep -E "post-update-smoke-test|backup-projects"
# both must appear

Script 2 — post-update-smoke-test.sh

• Path: /home/aztechguy/.local/bin/post-update-smoke-test.sh
• What it does (READ-ONLY): confirms the trading/health stack still works after a system or Python update. It (a) runs each venv’s interpreter, (b) imports the libraries each venv actually needs, and (c) compiles every live script. It never logs into Robinhood and never trades. Exit is informational; it prints N passed, M failed.
• Run it: post-update-smoke-test.sh

Which venv runs what (the map the smoke test checks)

All three symlink to the system /usr/bin/python3.12, so a 3.12 patch update (same minor version) is ABI-safe and needs no venv recreation. Only a minor-version jump (3.12 → 3.13) would force recreating the venvs.

Known-good output (baseline — 2026-07-13, on Python 3.12.3-1ubuntu0.15)

== venv interpreters run ==
  PASS  trading_env python
  PASS  antigravity python
  PASS  calendar-automation python
== antigravity venv imports (bots, portfolio2, watchdog, fill logger, paper sim) ==
  PASS  antigravity: import robin_stocks
  PASS  antigravity: import pandas
  PASS  antigravity: import numpy
  PASS  antigravity: import openpyxl
  PASS  antigravity: import dotenv
  PASS  antigravity: import pandas_market_calendars
== trading_env venv imports (premarket scanner) ==
  PASS  trading_env: import yfinance
  PASS  trading_env: import dotenv
== calendar-automation venv imports (calendar buffer + briefing + health drive) ==
  PASS  calendar: import googleapiclient
  PASS  calendar: import google.oauth2.credentials
  PASS  calendar: import google_auth_oauthlib
== live scripts compile (syntax intact) ==
  PASS  compile qqq_options_project/trader.py
  PASS  compile dia_options_project/trader.py
  PASS  compile qqq_0dte_project/trader.py
  PASS  compile antigravity/portfolio2_trader.py
  PASS  compile shared_trading_lib/watchdog_checkin.py
  PASS  compile shared_trading_lib/fill_logger.py
  PASS  compile shared_trading_lib/paper_volume_sim.py
  PASS  compile options_project/daily_checkin.py
  PASS  compile dashboard/dashboard_server.py
  PASS  compile health-automation/health_import.py
  PASS  compile bin/morning-briefing.py
== NVIDIA GPU runtime power management (display-freeze fix, 2026-08-15) ==
  PASS  GPU: DynamicPowerManagement=0 (RTD3 off)
  PASS  GPU: runtime-suspended <1% of uptime

RESULT: 27 passed, 0 failed.
ALL GREEN -- stack is healthy.

The two GPU checks (added 2026-08-15)

The desktop freezes of 08-08, 08-10 and 08-14 were GSP boot failures on RTD3 resume. The fix is /etc/modprobe.d/nvidia-runtimepm.conf, which only works because it shares a basename with Ubuntu’s /usr/lib/modprobe.d/nvidia-runtimepm.conf — kmod sorts config files by basename across all directories and the last assignment wins.

That makes an NVIDIA driver update the one event that can silently undo it, by renaming the vendor file to something that sorts later. A package update is therefore exactly when to check, which is why these live here rather than in a daily cron.

power/control is deliberately NOT checked. gpu-manager rewrites it to auto at every login and that is harmless once RTD3 is off at the driver. Judging by that knob is what made the original diagnosis take a week.

Both checks are guarded by [ -r /proc/driver/nvidia/params ] and print SKIP on a host with no NVIDIA driver — so the media server’s expected result stays 25, not 27.

Full detail: NVIDIA GPU Hangs.

If something fails

• A FAIL in the smoke test names the exact library or script. An import X fail after an update usually means a package needs reinstalling in that venv: <venv>/bin/pip install --force-reinstall X. A compile fail means a syntax problem (rare from a system update — more likely an unrelated edit).
• A package left “half-configured” after an update (dpkg -l <pkg> shows iF, not ii): finish it with sudo dpkg --configure -a. If it fails on a service restart (e.g. openssh-server: Cannot bind any address / process … remains running after unit stopped), a stale process is squatting the port — kill it then reconfigure:

  sudo pkill -x sshd            # (example: sshd holding the SSH port)
  sudo dpkg --configure -a
  

  This exact issue happened 2026-07-13 with openssh-server; the fix above cleared it.
• The Claude Desktop update ends the current session (it’s an apt package, claude-desktop). To keep working, either apt-mark hold it (above) or use /remote-control to continue from a phone before letting it update.
• Worst case — restore from backup: the encrypted 7z archives on the secondary drive are the fallback. Extract with 7z x <archive>.7z -p"$(cat ~/.config/backup/backup.pass)".

Run log

• 2026-08-21 — kernel bump 6.8.0-137 → 6.8.0-138 (linux-image-generic), plus libcurl (GnuTLS + OpenSSL flavours), BIND 9 client tools + libs, libheif/HEIF decoder plugins, Google Chrome, and power-profiles-daemon. No glibc or openssl-proper — the “OpenSSL flavour” entries were libcurl, not openssl, so the reboot trigger was purely the kernel. Applied via the GUI Software Updater ~14:13, after the 2 PM bot-cron close (Fri is Kali scan day). First kernel change since this runbook was written — first real exercise of the VMware rebuild.

  • Step 0 baseline: 27 passed, 0 failed. (Note: this ran on the old 137 — see the reboot trap below — so it validated the outgoing kernel, not 138.)
  • Step 1 backups: projects + antigravity ran; fresh archives verified.
  • Step 2 update: GUI updater, all applied clean, nothing held back or half-configured.
  • ⚠ THE FALSE GREEN (checks-that-dont-measure #12): after the update, cat /var/run/reboot-required returned “No such file” — read as “no reboot needed.” It was not. 6.8.0-138 was installed (dpkg log 14:13:26) while 6.8.0-137 was still running (uname, 6-day uptime). The flag is simply never created on this box: update-notifier-common is installed and the helper /usr/share/update-notifier/notify-reboot-required exists, but no hook in /etc/kernel/postinst.d/ calls it on a kernel install — so a pending-kernel reboot leaves the flag absent and the check reads clean exactly when a reboot matters most. Reliable check: uname -r vs the newest /boot/vmlinuz-*. Step 5 and the Notes section were rewritten to use it.
  • Step 3 reboot: vm-shutdown.sh brought Windows 10-purple down cleanly, then rebooted onto 138.
  • Step 4 VMware: vmmon/vmnet rebuilt automatically on first launch — modinfo vmmon vermagic now 6.8.0-138-generic. Clean, no headers mismatch.
  • Step 5 re-verify (on 138): 27 passed, 0 failed — nothing flipped PASS→FAIL. Both NVIDIA GPU checks passed on the new kernel: the RTD3 freeze fix survived. The driver stayed 580.173.02 (unchanged) — “Extra drivers for nvidia-580-open” in the batch was the per-kernel module rebuild for 138, not a driver version bump, so the vendor modprobe file was never touched.
  • Step 6 Kali: both guests back up; tonight’s scan will fire.
  • Doc changes this session: top block rewritten into the full step-0-to-6 sequence; market-hours note corrected (bot cron */5 6-13 fires to 1:55 PM, not 1:00); step 2 changed to apt full-upgrade with the reason inline; and the reboot check fixed per the false-green above.

• 2026-08-08 — not an update run. Found that no automation script was backed up: both backup scripts, the smoke test, and every cron helper live in ~/.local/bin and /usr/local/bin, neither of which was in any archive. Fixed by extending backup-projects.sh (see the coverage section above). Cost: +1 MB per nightly archive. Also on this day: the GPU-hang and NAS-throughput investigations — see Troubleshooting.

• 2026-08-02 — glibc 2.39-0ubuntu8.7→8.8, OpenSSL 3.0.13-0ubuntu3.11→3.12, Samba 4ubuntu9.6→9.7, FreeRDP, tzdata 2026b→2026c, Chrome 150→151. No kernel change. Smoke test 25 passed / 0 failed after the reboot. Notes from this run:

  • First batch to actually require a reboot (glibc/OpenSSL) — see the reboot rule below.
  • tzdata 2026c only changed Alberta and Morocco; Phoenix and New York were unaffected, so the bots’ timezone handling was never at risk. Always check the changelog rather than assuming.
  • The secondary drive was carrying a 509-day-old ext4 error (error count since last fsck: 1, orphan inode list from a cloud sync interrupted 2025-03-10). Cleared with sudo e2fsck -f while unmounted. e2fsck -fp (preen) will refuse orphan-list repair and exit with “UNEXPECTED INCONSISTENCY” — that means “needs a human,” not “drive is failing.” Re-run plain -f interactively and answer yes, then run it again until it reports no modifications.
  • smbd/nmbd were found listening on 0.0.0.0:139/445 serving only stock printer shares (browseable = no, guest ok = no, no samba user ever created — so the share never actually worked). Disabled with sudo systemctl disable --now smbd nmbd. CUPS printing is unaffected — it listens on localhost:631 only and is independent of Samba.
  • Miss to avoid next time: this runbook wasn’t read until after the update. The backup phase got skipped entirely. Read this doc first — the smoke test is better evidence than reasoning.

Notes

• Does it need a reboot? Depends what’s in the batch.
  • No reboot — app/userspace-only updates (Python, curl, zlib, openssh). Desktop bits (mutter/libinput) may want a re-login to fully apply.
  • Reboot required — anything touching glibc (libc6) or OpenSSL (libssl3t64), and always a kernel (linux-image). Running processes keep the old library mapped in memory until they restart, so the patch isn’t actually live until you reboot.
  • Do NOT rely on /var/run/reboot-required here — it is a false green for kernels. This box has neither needrestart nor a working reboot-required hook for kernel installs, so a fresh kernel leaves the flag absent and “No such file” reads as “all clear.” Confirmed 2026-08-21: 6.8.0-138 installed, flag never created, uname still 137. The reliable check is the running kernel vs the newest installed one:

    uname -r; ls -1 /boot/vmlinuz-* | sed 's#.*/vmlinuz-##' | sort -V | tail -1
    # differ → reboot still pending
    

  • Before rebooting, check apt list --upgradable | grep linux-image. No kernel change means VMware’s vmmon/vmnet won’t need rebuilding — that’s the usual thing that breaks Workstation after an update. Shut both VMs down from inside first (VM → Power → Shut Down Guest).
• Full context of the trading system this protects: Trading Bot Systems (Sections 6.5/6.6 cover the backup + fill logger).