Search terms: log alert · log size threshold · logrotate · disk filling up · repeated
error · same error thousands of times · retry loop · log-repeat-check.sh · silent failure ·
Address already in use
Built 2026-08-10. Alerts to the Discord bot-failures channel when a single error message repeats 50+ times since the previous run.
Script: ~/.local/bin/log-repeat-check.sh (laptop only).
dashboard_server.log reached 35 MB containing 34,151 copies of one line:
OSError: [Errno 98] Address already in use
A duplicate @reboot cron entry was racing the systemd unit for port 8080. The loser retried
every 10 seconds for 16 days. Underneath that noise, the winner was running the wrong Python
interpreter and a dashboard feature had been silently dead the entire time.
A size alert would not have helped. The disk had 679 GB free — no sensible threshold fires at 35 MB. And had one fired, the natural response is to truncate the file, which destroys the evidence without ever revealing the actual fault.
The signal was never the size. It was the same line, 34,151 times.
| Problem | Symptom | Tool |
|---|---|---|
| Logs grow without bound | disk fills, eventually | logrotate — rotate, compress, expire. No alert, no decision. |
| Something is failing on a loop | one message repeated thousands of times | this script — alert, because a human must look |
Reaching for logrotate alone is the trap: it would have quietly rotated that 35 MB away every week and nobody would ever have found the dead feature. Rotation makes the disk problem invisible, which is fine — as long as something else is watching for the fault.
log-repeat-check.sh # alert only if something is repeating (daily cron)
log-repeat-check.sh --digest # always report (weekly cron)
log-repeat-check.sh --dry-run # print locally, post nothing
Cron — the user crontab, crontab -e:
0 14 * * * /home/aztechguy/.local/bin/log-repeat-check.sh
10 8 * * 0 /home/aztechguy/.local/bin/log-repeat-check.sh --digest
Not
sudo crontab -e— the opposite ofsmart-health-check.sh. That one needs root becausesmartctltalks to hardware. This one only reads files you already own, so running it as root would grant privilege it has no use for. Decide crontab placement from what the script actually needs, not from where the other jobs happen to live.
14:00 is one hour after the market close, so a weekday run sees the entire trading day.
24 log files: every bot’s options_cron.log and scanner_cron.log, the shared_trading_lib
paper strategies, watchdog.log, token_staleness.log, the Antigravity cron logs, and
dashboard_server.log.
~/drive-health.log is deliberately excluded, even though it sits right there. It already
belongs to smart-health-check.sh, which posts it to drive-alerts. Watching it in both
places means one fault alerting twice in two rooms — which teaches you to ignore both.
One log, one owner, one channel. Duplicate alerting is not redundancy; it is noise that looks like redundancy.
All 24 logs are trading, so stock-alerts looks like the obvious room. It is the wrong one, and the reason generalises.
stock-alerts is a reporting channel. It receives the 5:30 pre-market sentiment report and
the 5:45 morning briefing — dense, expected, read-every-day traffic. A failure notice arriving
among them is a failure notice that gets skimmed past.
Every other room here is named for something being wrong: calendar-automation-issues,
backup-failures, domain-cert-monitor, workout-issues, drive-alerts. So this one went to
a new bot-failures room, webhook at ~/.config/discord-webhooks/botlogs.txt (mode 600).
Route alerts by purpose, not by subject. “It’s about trading, so it goes in the trading room” puts a rare, urgent message inside a stream of expected ones. The question to ask is “what does the reader do when this arrives?” — read it, or go fix something.
This matters most for checks that are silent for weeks. Their value comes entirely from the silence being meaningful, and silence is only meaningful in a quiet room.
install -m 600 /dev/null ~/.config/discord-webhooks/botlogs.txt
nano ~/.config/discord-webhooks/botlogs.txt # paste, save
install -m 600 /dev/null creates the file already locked down, so the secret is never briefly
world-readable in the window between touch/echo and chmod. Pasting into an editor rather
than echo-ing it also keeps it out of shell history. Verify without printing it:
ls -l ~/.config/discord-webhooks/botlogs.txt # want -rw------- and 122 bytes
grep -oE 'webhooks/[0-9]+' ~/.config/discord-webhooks/botlogs.txt # id only, no token
curl -s -o /dev/null -w '%{http_code}\n' "$(tr -d '\r\n' < ~/.config/discord-webhooks/botlogs.txt)"
A GET returns 200 for a valid webhook and 404 for a deleted one — a liveness check that
posts nothing. That same GET is what proved a rotated webhook had actually been deleted rather
than merely replaced.
Every one of these was caught by running --dry-run before the script went near cron.
The first run recorded byte offsets and analysed nothing. Without that, the very first dry-run reported a Robinhood authentication cascade in full alarming detail — a cascade that turned out to be 1,358 lines stale, long since recovered.
if [ ! -f "$KEY" ]; then
printf '%s' "$SIZE" > "$KEY" # baseline at EOF
continue # everything here predates monitoring
fi
A new monitor pointed at an old log will always find history and present it as news. Every subsequent run reads only the bytes added since the last one.
basenameThe first dry-run produced two separate alerts both headed options_cron.log. There are
four files with that name here — one per bot. Now it reports ${f#/home/aztechguy/}.
Projects/*/*.log and Projects/*/logs/*.log can match the same file, which reported it twice
and read as two independent faults. Fixed with readlink -f | sort -u.
SCANNED counted files examined; the digest described it as files that had new content. On
a quiet run it announced “25 had new content” when nothing had changed at all. Split into
SCANNED and NEWDATA.
A monitor that misdescribes its own numbers is worse than one that says nothing — you will believe it later, when you have forgotten how it was written.
Deliberately narrow. The dashboard logs roughly 8,640 HTTP 200 access lines per day, and those are not errors — a naive “repeated line” check would alert every single day and be muted within a week.
Traceback|Exception|ERROR|CRITICAL|Errno|refused|denied|Failed|FAILED|timed out|Timeout|"[A-Z]+ [^"]*" (4[0-9][0-9]|5[0-9][0-9])
Messages are then normalised so that timestamps, PIDs and line numbers collapse — otherwise the same fault at 10:01 and 10:02 counts as two distinct messages:
sed -E 's/[0-9]+/#/g; s/[[:space:]]+/ /g'
Errno 98 at 10:01:33 and Errno 98 at 10:02:41 both become Errno # at #:#:#.
Inherited wholesale from
smart-health-check.sh, because
they were paid for once already:
PATH export — cron’s default is /usr/bin:/bin and misses /usr/sbin.curl — Discord sits behind Cloudflare, which rejects
Python-urllib/3.x with HTTP 403.Verify under real cron conditions before trusting it:
env -i PATH=/usr/bin:/bin HOME=/home/aztechguy ~/.local/bin/log-repeat-check.sh --dry-run