This document outlines the current production setup for four automated strategies (QQQ Weekly, DIA Weekly, and QQQ 0DTE Options Bots, plus the Portfolio 2 Tracker), the shared safety infrastructure protecting all four, the dashboard, and Roth IRA tracking.
SPY was retired from the rotation on 2026-07-08 (capital reallocated to the QQQ+DIA pair) and no longer appears below except in historical context. The one-active-at-a-time sibling gate that previously governed SPY/DIA/QQQ was also removed on 2026-07-08 – QQQ and DIA now trade fully independently and can hold positions simultaneously.
Trades QQQ (Nasdaq 100) weekly options. Originally added 2026-07-06 as a smaller, secondary bot alongside SPY; became the primary weekly strategy once SPY was retired 2026-07-08, at which point its sizing was increased to match.
718316607 (“Individual” account), Options Level 2, margin tier 2, Robinhood Gold. This is the one live trading account shared by all four bots.state.json, current as of 2026-07-11). A bump to 3 contracts is planned/scheduled for around 2026-07-30, not yet executed.return_pct >= 30.0, trader.py:396); an earlier revision misdated this as 2026-07-12 – backtested across ~30/60/90-day recent windows, the tighter target locked in gains before choppy-regime reversals gave them back; the 90-day window’s advantage survived the single-best-trade robustness check at +63.4% vs -48.1%. Scope caveat, added same day after a full 10-year 3-regime sweep of 30/35/40/45/50% targets: no target changes the long-run outcome – all five settings produce ~-99% over every full regime, the per-regime “best” target flips each time (50% design, 30% validation, 35% holdout), and every best-trade-removal check is deeply negative. The 30% setting is a regime-local fit to the current chop, not a fix or a sweet spot; further tuning of this dial is explicitly not supported by the data.)state.json before buying. daily_bias in ["calls", "both"] allows a CALL entry, daily_bias in ["puts", "both"] allows a PUT entry. "neutral" blocks both directions entirely.SIBLING_STATE_FILES = [] in code – trades independently of DIA, no longer checks or waits on it. See Section 4 (historical) for why this existed and why it was removed.pandas_market_calendars, not just day-of-week. Fails open (assumes market is open) if the calendar check itself errors.run_cycle.sh uses a flock lockfile (/tmp/options_qqq.lock)./home/aztechguy/Projects/qqq_options_project/trader.py/home/aztechguy/Projects/qqq_options_project/fetch_data.py/home/aztechguy/Projects/qqq_options_project/state.json/home/aztechguy/Projects/qqq_options_project/run_cycle.sh/home/aztechguy/Projects/qqq_options_project/options_cron.logCommands intended to update state.json’s daily_bias field (QQQ and DIA each have their own; the natural-language mapping rules live in ~/.agents/AGENTS.md):
approve puts / approve bearish -> "puts" (PUT entries only)approve calls / approve bullish -> "calls" (CALL entries only)approve both -> "both" (either direction)approve neutral / no approval -> "neutral" (no new positions opened at all)2, DIA 1.Independent instance of the same strategy as QQQ, trading DIA. All the same mechanics apply identically (live-quote exit pricing, holiday awareness, flock overlap guard).
/home/aztechguy/Projects/dia_options_project/trader.py/home/aztechguy/Projects/dia_options_project/state.json/home/aztechguy/Projects/dia_options_project/run_cycle.sh (own lockfile: /tmp/options_dia.lock)/home/aztechguy/Projects/dia_options_project/options_cron.log/home/aztechguy/Projects/dia_options_project/force_buy.py (break-glass manual re-entry tool)Note: the crontab comment for this job historically said # GLD Options Bot (DIA’s predecessor) – since corrected in the current crontab, see Section 9.
Fourth, independent live strategy. Explicitly built small-scale to gather real same-day-expiration fill data, not to be a scaled position.
expiry_str = today.strftime("%Y-%m-%d") – the opposite of the weekly bots’ same-day filter.days_to_expiry == 0 and past the 12:30 PM PT cutoff."both" and has no manual approval step – unlike the weekly bots, evaluates both call and put signals automatically every day without a daily approve command./home/aztechguy/Projects/qqq_0dte_project/trader.py/home/aztechguy/Projects/qqq_0dte_project/fetch_data.py/home/aztechguy/Projects/qqq_0dte_project/state.json/home/aztechguy/Projects/qqq_0dte_project/run_cycle.sh (own lockfile: /tmp/options_qqq_0dte.lock)options_cron.log in the same folderPreviously, before SPY, DIA, or QQQ opened a new position, each checked whether either of the other two already had an open position and skipped the entry if so, enforced in code via is_sibling_bot_active(). This section is kept for historical context only – the gate no longer exists. SIBLING_STATE_FILES = [] in both qqq_options_project/trader.py and dia_options_project/trader.py, confirmed directly in code. QQQ and DIA now trade fully independently and can hold positions simultaneously; the capital-sizing analysis behind this decision (why simultaneous QQQ+DIA positions are an acceptable, bounded risk at current contract sizes) lives outside this document.
Places real Robinhood orders directly (rh.orders.order_buy_fractional_by_price, rh.orders.order_sell_fractional_by_quantity) when its own signal fires – does not wait for manual execution and reporting back.
state.json alone)allocated_cash: 0.0), highest_price_since_buy: 77.2264.SFYF (SoFi Social 50 ETF), ~0.26 shares, ~$16 also sits in the trading account. This is not part of the tracked SPMO/SMH/TQQQ basket, not in ALLOCATION_WEIGHTS, and not traded or monitored by any bot – a static leftover from before the current rotation was finalized. Expected to show up in any positions query; not a bug or data error.time_curr_5m), always true during backlog catch-up, instead of the actual 30-minute MACD crossover bar (time_curr_30m). Let a genuinely 4-day-old crossover fire a real buy as if fresh. Fixed by comparing time_curr_30m’s date instead.time_in_force bug (2026-07-07, fixed twice): the original sell used order_sell_market() (defaults to timeInForce='gtc', invalid for fractional orders). The first fix switched functions but explicitly re-specified timeInForce="gtc" as an override, reintroducing the exact same invalid value and failing identically on the next real sell. Root cause confirmed by reading the robin_stocks library source directly: both fractional buy and sell functions default to 'gfd'. Final fix: removed the override entirely.fetch_portfolio2_data.py (found and fixed 2026-07-11): queried account_number="453789794" for both positions and cash – the retired “Agentic” account, which never supported options and was replaced by the current live account 718316607. Traced the actual consumption of that fetched data in portfolio2_trader.py before fixing: load_state() only uses the fetched positions on first-ever initialization (a no-op here, state.json already exists), and the fetched cash value is written to a scratch file but never read by the trading logic – confirmed inert, not a live trading bug, before correcting to 718316607. Verified post-fix: now returns real positions (TQQQ, SFYF) and a real cash balance.shares_held, allocated_cash) before confirming a Robinhood order actually succeeded, with no rollback on failure. Worked out fine by coincidence of price staying flat on both past incidents; the underlying pattern (optimistic state update, no rollback) is still there./home/aztechguy/.gemini/antigravity/portfolio2_trader.py/home/aztechguy/.gemini/antigravity/fetch_portfolio2_data.py/home/aztechguy/.gemini/antigravity/portfolio2_state.json/home/aztechguy/.gemini/antigravity/portfolio2_cron.log/home/aztechguy/Projects/shared_trading_lib/, imported by all four bots. Built directly in response to a real incident (Section 8), scoped deliberately to what fits a locally-run, cron-based, single-account setup.
robinhood_login_lock.py)All four bots read/write the same ~/.tokens/robinhood.pickle. Previously each bot called rh.login() independently with no coordination – if the shared token went stale, all four could simultaneously trigger their own MFA challenge and hammer Robinhood’s push-status-polling endpoint at once (Section 8). login_with_lock() wraps every login call across all bots in a flock-based mutex (/tmp/robinhood_login.lock, 90s bounded wait) so only one bot is ever inside the login/MFA flow at a time. Normal case: negligible overhead (cached-session fast path returns almost instantly). Stale-token case: one bot does the real MFA login (one human phone approval), writes a fresh pickle, and every other bot then succeeds immediately off the now-fresh token with no MFA prompt of its own. Verified with a real fork test (child process genuinely blocked for the exact duration the lock was held) and a real live login through the wrapper.
Used by: qqq_options_project/{fetch_data.py,trader.py}, dia_options_project/{fetch_data.py,trader.py,force_buy.py}, qqq_0dte_project/{fetch_data.py,trader.py}, antigravity/{fetch_portfolio2_data.py,portfolio2_trader.py}, options_project/daily_checkin.py, shared_trading_lib/watchdog_checkin.py.
kill_switch.py, toggle_kill_switch.py)One shared flag file (shared_trading_lib/kill_switch.json), checked immediately before every order (buy and sell) in all four bots. Fails closed: a missing or corrupted flag file is treated as “not armed,” never as “go.” No code deploy or cron edit needed to flip it:
VENV=/home/aztechguy/.gemini/antigravity-cli/brain/23d9e61b-4566-4487-9840-0d0b4ef78445/.venv/bin/python3
$VENV /home/aztechguy/Projects/shared_trading_lib/toggle_kill_switch.py safe "reason here"
$VENV /home/aztechguy/Projects/shared_trading_lib/toggle_kill_switch.py arm "reason here"
Default/current state: armed.
daily_drawdown_guard.py)Blocks new entries only (existing positions still get managed/exited normally) when account 718316607’s equity is down 5% or more versus Robinhood’s own adjusted_equity_previous_close field – the same number the app displays as “Today.” Computed fresh at decision time on every check, no separate cache/scheduler. Fails open (does not pause) if the equity fetch itself errors – an API hiccup here shouldn’t silently freeze trading; the kill switch is the correct tool for that. Verified against the real account both ways: at a real -4.35% reading it correctly did not trigger against the default -5% threshold, and correctly triggered when tested against a tighter -3% threshold.
watchdog_checkin.py, added 2026-07-11)Distinct from the daily 6 AM check-in (Section 7) on purpose: that one sends a full status embed once a day, which is fine pre-market but would be noise at higher frequency. This one checks nothing but the essentials – can it log in via the exact same shared lock all four bots use, and can it read account positions – and says nothing on success, alerting to Discord immediately on failure. Runs every 30 minutes, 6 AM-1 PM PT, weekdays, holiday-aware.
Built specifically because the 6 AM-only check couldn’t have caught the 2026-07-10 outage (which started mid-session, after that morning’s check had already passed) and because 0DTE positions have a same-day theta-clock deadline (~12:30 PM PT) that a multi-hour blind spot doesn’t leave enough room to react to manually. At 30-minute granularity, worst-case detection lag is well inside that deadline.
Verified: real successful run (silent, no Discord post) and a mocked-failure run (confirmed the correct alert content is built and would be sent) before deploying.
Crash detector added 2026-07-13 (closes the “GREEN while dead” gap). The morning of 2026-07-13 all three options bots crashed every cycle with an UnboundLocalError (a redundant inner import sys shadowed the module-level one, breaking the early sys.exit(0) guards) – yet every health check read GREEN, because login/state/cron-presence were all fine; nothing verified that trader.py runs to completion. The watchdog now also runs check_bot_crashes(): it scans each bot’s cron log (3 options + Portfolio 2) and alerts if the most recent cycle ended in a Python Traceback. The “live crash” test is that the last Traceback sits AFTER the last Logged out successfully, so a recovered bot does not false-alarm. Verified both ways: no alert on the recovered post-fix logs, correct detection on a synthetic crashed log.
/home/aztechguy/Projects/shared_trading_lib/watchdog_checkin.py/home/aztechguy/Projects/shared_trading_lib/run_watchdog.sh (own lockfile: /tmp/watchdog_checkin.lock)/home/aztechguy/Projects/shared_trading_lib/watchdog.logbackup-antigravity.sh, added 2026-07-13)~/.gemini/antigravity/ – which holds portfolio2_trader.py (the one bot that places real orders directly), its live portfolio2_state.json, the fetch scripts, and the .env with Robinhood credentials – was not covered by backup-projects.sh (that only backs up ~/Projects). Found 2026-07-13 while validating the staleness fix. backup-antigravity.sh closes it: nightly encrypted 7z (same password file + secondary drive as the Projects backup, 14-day retention) of just the trading code/state/creds/config (*.py *.json *.sh .env *.pbtxt), excluding the 58 MB of Gemini-CLI internals (.venv, brain, conversations, logs, scratch). Verified the archive contains portfolio2_trader.py + .env and is password-encrypted.
/home/aztechguy/.local/bin/backup-antigravity.sh/media/aztechguy/.../antigravity-backup//home/aztechguy/backup-antigravity.logfill_logger.py, added 2026-07-11)READ-ONLY: never places, cancels, or modifies orders. Fetches account 718316607’s executed fills (stock and option orders, state == "filled") from Robinhood’s order history via get_all_stock_orders / get_all_option_orders, and appends new ones to shared_trading_lib/fills_log.csv, deduplicated by order id. Records the real average fill price and executed quantity – the numbers the placement-time Discord alerts do NOT have (those show signal price and intended amount). Each row carries a best-effort source_bot attribution (SPMO/SMH/TQQQ -> portfolio2; QQQ options with same-day expiry -> qqq_0dte, else qqq_weekly; DIA -> dia_weekly; SFYF -> untracked-leftover). The CSV opens directly in any spreadsheet app.
Runs weekdays at 1:20 PM PT (after bots stop at 1:00) with --days 3 --discord: a lookback window that covers weekends/holidays plus a Discord summary embed of new fills. Manual use: --days N to backfill, e.g. the initial 2026-07-11 backfill captured 29 fills over 7 days and cross-verified against Portfolio 2’s state (TQQQ 0.6504 shares @ $76.865 actual vs $76.92 signal price – slippage is now visible per trade). Login goes through the shared login lock. Fills in other accounts (e.g. Roth IRA recurring buys) are intentionally out of scope.
Trade ledger (added same day): every run also rebuilds trade_ledger.csv – buys and sells paired into round trips in the Master Ledger format of the user’s ~/Documents/Me/Tradesv2.xlsx (Date = entry date, true average fill prices, Win/Loss/Open status), plus Source Bot / Exit Date / Notes columns. Open positions get Status “Open” with blank exit/PnL; a round trip whose position partly predates the log gets Status “Unknown-Basis” and a blank PnL rather than a cash-flow number masquerading as profit. Daily workflow: after the 1:20 PM run, copy any new rows from trade_ledger.csv into the Master Ledger tab (columns A-H align 1:1; the PnL and Status columns there are formulas, so paste only A-F and let the sheet compute). --ledger-only rebuilds the CSV from already-logged fills without contacting Robinhood.
/home/aztechguy/Projects/shared_trading_lib/fill_logger.py/home/aztechguy/Projects/shared_trading_lib/fills_log.csv/home/aztechguy/Projects/shared_trading_lib/trade_ledger.csv/home/aztechguy/Projects/shared_trading_lib/eod_balance.csv – each run also records account 718316607’s equity (load_portfolio_profile().equity, same field family as the drawdown guard) as date,equity, idempotent per day; feeds the Daily Ledger’s “Live Bot Balance EOD” column and appears in the Discord summary embed/home/aztechguy/Projects/shared_trading_lib/fill_logger.logAll four bots share one ~/.tokens/robinhood.pickle, and that token expires roughly 5 days after a real (device-approved) login. When it lapses mid-session the next bot’s login_with_lock -> rh.login() hits Robinhood’s device-approval and freezes holding the shared lock, so every other bot 90s-TimeoutErrors – a silent outage. Neither existing “connected” probe foresees it: the 5:45am briefing and the 30-min watchdog only prove the token works at the instant they run, not that it will survive the day. Because the token dies ~5 days after login (whatever clock time that lands on), a probe can truthfully report “connected” on the very morning it later expires.
The reliable early signal is the pickle’s mtime – robin_stocks only rewrites the file on a fresh login, not on cached-session reuse, so mtime = “last real auth” (do NOT read it as “last time login worked” – that exact conflation caused the 2026-07-11 outage misdating in Section 9).
token_staleness_check.py (proactive, added 2026-07-16) – stdlib-only, does NO login (it reads the file’s timestamp only, so it can never itself hang or touch the lock). Cron’d 6 AM + 2 PM daily; silent when healthy, pings the stocks Discord channel at day 4 (⚠️ warn) and day 4.75 (🚨 critical) with the re-login command baked in. The 5:45am morning briefing (Section 7) now also prints a live countdown line – 🔑 RH token: N days old -- good until ~<date> – so the wall is visible every morning, not just at the alarm.
Re-login runbook – the user runs this; an AI cannot (it needs the phone’s device-approval):
/home/aztechguy/.gemini/antigravity/.venv/bin/python3 /home/aztechguy/Projects/options_project/interactive_login.py
Run it proactively, while the countdown is still ticking: it writes a fresh pickle, resets the ~5-day clock, and every bot resumes off the new token on its next cycle – a clean reset, nothing else to do. Only if the token has already expired and jammed a bot do you also kill -9 the frozen fetch_data/trader process first (a fresh token does NOT un-stick an already-hung one – see Section 6.1’s diagnosis steps). The entire point of the day-4 warning is that you never reach that state.
/home/aztechguy/Projects/shared_trading_lib/token_staleness_check.py (--test fires a sample alert to verify the Discord path)/home/aztechguy/Projects/shared_trading_lib/token_staleness.log/home/aztechguy/.tokens/robinhood.pickle (its mtime is the staleness clock)~/.gemini/antigravity/.agents/AGENTS.md (the deprecated MACD-agent workflow’s instructions, which still name it for buy/sell orders – no active cron invokes that workflow) and a comment in fetch_portfolio2_data.py. The fetch_data.py holdout was quarantined 2026-07-11 (renamed fetch_data.py.retired_20260711), but an earlier revision of this KB overstated that as “absent from every code path / strictly true” – it isn’t, until AGENTS.md is also retired. Practical risk is low (all remaining references are dormant with no scheduler), but the absolute claim was wrong and is corrected here.cash (settled only) + unsettled_funds (pending settlement, typically T+1/T+2) = portfolio_cash (what the app displays as “Cash”). portfolio_cash + borrowable margin = the app’s displayed “Buying power.” The bots’ own buying_power field reflects non-margin buying power – none of the four bots currently borrow on margin.Local web dashboard at http://localhost:8080. Four tabs (Individual Account, Roth IRA, P&L Calendar, Volume Sim):
Individual Account tab: combined equity across QQQ + DIA + QQQ 0DTE options and the Portfolio 2 stocks (single shared Robinhood cash balance – not summed once per bot, which would multiply-count the same buying power). 0DTE inclusion added 2026-07-11: the dashboard originally omitted the 0DTE bot entirely (this document wrongly claimed otherwise); fixed same day – qqq_0dte payload key in dashboard_server.py plus an asset-dropdown entry in index.html, verified live against /api/data after a server restart. Pre-fix backups: dashboard_server.py.bak_20260711, index.html.bak_20260711. Per-asset dropdown with live chart, MACD/ATR indicators, active position, and performance metrics. Target Progress widget: goal window updated to a roughly 52-day period ending around September 1, 2026 (goal_config.json) – supersedes the original 30-day window starting 2026-07-06.
Roth IRA tab: manually-maintained holdings in roth_ira_holdings.json, priced live via yfinance. Not traded by any bot. The planned sell-off/rebalance (TQQQ/VIG/JEPQ/MAIN/EPR/SPCX out; SPMO/VGT/SMH/SCHD/VOO/CVX in) completed 2026-07-08, recorded in roth_ira_holdings.json and confirmed against a live account screenshot 2026-07-11; biweekly $100 recurring buys filled 2026-07-10. Expected display drift: the tab prices holdings with yfinance regular-session quotes and the manually-recorded cash, so it lags Robinhood’s displayed balance during after-hours moves (observed 2026-07-11 evening: dashboard $11,149.75 + $10.43 after-hours + $3.01 unrecorded cash ≈ Robinhood’s $11,164.63). Share counts still require a hand edit after any trade; prices refresh on every page poll.
P&L Calendar tab (added 2026-07-12): colored month grid of realized daily P&L (green win / red loss), by trade exit date, with ← → month navigation and a per-month realized total. Data from shared_trading_lib/trade_ledger.csv (closed win/loss round trips; Open and Unknown-Basis rows and the untracked SFYF holding excluded), exposed via the calendar_pnl key in /api/data. Verified against a hand-recompute: July 2026 = -$231.48 across 23 trades, June 29 = +$493 (1 trade).
Volume Sim tab (added 2026-07-12, expanded 2026-07-13): live forward paper-test comparing entry/exit variants against the live strategy. READ-ONLY, never places real orders. Engine: shared_trading_lib/paper_volume_sim.py (reuses the bots’ own scratch/historicals.json bars + state.json bias; no Robinhood login), cron 1-59/5 6-13 * * 1-5 (1 min after the bots), state in paper_sim_state.json, served via /api/papersim. Accounts (each $2300, all reset to a clean common start 2026-07-13 evening, seed fresh next open):
ungated – every 30-min MACD signal the daily bias allows (mirrors the live bot).gated – same, but only if time-of-day relative volume >= 1.25x (the volume-filter test, Finding 3).momentum – opening-momentum entry (intraday move from open >= 0.5%) instead of the MACD cross (Finding 2).scaleout – QQQ only (needs 2+ contracts; can’t bank half of DIA’s 1-lot): bank half at +30%, run the rest on the chandelier (Finding 5).CONTRACTS = {"QQQ":2,"DIA":1}), so paper dollars map to real position sizes. QQQ shows 4 accounts, DIA shows 3 (no scale-out).Performance Metrics card (added 2026-07-12, on the Individual Account tab). Replaced a dead placeholder card whose tiles were never wired to data. Now computed server-side from trade_ledger.csv and segmented via an All / Options / Portfolio 2 toggle – because mixing $0.78 Portfolio-2 ETF scalps with $493 options trades in one blended number was meaningless. Shows win rate, profit factor, expectancy/trade, avg win:avg loss, total trades (W/L), net P&L, account max drawdown, and best/worst trade, plus a small-sample warning banner (auto-hides above 30 trades). Verified against an awk hand-recompute to the cent (as of 2026-07-12: Options 12 trades, 50% WR, PF 1.51, +$21.42/trade expectancy, +$257 net; Portfolio 2 12 trades, 58.3% WR, PF 3.94, but only +$4.52 net – i.e. options carry the account, P2 wins often for pennies). Dropped two old per-asset tiles: “Est. Sharpe” (meaningless at this sample) and “Macro Bias Accuracy” (its sentiment_history source was never populated, so it always showed “–”; the real bias-agreement record lives manually in Tradesv2.xlsx column I “Agree on Bias?”).
Equity chart – QQQ Buy & Hold benchmark (added 2026-07-12). Orange dashed line on the Combined Overview equity chart answering “did I beat just holding the index?” Anchored to actual equity on the first date QQQ price data exists so both lines start together and diverge fairly; computed server-side via yfinance (qqq_benchmark key), cached 30 min so the 10-second poll doesn’t hammer yfinance. As of 2026-07-12 it shows the strategy lagged a plain QQQ hold over this window (strategy ~-18% vs QQQ ~flat, 07-06 to 07-12).
Pre-change backup for the 2026-07-12 dashboard work: dashboard_server.py.bak_20260712b. A .claude/launch.json was added under ~/Projects so the preview tooling can manage the server.
The dashboard runs as a systemd user service — it auto-starts at boot (user
lingering enabled), auto-restarts if it crashes, and logs to a file instead of a
terminal. Do not launch it by hand with python dashboard_server.py & — that
ties the server to your terminal (it dies when the window closes) and sprays HTTP
request logs over whatever you’re typing (both observed 2026-07-25 after a reboot,
which is what prompted this section).
systemctl --user restart dashboard_server # restart (the one you usually want)
systemctl --user status dashboard_server # is it running?
tail -f /home/aztechguy/Projects/options_project/dashboard/dashboard_server.log
Unit file: ~/.config/systemd/user/dashboard_server.service (WorkingDirectory
= the dashboard dir, ExecStart = trading_env python, Restart=on-failure).
After editing the unit file itself: systemctl --user daemon-reload first.
Boot behavior: enabled via default.target + loginctl enable-linger — it comes
up on reboot with no login and no manual step. If it’s somehow down, the restart
command above is the whole runbook.
History: before 2026-07-25 this was a bare long-running process (and briefly a transient systemd unit that a reboot erased); a post-reboot outage on 2026-07-25 converted it to the permanent service above.
Duplicate starter removed 2026-08-10 – the unit had never actually served a
request. A leftover @reboot nohup python3 ... dashboard_server.py & line survived
the 2026-07-25 conversion in the user crontab. It won the boot race every time, so
the systemd unit failed on Address already in use every 10 s from boot to shutdown
– 34,151 logged failures, 35 MB, into the same file the working copy wrote to.
The damage was not the noise: cron’s bare python3 is /usr/bin/python3.12, which
has no yfinance, so qqq_benchmark had been dead since 2026-07-25 while the
dashboard returned HTTP 200 and looked perfectly healthy. The unit’s ExecStart uses
trading_env python, which has it.
After any reboot, check which copy is really serving:
systemctl --user show dashboard_server -p ActiveState -p NRestarts -p MainPID
tr '\0' ' ' < /proc/$(systemctl --user show dashboard_server -p MainPID --value)/cmdline
Want ActiveState=active, NRestarts=0, and a cmdline beginning
/home/aztechguy/trading_env/bin/python3. readlink /proc/<pid>/exe is useless
here – the venv python is a symlink to /usr/bin/python3.12, so it reads
identically either way. The cmdline is the only honest evidence.
Bound to localhost 2026-08-10. Line 801 used
ThreadingTCPServer(("", PORT), ...), and "" binds 0.0.0.0 – every interface.
Since /api/data embeds each bot’s discord_webhook verbatim, any host on the LAN
could read a live alerting credential, unauthenticated. Now ("127.0.0.1", PORT);
backup at dashboard_server.py.bak_20260810. The access log made the decision free:
135,321 requests from 127.0.0.1 and not one legitimate remote request.
Daily pre-market check-in (daily_checkin.py) – RETIRED 2026-07-12, folded into the Morning Briefing’s Infrastructure section. Its cron line is commented out; the script is kept for reference. The morning briefing now does the live Robinhood connectivity + buying-power check (via shared_trading_lib/robinhood_status.py), the dashboard and cron checks, a token-age countdown (added 2026-07-16 – 🔑 RH token: N days old -- good until ~<date>, pairing the point-in-time “connected” line with days-to-the-5-day-wall; see Section 6.7), and covers all four bots + Portfolio 2 in its Trading section. Historical description of what it did:
(historical) Daily pre-market check-in (ran once at 6 AM PT): full Discord status embed covering Robinhood login, all four bots’ state health (QQQ, DIA, QQQ 0DTE, and Portfolio 2 – coverage extended to all four 2026-07-11), cron wiring, and the dashboard server. The cron check now verifies all four bot schedules plus scanner and watchdog, and only counts active (uncommented) crontab lines. A --force flag runs the check-in and sends the report even on closed-market days (manual health checks, post-change verification). Verified 2026-07-11 with a real --force run: login via the shared lock, all checks executed, embed delivered. Pre-change backup: daily_checkin.py.bak_20260711. Remaining structural limit: as a once-daily pre-market check it cannot catch a same-day mid-session failure – that is Section 6.4’s watchdog’s job.
/home/aztechguy/Projects/options_project/dashboard/dashboard_server.py/home/aztechguy/Projects/options_project/dashboard/index.html/home/aztechguy/Projects/options_project/dashboard/roth_ira_holdings.json/home/aztechguy/Projects/options_project/dashboard/individual_holdings.json – SGOV/SMH core deployment + SFYF leftover + margin_loan (subtracted from combined equity: borrowed dollars are not equity). Hand-edit after buys/sells, same as the Roth file. Added because combined equity ignored the 2026-07-24 deployment and falsely showed “BEHIND PACE -56%”.shared_trading_lib/insider_monitor.py (cron weekday 18:05) scrapes openinsider’s latest cluster buys into insider_signals.csv, scores every signal at +3d/+1w/+1mo from the FILING date (public-actionable, not hindsight) with SPY-adjusted alpha per window, writes insider_state.json -> dashboard 🕵️ Insiders tab (/api/insiders). Info feed with an honest back-checker – nothing is traded from it. First-run seed (100 signals, 82 scored @1mo): 72% win, +6.7% avg alpha – treat with suspicion (one regime window, thin micro-caps); the forward scoring is the real test. FISHBOWL HOLD until 2026-10-01 – banner on the tab, frozen baseline stored in insider_state.json under baseline, one-time FRIDAY review scheduled (scheduled-task insider-fishbowl-review) to compare live out-of-sample stats vs baseline and decide promote / extend / bin./home/aztechguy/Projects/options_project/dashboard/goal_config.json/home/aztechguy/Projects/options_project/dashboard/equity_history.json/home/aztechguy/Projects/shared_trading_lib/trade_ledger.csv (maintained by the fill logger, Section 6.5)/home/aztechguy/Projects/options_project/daily_checkin.pyA standing rule, now in ~/.agents/AGENTS.md, requiring any factual claim about current price, system state, file contents, or library/API behavior to be verified directly (read the file, run inspect.getsource(), check crontab -l/ps, re-run the code path) rather than asserted from memory or training data. Adopted after a run of incidents that all shared the same root cause – confidence without verification. Running list, oldest first:
trade_history timestamps in state.json were checked directly, which pinned the real outage to a single Friday afternoon.state.json files and reported clean without checking Portfolio 2, which had a real, live, unmonitored TQQQ position the entire time. Caught by the user, not self-caught.profile.get("cash") (settled funds only) as “the” balance without checking whether the API exposes other cash-related fields for different purposes (portfolio_cash, which is what the app actually displays, also includes unsettled_funds). Found and corrected after a real account screenshot didn’t match.A written rule is necessary but not sufficient – treat it as something to keep spot-checking against actual behavior, not a guarantee on its own.
Theta-blind exits (2026-07-03): synthetic delta approximation instead of live quote for exit pricing. Fixed; now persists last_live_value/last_live_quote_success into state.json.
No cron overlap guard (2026-07-03): added flock to all run_cycle.sh scripts.
Stray GLD trading (2026-07-03): traced to a separate direct MCP trading connection, not any script in this repo. Confirmed closed out with no residual position.
Duplicate stale folders (2026-07-03): orphaned, cron-unreferenced copies removed.
QQQ vs SPY backtest bug (2026-07-03): a comparison script hardcoded a flat 8-day option expiration, inflating QQQ’s modeled premium and producing a false “QQQ is better” conclusion. Fixed.
Market holiday blindness (2026-07-03): none of the cron scripts checked a real holiday calendar. Fixed across all scripts via pandas_market_calendars.
Portfolio 2 stale-signal bug (2026-07-07): see Section 5.
Portfolio 2 sell-order time_in_force bug (2026-07-07, fixed twice): see Section 5.
No retry on Gemini API failures (2026-07-08): premarket_scanner.py’s sentiment call gave up after one failed attempt, defaulting all assets to NEUTRAL on any error, including a confirmed-transient Gemini 503. Fail-safe-to-neutral behavior itself was correct and unchanged; added a 3-attempt retry with a 5-second delay before falling back.
Suggested commands could contradict the allocation suggestion (2026-07-08): the “approve X” command text and the “Suggested Allocation” text were built from two separate, unreconciled Gemini output fields, so a “0” allocation could still suggest a directional approve X puts/calls command at full size, contradicting the “sit it out” recommendation. Fixed by parsing the allocation text and folding the quantity directly into the suggested command – a “0” allocation now suggests approve X neutral.
SPY retired, sibling gate removed (2026-07-08): see intro and Section 4.
Wrong account number in fetch_portfolio2_data.py (2026-07-11): see Section 5.
Dashboard combined equity excluded the QQQ 0DTE bot (found & fixed 2026-07-11): dashboard_server.py never referenced qqq_0dte_project, so the combined total silently understated exposure whenever 0DTE held a position. Fixed by adding it to get_individual_account_payload() and the frontend asset dropdown; verified live via /api/data. See Section 7.
Manual fill-price spreadsheet step automated (2026-07-11): actual executed fills were only visible by logging into Robinhood and copying prices by hand; the placement-time Discord alerts show signal price, not fill. Added the read-only fill logger (Section 6.5) + cron; initial backfill verified against Portfolio 2 state.
daily_checkin.py covered only 2 of 4 bots (fixed 2026-07-11): added QQQ 0DTE and Portfolio 2 state checks, extended the cron check to all four bots + scanner + watchdog (active lines only), added a --force flag for closed-market manual runs. Verified with a real send. See Section 7.
Stale +50% docstring in qqq_0dte_project/trader.py (fixed 2026-07-11): now reads +30%/-30%, matching the code.
Retired-account script quarantined (2026-07-11): the dormant ~/.gemini/antigravity/fetch_data.py (hardcoded 453789794, referenced only by the commented-out run_macd_cycle.sh cron line) renamed to fetch_data.py.retired_20260711. Correction (fable-judge, 2026-07-12): this was reported at the time as making “453789794 appears in no live code path” strictly true. It did not – the account number still lives in the dormant .agents/AGENTS.md (deprecated MACD-agent instructions) and a comment in fetch_portfolio2_data.py. The rename is real and correct; the “strictly true / every code path” escalation was an overclaim (verified filename references, not the account number itself). See the corrected Section 6 account reference. Remaining references are dormant (no active scheduler); quarantining .agents/AGENTS.md + the macd_trader.py workflow would make the absolute claim actually hold.
force_buy.py hardcoded a stale expiration (found & fixed 2026-07-11): expiry_str = "2026-07-06" – already 5 days stale, would have requested an expired contract if used. Now computes next-Friday (or nearest valid non-0DTE expiration) with the same logic as trader.py, so it never needs manual date updates again. Date logic verified against mocked chains (Saturday -> next Friday; entry on a Friday -> following Friday; holiday Friday -> nearest valid). Pre-fix backup: force_buy.py.bak_20260711.
Shared-token collision / Robinhood 429 lockout (outage 2026-07-10, root cause understood + fixed 2026-07-11): the shared ~/.tokens/robinhood.pickle session expired mid-Friday-afternoon (ordinary session expiry, not a multi-day outage – see Section 8’s GTE note). Because all four bots independently call rh.login() on their own 5-minute cron cycle with zero coordination, all four simultaneously fell through to a fresh MFA challenge and began polling Robinhood’s get_prompts_status endpoint every cycle, which Robinhood rate-limited with 429 Too Many Requests, permanently failing that challenge. Because each bot’s run_cycle.sh uses set -euo pipefail, the failed login aborted the entire cycle before trader.py ever ran – no bot evaluated any entry or exit signal for the rest of that session. No position was left unmonitored as a result except Portfolio 2’s open TQQQ position, which sat unwatched from the last successful check (11:10 AM PT Friday) through market close and the following weekend. Fixed by the shared login lock (Section 6.1); manually re-authenticated once to restore a valid shared token, and the lock now prevents this specific collision mechanism from recurring.
All 3 options bots crashed every cycle – UnboundLocalError on sys (introduced 2026-07-11 with the login-lock work, found & fixed 2026-07-13 at market open): the login-lock edit added import sys inside main() in all three options trader.py files, which makes Python treat sys as a function-local for the entire scope – so the earlier sys.exit(0) market-hours guards raised UnboundLocalError and aborted every cycle before any trading logic ran. None of the three could trade Monday morning until the redundant inner import sys was removed (module-level sys already exists). Verified: all three now exit 0 cleanly at the guard, compile, and run clean in the live cron. Lesson (Section 8): py_compile does not catch UnboundLocalError (runtime, not syntax); the login-lock work was compile-checked but its early-exit path was never runtime-exercised. Also exposed a monitoring gap – the watchdog/briefing reported GREEN while all three bots were dead, because they check login/state/cron-presence, not whether trader.py runs to completion.
Portfolio 2 morning trailing-stop dead-window (found live & fixed 2026-07-13): the 5-minute chandelier trailing stop was gated on is_stale_bar, which is derived from the last closed 30-minute bar. For the first ~30 min after the open (before today’s first 30m bar closes), that flag is the prior session’s date, so a valid same-day 5m stop-out was suppressed as “stale.” Caught live when a real TQQQ stop (price below stop level) held instead of selling, then fired ~20 min late once the first 30m bar closed. Fixed by adding a separate is_stale_5m (derived from the 5m bar’s own date) and gating only the trailing-stop branch on it; the 30m MACD crossover paths correctly still use is_stale_bar. Verified: compiles, date logic proven under the exact failure scenario (5m current -> stop fires; 30m stale -> MACD still gated), clean in live cron. Note: this was a mid-session edit to live trade logic made without a pre-edit .bak (convention miss); the change is the two is_stale_5m edits and is trivially reversible.
paper_30s_scalper.py silently polled the WRONG DATA SOURCE for its entire life (found & fixed 2026-08-01): the quote fetcher called rq.get_quote(symbol). That function does not exist – rh_quotes.py exposes stock_quote. Every single poll raised AttributeError, was swallowed by a bare except Exception whose only check was if "429" in str(e) (it never matched), left price = None, and fell through to the yfinance branch. Result: 100% of ~900 recorded paper trades used Yahoo prices while the dashboard’s “API health / latency” panel implied it was measuring Robinhood, and rate_limit_429_count: 0 was reported as evidence of health when it was really evidence that the code path never ran. Fixed: correct function name, a _warn_once() that makes a failing source log loudly exactly once, and fetch_live_quote() now returns the source actually used so state/dashboard read HEALTHY (yfinance) rather than a bare HEALTHY.
Deliberately NOT “repointed at Robinhood”: at 1 req/sec that is ~23,400 RH calls/session, and rh_quotes acquires the shared login lock the live bots use – a 1-second loop contending for that lock is a credible repeat of the 2026-07-10 lockout. RH is now opt-in behind USE_RH = False for short supervised A/B runs only. The bug was, accidentally, protecting the live bots.
Same file: is_simulated was accepted and never read (found & fixed 2026-08-01): process_scenario(..., is_simulated=False) took the flag, run_test_mode() dutifully passed is_simulated=True, and the function body never referenced it – so --test random-walk trades were appended to the same paper_30s_scalper_trades.csv as live ones, indistinguishable after the fact. Fixed by gating both append_trade() calls; verified the CSV row count is unchanged across a full --test run.
Same file: no market-hours gate (found & fixed 2026-08-01): the daemon traded the instant it launched, at any hour, on any day. The stated workflow (“launch ~5 minutes before the open”) would have opened the first positions on thin pre-market SOXL. Added MARKET_ONLY / market_is_open(); launching early now waits and starts clean at 09:30 ET. Verified: announces once, does not poll, leaves state untouched outside RTH.
Added at the same time – quote FRESHNESS instrumentation. A 1s/3s scenario comparison is only meaningful if the feed updates at that resolution; yfinance.fast_info is a cached endpoint, not a tick feed. New quote_health block counts polls, distinct prices, stale streaks and distinct-prices-per-minute, and logs a verdict: TICKING (>=30/min), COARSE – sub-second sampling is fiction (5-29), STALE – results are meaningless (<5). Read this BEFORE reading the P&L.
Lesson (Section 8) – the silent-fallback anti-pattern. Both scalper bugs are one mistake: a bare except wrapped around a fallback converts a permanently broken primary into a permanent silent swap. Nothing errors, nothing alerts, and the dashboard keeps reporting the healthy-looking metric it is no longer measuring. This is the same family as the 2026-07-13 UnboundLocalError (watchdog GREEN while three bots were dead) and the inverse of the 2026-08-01 briefing false alarm (a ❌ that fired daily for an intended state, training the eye to ignore ❌). Rule: a fallback must name the source it actually used, and log the first failure loudly. A monitor that cannot fail visibly is not a monitor.
Dashboard served by the wrong Python for 16 days (found & fixed 2026-08-10): see
Section 7. A duplicate @reboot cron starter beat the systemd unit to port 8080 on
every boot; because it used system python3 rather than trading_env, yfinance was
missing and qqq_benchmark silently returned nothing. HTTP 200 throughout.
Alerting webhook readable from the whole LAN (found & fixed 2026-08-10): the
dashboard bound 0.0.0.0 and /api/data returned each bot’s discord_webhook in
clear text. Anyone on 10.0.0.0/24 could have posted convincing fake alerts into
the channel the whole monitoring setup depends on – worse than a silent channel.
Fixed by binding 127.0.0.1 and rotating the webhook.
The webhook was hardcoded in nine scripts, not read from a file (2026-08-10):
rotating ~/.config/discord-webhooks/stocks.txt was not enough. trader.py:112 and
eight siblings embed the URL as a default written into state.json, so fixing the
state files alone regresses the moment a bot rebuilds state. Seven of the nine run on
cron. Grep the whole tree for the old webhook id after any rotation – 57 files
still contained it after the “complete” rotation:
grep -rl '<old-webhook-id>' ~/Projects ~/.gemini ~/.config | grep -v /public/
The durable fix is for every script to read ~/.config/discord-webhooks/stocks.txt
(mode 600) instead of embedding the URL. Not yet done – see Section 10.
Lesson – a dead credential is safer than an edited one. Deleting the old webhook in
Discord turned 42 stale Jarvis transcripts and 5 .bak files from credential leaks into
harmless clutter, and made the incomplete rotation visible (HTTP 404) instead of
silent. Rotate at the source; do not chase copies.
# Portfolio 2 Multi-Timeframe Check (30m MACD + 5m ATR Trailing Stop)
*/5 6-13 * * 1-5 /home/aztechguy/.gemini/antigravity/run_portfolio2_cycle.sh >> /home/aztechguy/.gemini/antigravity/portfolio2_cron.log 2>&1
# SPY Options Bot -- RETIRED 2026-07-08, capital reallocated to QQQ+DIA hedge pair
30 5 * * 1-5 /home/aztechguy/trading_env/bin/python3 /home/aztechguy/Projects/options_project/premarket_scanner.py >> /home/aztechguy/Projects/options_project/scanner_cron.log 2>&1
# DIA Options Bot (30m MACD Entry + 5m ATR & Target Profit exits, 1 contract, trades independently -- no sibling gate as of 2026-07-08)
*/5 6-13 * * 1-5 /home/aztechguy/Projects/dia_options_project/run_cycle.sh >> /home/aztechguy/Projects/dia_options_project/options_cron.log 2>&1
# QQQ Options Bot (30m MACD Entry + 5m ATR & Target Profit exits, 2 contracts, trades independently -- no sibling gate as of 2026-07-08)
*/5 6-13 * * 1-5 /home/aztechguy/Projects/qqq_options_project/run_cycle.sh >> /home/aztechguy/Projects/qqq_options_project/options_cron.log 2>&1
# QQQ 0DTE Options Bot (EXPERIMENTAL, added 2026-07-09) -- same guardrails as the weekly QQQ bot but same-day expiration, 1 contract, "both" bias (no manual approval), ~$200 budget, independent of QQQ/DIA/Portfolio2
*/5 6-13 * * 1-5 /home/aztechguy/Projects/qqq_0dte_project/run_cycle.sh >> /home/aztechguy/Projects/qqq_0dte_project/options_cron.log 2>&1
# Daily pre-market status check-in
0 6 * * 1-5 <antigravity-cli venv>/bin/python3 /home/aztechguy/Projects/options_project/daily_checkin.py >> /home/aztechguy/Projects/options_project/checkin.log 2>&1
# 30-min silent-unless-broken watchdog for all 4 bots (added 2026-07-11) -- catches mid-session auth failures the 6am daily_checkin.py can't
*/30 6-13 * * 1-5 /home/aztechguy/Projects/shared_trading_lib/run_watchdog.sh >> /home/aztechguy/Projects/shared_trading_lib/watchdog.log 2>&1
# Fill logger (added 2026-07-11) -- READ-ONLY: records actual Robinhood fills to shared_trading_lib/fills_log.csv after close, posts Discord summary. Never places orders.
20 13 * * 1-5 <antigravity-cli venv>/bin/python3 /home/aztechguy/Projects/shared_trading_lib/fill_logger.py --days 3 --discord >> /home/aztechguy/Projects/shared_trading_lib/fill_logger.log 2>&1
# RH token staleness check (added 2026-07-16) -- read-only, NO login; warns at day 4 / day ~5 before the ~5-day token wall jams the bots. Runs EVERY day (not just weekdays -- the token can age out over a weekend).
0 6 * * * /usr/bin/python3 /home/aztechguy/Projects/shared_trading_lib/token_staleness_check.py >> /home/aztechguy/Projects/shared_trading_lib/token_staleness.log 2>&1
0 14 * * * /usr/bin/python3 /home/aztechguy/Projects/shared_trading_lib/token_staleness_check.py >> /home/aztechguy/Projects/shared_trading_lib/token_staleness.log 2>&1
All bot entries skip execution on real NYSE market holidays, not just weekends.