I had grown my paper trading bots to seven. One day I stood them all up and audited them with a single sentence: "Did this profit come from skill, or just from a rising market?" Five died on the spot.
One question first. There's a green +return on your bot dashboard. Does that number mean you beat the market, or that the market went up? Is there a metric on that dashboard that tells the two apart? If not, that green is probably lying to you.
What I built
Seven bots, each a different hypothesis. All paper — I never even wrote the live-order code.
- LLM-as-judge bot (5-min and 15-min cadences): direction call delegated to an LLM.
- Turn-of-month multi-ETF: a calendar anomaly (buy the 4th-to-last trading day, sell the 3rd of next month), 10 ETFs at once.
- Upbit crypto swing: an LLM screener picks 3 coins daily.
- Insider-buying event: only open-market purchases from filings (Form 4), traded as an event study.
- Low-PBR value: mid-cap low-PBR top 20, equal weight, monthly rotation.
- Leverage infinite-buy: a dollar-cost-average-down strategy on a 3x ETF.
One discipline was baked in from the start: pre-registration — writing the kill/keep gate before the data arrives. What n, what win rate, what excess return vs benchmark — the pass conditions hardcoded up front, never touched again. Without it, this post wouldn't exist.
The audit — verdicts
I didn't trust the dashboard. I tore each one open with the ledger, live prices, and git history.
Turn-of-month multi-ETF — gate reached, FAIL. It filled all 30 closed trades. Absolute return +2.06%. Looks like a pass. But the excess return vs same-window SPY was −0.17%p. That 77% win rate wasn't "the calendar worked" — it was "the market went up for three straight months." The gate honestly stamped FAIL.
Upbit crypto swing — negative edge. 18 closed trades, 16.7% win rate. Versus just holding BTC: −3.07%p per trade. BTC was roughly flat this window, so this isn't even "alpha buried under a bull run." The LLM picks were worse than doing nothing.
Bithumb SMA50 — no data at all. After deployment the signal stayed CASH the whole time, so zero trades. Worse, the SMA50 parameter itself was chosen post-hoc by running 9 indicators over the same data — the doc literally confessed the bias. No sample to validate, and no standing to validate in the first place.
LLM-as-judge bot — KILL. The sample was plenty (5-min n=324, 15-min n=106). Both losing (−$56, −$44), win rate below breakeven. No edge, statistically confirmed.
What would you look at here?
By absolute return alone, turn-of-month is "+2%, 77% win rate." A number you'd want to brag about. The infinite-buy bot also had positive realized PnL. Would you push either to a live account on that?
Every trap came from the same place: bull-market beta. Because the last few months were favorable, nearly every bot's absolute return was inflated. Turn-of-month is the proof — absolute +2%, excess negative. Profit you get just from leaving market exposure on, mistaken for the strategy's doing. It's exactly the trap from when all 9 timing strategies lost to SPY and only static leverage won. There too, what won was leverage, not skill.
The 2 that survived (plus 1 reprieve)
Low-PBR value. Only one rebalance so far, essentially no data. But the design was the sturdiest — the benchmark isn't a cost-free index, it's an eligibility-matched equal-weight portfolio carrying the same frictions, so the benchmark absorbs Korean-market long beta and you only see the pure alpha of the low-PBR tilt. It structurally excludes "I made money because it was a bull market." And the value premium is academically documented. It just has no data to conclude on — the direction isn't wrong.
Insider-buying event. At n=3 the statistics are meaningless, but the direction was positive, and insider-buying drift is also a documented anomaly. Higher prior than an arbitrary LLM pick or a post-hoc parameter.
The LLM-as-judge bot is a KILL, but I didn't kill it. A counterfactual experiment (measuring the return if you'd exited just before expiry) isn't finished, so it keeps accumulating sample as paper that burns no capital.
The common thread among survivors: they bet on already-documented anomalies, not arbitrary signals. The other five were calendar, LLM discretion, post-hoc selection — low-prior things.
Self-check — 3 questions to ask your bot right now
- Is there an excess-return gate? Does "vs benchmark" excess — not absolute return — enter the pass criterion? If not, the bull market is writing your report card for you.
- Have you changed the gate after seeing data? Check the threshold's change history in git log. Moved it even once, and that's post-hoc selection.
- Did skill win, or exposure? See if the profit survives after stripping leverage and beta. Usually it doesn't.
The honest part
What died in this audit was the strategies, not the honesty. The bots were marking losses as losses and paper as paper — pre-registered gates, live-order isolation, no unrealized-PnL optimism. The self-deception guards worked. The problem was edge, not accounting.
And that's the point. Most trading self-deception comes not from a bad strategy but from a good-looking number. Green +returns, 77% win rate. The moment you stand one line of excess return next to it, it all collapses.
Do just one thing right now: write your bot's absolute return next to "just holding the benchmark over the same period." If the two are close, or the benchmark wins — your alpha was beta.
In the next post I write the hands-on process of retiring these dead bots without regret: from stopping launchd, to cleaning up dashboards, to archived deletion.