Does your experiment have a clause that declares "winning wouldn't matter"? Not a clause that detects losing — that's a different thing.
I turned off a paper-trading bot today. It read insider buys from SEC Form 4 filings and tried to capture the post-filing drift over a five-day hold. It ran at noon Tuesday through Saturday, benchmarked against SPY. I stopped it at 32 closed trades.
The numbers first
| Metric | Value |
|---|---|
| Closed trades | 32 (9 still open, observed 2026-07-22 to 08-27) |
| Win rate | 34.4% (11/32) |
| Mean excess vs SPY | −1.58pp |
| Cumulative PnL | −$449.25 |
| Sign test p | 0.110 |
| Bootstrap 95% CI | [−3.64, +0.61]pp |
The easy misreading is "34% win rate and −$449, so it lost, so he killed it." Wrong. The sign test comes out at p=0.110. I have no basis to say it lost either. Thirty-two samples prove nothing in this distribution.
So why kill it
Three weeks ago I rewrote the gate and nailed down a stopping clause:
FUTILITY (stop) — upper bound of the excess-return interval < minimum detectable effect (MDE). This clause fires even before the minimum sample is reached.
The MDE wasn't pulled from the data. It was fixed in advance. At $1,000 notional per trade, a five-day hold, 5bp each way, round-trip friction is 0.1pp; add generous slippage and you're still inside 0.3pp. Six concurrent positions gives roughly 313 trades a year. On that basis I fixed MDE = +1.0pp per trade: below this, there's nothing left after friction, so there's no reason to run it.
Today the upper bound was +0.61pp. That means even the best case this strategy could have doesn't reach the 1.0pp line. It isn't about winning or losing; it's that winning gets eaten by friction, and the data has now confirmed it.
What would you do?
Nine trades were still open. And with this sample's standard deviation of 6.24pp, a powered verdict needs 306 closed trades — over a year at the current pace. Paper trading costs nothing to keep running. Would you leave it on, or turn it off now?
I turned it off. Another year buys me a more precise measurement of a sub-friction effect. A more precise zero is still zero.
Under the v1 gate it would still be running
The bot's first gate read: closed n≥30 AND mean return > 0 AND excess vs SPY > 0.
That's a sign comparison. Feed today's data to it and you get: "n=32 is met, but the mean is negative, so not a pass yet — continue." With no stopping clause, continue forever. The reason more samples never produce an answer is that the rule has no concept of effect size, and the rule can't tell you that about itself.
I'd made the same mistake once already. In the retrospective for a different bot I listed "no power calculation before starting" as a gate defect (the experiment had no statistical power), and moving that lesson into this bot's v2 is exactly what made today's shutdown possible. A gate that can't reach a verdict has the same disease as a gate with no denominator.
Three things to check
Look at the experiment, A/B test, or paper strategy you're running right now:
- Is the minimum effect written as a number before you start? If not, you'll invent "that seems meaningful" after seeing the result.
- Is there a stopping clause? A gate with only pass conditions has no way to declare failure, so it extends indefinitely.
- Do you recompute the required sample on every run and display it? When the standard deviation grows, the required sample grows with it — and that growth is itself information about the signal-to-noise ratio.
Shutting down is also a procedure
I booted out two launchd jobs (daily and dashboard) and deleted the plists. I checked with lsof that the port was actually dead. I archived the repo — including the SQLite ledger — as a 42MB tarball and deleted the original, so the data is still there if I ever revisit event drift.
Then I removed the dead links from the three dashboards that still pointed at it. Skip that step and you'll be clicking your own 404 a few weeks later. I wrote up the shutdown procedure itself a while back.
The honest part
The bot ran for five weeks and the result is "I don't know whether this strategy has an edge." The one thing I do know: finding out takes 306 trades, and the most I can hope for over that stretch is smaller than my friction. That's knowledge about the experiment, not about the strategy. Still, knowing it in five weeks beats knowing it in 2.6 years.
Pick one experiment you're running and write its stopping clause today — the number below which you quit even if you win — before you look at the data.