I Learned It From a Dead Experiment and Never Applied It to Four Live Bots

All paper trading. Sample counts, standard deviations and returns are real ledger values.

I shut down an experiment and wrote the postmortem. The last section was titled "gate design defects — must carry into the next experiment," and item six read:

No power calculation before launch — I nearly burned six months at a sample rate that could never produce an answer.

I committed the document and felt good about it. A few days later, while checking on the paper bots that were still running, I noticed something. One dashboard read: "collecting sample 6/30 — 24 more trades until verdict."

I could not remember where that 30 came from.

If your project has a "decide once we have N" rule, how was that N chosen?

Suspect the cheap explanation first

My first thought was "the sample is small, so we don't know yet." Time fixes that. Just wait.

I ran the numbers to confirm. Across six closed trades, mean excess return was +1.09pp with a standard deviation of 9.59pp. Standard error 3.91, t = 0.28.

So far "small sample" holds. The problem was the next line. Filling n to 30 only shrinks the standard error to 1.75. For significance you would then need a mean excess of 3.50pp — three times the observed point estimate.

The gate cannot produce an answer even at n = 30. Waiting was not going to fix it.

I calculated the sample needed to detect the observed effect at 80% power:

n = (1.96 + 0.842)^2 x sd^2 / effect^2
  = (2.802 x 9.59 / 1.09)^2
  = 607

Recomputed against an economically meaningful floor of 1.0pp: 722 trades. At a closing pace of 0.76 per day, that is roughly 2.6 years.

The 30 had no basis. It was "that should be about enough."

And there was another one

Had I stopped there I would have fixed one bot and moved on. Instead I opened the other three.

KRX low-PBR value bot. The top KPI on its dashboard read:

Sharpe 27.84 (benchmark 12.93)

That came from one month of forward observation. An annualized Sharpe from a single observation is noise, but it sat large in the gate-metric slot, so anyone reading it reads "we're winning." The gate rule was cumulative excess > 0 AND Sharpe > benchmark Sharpe. A sign comparison.

Dual momentum rotation bot. Its pre-registered criterion was "paper-period max drawdown < index buy-and-hold max drawdown." Reasonable for a defensive strategy. But that claim is only tested when the index actually falls. Max index drawdown over the observation window was 1.5%. Nothing happened. Meanwhile the screen showed +3.7% cumulative, which reads as "working."

Two leveraged-ETF averaging-down bots. Their criteria had no comparison at all. That one got long enough to need its own post.

Four out of four. All the same shape.

The gate always returns a verdict. It never asks whether that verdict beats chance.

Why only the dead ones

This is the actual subject of the post. It did not happen because I don't know what statistical power is. I had already killed one experiment before launch on exactly these grounds, added a minimum-sample floor to a self-learning loop, and even rejected my own edge with a multiple-comparison correction.

Every lesson landed on an autopsy subject. I learned it while cleaning up a failure, and I wrote it into the document about that failure.

Live projects were not audit targets. The lights were green. But the source of those green lights was the defective gate itself. The signal saying "this is working" was produced by the very rule that had never been validated.

The same sweep turned up more. A daily report job for a bot wound down three weeks earlier was still sending Telegram messages every morning. A live-order bot that stopped two weeks ago was still registered in the dashboard with a start button attached — one misclick from restarting a strategy with no edge.

I have published a checklist for killing bots cleanly. "Remove the dashboard registration" is on it. I did not follow my own checklist.

What would you do here

There is a fork. The math says 2.6 years to a verdict.

Kill the bot — 2.6 years is too long
Raise the threshold — change 30 to 722 and keep going
Change the shape of the gate

Option 1 is wrong. A paper bot costs essentially nothing to run. Slow is not a reason to stop.

Option 2 is not enough either. Even with 722 trades, if the question is still "is the mean positive," a single tail observation can still manufacture the verdict. In fact, for this bot, dropping the single best of six trades flips the mean from +1.09pp to -2.07pp.

I took option 3.

Make "cannot decide" a first-class state

Here is the structure I put into all four bots.

One: fix the minimum detectable effect (MDE) before looking at the data. It is the value below which the strategy is not worth running, set above friction costs. 1.0pp per trade for the insider-buying bot, 5pp annual excess for the value bot. Not extracted from the data — nailed down first.

Two: compute the required sample and show it permanently. If the answer is 2.6 years, that number is the conclusion. Hiding it is the failure mode.

Three: PASS requires three axes simultaneously.

Bootstrap 95% CI lower bound > 0 — magnitude
10% trimmed mean > 0 — tail immunity
Sign test p < 0.05 — directional consistency

I use a bootstrap rather than a t-interval because event-driven return distributions have fat tails. The trimmed mean and the sign test each block "a mean built by one trade" in a different way.

Four: a futility clause. If the upper CI bound falls below the MDE, stop before filling the sample. There is no reason to keep collecting once "even a win leaves nothing" is settled.

Five, and this is the core — make "cannot decide" its own state. When the sample falls short of what is required, declare neither PASS nor FAIL. You don't have the power to say "it lost" either. In that state, win rate, returns and Sharpe are demoted to grey "reference only" figures with a warning banner.

gate v2: collecting — 24 trades until verdict opens
  excess CI95 [-5.28, +8.68]pp · trimmed mean +1.09pp · sign test p=1.000
  sample needed to detect 1.0pp: 722 (currently 6, sd 9.59pp)
  undecided — win rate and mean above are reference only, not grounds for adoption

Defensive strategies get one more clause: a trial condition. A bot claiming drawdown protection is only tested when a decline actually occurs. If the observation window closes without the index falling past a threshold, the result is neither success nor failure but an explicit no-verdict close. Without that clause, a defensive strategy that only ever saw a bull market gets recorded as "12 months, positive returns, success."

Three-line self-check

For whatever automation, experiment or bot you are running right now:

Where did the N in "decide once we have N" come from? If it is intuition rather than a power calculation, your gate will produce a verdict, but that verdict has no basis.
Can your dashboard display "cannot decide"? A screen with only PASS and FAIL will pick one of them even at n = 1.
Does dropping your single largest observation flip the conclusion? If it does, what you are looking at is not a strategy — it is that one observation.

The honest part

Fixing the gates improved exactly zero strategies. If anything it went the other way. Before the fix, all four bots read "progressing normally." After, all four read "cannot decide, sample below requirement." One of them now says the verdict is 2.6 years out.

That is the improvement. The previous screen was not information — it was reassurance.

Let me also record this post's own defect. I set the MDE after seeing six observations. Strict pre-registration requires setting it before. So I derived the MDE from friction costs (1.0pp) rather than from the observed effect size (1.09pp). That is a mitigation, not a clean pre-registration.

One last summary of where the four bots stand: all of them are waiting for a downturn. A strategy that claims defense is only tested when there is something to defend against, and I don't get to choose when that arrives.

If you have even one experiment running, calculate a single line: (2.802 * sd / minimum_effect)^2. If that number is larger than your plan, the green light you're looking at doesn't mean anything yet.