I was paper-trading a low-P/B value strategy on Korean mid-caps. The backtest looked decent — out-of-sample Sharpe 0.77, +51.3% excess, robust from 10 to 30 holdings. So I wrote a pre-registration doc, locked the rules, and started rebalancing monthly.
A month later I opened the ledger.
n=1 cum_excess=-3.43%p sharpe=4.82(bench 4.81) mdd=5.6%
state=running decidable=false required_n=nullrequired_n was null.
Is the "required sample" field on your experiment filled in right now? I built that field and never once filled it.
null didn't mean "not known yet"
The reason was mundane. Required sample is computed from the standard deviation of forward monthly excess, and with one observation there is no standard deviation. So: None.
Technically correct. But that is "I didn't think to get it elsewhere," not "the data doesn't exist." The same strategy had eight years of backtest sitting right there.
I pulled the monthly excess series for exactly the pre-registered strategy (mid-cap, 20 lowest-P/B names, benchmarked against all liquid names equal-weighted):
| Window | n (months) | Mean excess/mo | sd/mo | Annualized | Required n |
|---|---|---|---|---|---|
| Full | 104 | 0.76% | 3.94% | +9.1% | 703 months ≈ 59 years |
| IS 2018–2022 | 60 | 0.63% | 3.19% | +7.5% | 461 months ≈ 38 years |
| OOS 2023–2026H1 | 42 | 0.80% | 4.37% | +9.6% | 863 months ≈ 72 years |
The pre-registered minimum detectable effect is +5%p per year. Per month that's 0.417%p. The monthly noise is about 4%p — ten times the signal.
Even being generous and asking only to detect the edge the backtest actually showed (+9.6%/yr), it's 235 months, about 20 years.
The correct fix is what killed the experiment
This is the uncomfortable part.
Three weeks ago I strengthened this gate. The original PASS condition was a sign comparison: cumulative excess > 0 and Sharpe > benchmark Sharpe. That rule always returns a verdict, at any sample size. It never asks whether the verdict is distinguishable from chance.
There was a trigger. The dashboard was displaying an annualized Sharpe of 27.84 computed from one observation as a KPI. Perfect material for reading "we're winning." So I added a condition to PASS: bootstrap 95% CI lower bound of mean monthly excess > 0.
Adding a condition isn't moving the goalposts. I still think the call was right.
And that strengthening made the experiment structurally undecidable. The gate now has exactly two possible endings:
- Kill fires (cumulative excess ≤ −15%p, or drawdown > 40%) → stop
- Undecidable forever — PASS is unreachable
Kill isn't likely either. With monthly sd of 4%p, the sd of three-month cumulative excess is about 6.9%p, and the −15%p threshold sits roughly 2.2σ below zero under the null.
So I was running an experiment that could never end. Rebalancing monthly, green light on the dashboard, the appearance of "under validation" maintained.
What would you do here?
Three branches:
- Loosen the gate back to the sign comparison.
- Accept the backtest as the final evidence and go live with a small amount.
- Turn it off.
Option 1 is off the table. That's changing the bar after seeing results, and that sign comparison is the exact rule that was reading noise as a win.
Option 2 was a serious candidate. The backtest is all the evidence this strategy will ever have — forward paper trading cannot add statistical evidence to it. Which means continuing to paper-trade spends time without producing information.
I chose option 3.
−3.43%p does not mean it's losing
One thing I want to be precise about while shutting it down. The −3.43%p cumulative excess in the ledger is not a loss. Against a monthly sd of 4%, that's 0.85σ. Pure noise.
"Undecidable" and "failed" are different words. This bot did not fail. It was designed so it could neither succeed nor fail.
That makes it different from killing a bot because even winning wouldn't beat the friction. There the problem was the size of the edge — winning still fell short of the minimum effect. Here size isn't the problem; the signal-to-noise ratio is. No realistic horizon decides it.
It's the same family as designing an experiment that was statistically impossible, except that time I caught it before launching. This time I caught it a month in. The only difference is when I did the arithmetic.
Three self-checks
- Is the required-sample field on your experiment filled in? If it's empty, is that because you can't compute it, or because you didn't?
- If forward data is too thin to estimate variance, can a backtest or historical log estimate it? Usually yes.
- What is the ratio of your minimum effect to the noise per observation? That single ratio sets the horizon — at 1:10 you need hundreds of observations.
The honest part
This calculation took ten minutes. Done before launch, this bot either wouldn't exist or would have been designed around a different statistic — rank correlation, hit rate, something with far more observations.
I skipped it not out of laziness but because I had built the field. The field existed, so I trusted it would fill itself. null looks like "coming soon," not like "nobody asked."
And one generalization. A monthly-rebalanced equity factor strategy cannot be validated by a t-test on monthly excess within one person's timeframe. That's a property of the test design, not of the strategy. However good the backtest is, a plan to "prove it forward" needs the arithmetic first.
The code, backtests, data, and ledger all stay. What I deleted was the schedule and one row on a dashboard.
Do one thing now: pick an experiment you're running and compute minimum effect ÷ sd per observation. Roughly eight times the square of that ratio's inverse is the number of observations you need.