My paper-trading experiment's pass conditions included beats_qqq: beat a QQQ day-trading comparator under the same rules and costs. The verdict table showed QQQ day-trading at -8.93%, my strategy did worse, so it failed anyway.
But during verification something looked off. The comparator's round-trip count: zero.
Does the comparator inside your gate actually trade?
All 68 sessions: INSUFFICIENT_CASH
The cause was arithmetic. Converting the 1,000,000 KRW budget gives about $660; subtract the 2% cash reserve and you have $647. The cheapest QQQ close in the whole evaluation window was $661.73. Under whole-share rules, QQQ was a symbol the experiment could not buy on any single day. Same for SPY at $725+.
So the "QQQ day-trading" comparator sat in dollars for 68 sessions, and its -8.93% was not trading performance — it was dollar weakness measured in KRW. Its USD return of exactly 0.00% is the tell. The protocol said "a universe of four ETFs"; effectively there were two.
beats_qqq wasn't measuring "did you beat QQQ day-trading" — it measured "did you beat idle cash." My experiment lost even that by 22 points, so the verdict stood. But had the strategy beaten cash while losing to real QQQ day-trading, this gate would have issued a false PASS.
The temptation of cheap fixes
Once you see it, three fixes suggest themselves: raise the budget, shrink the universe, switch to fractional shares. All three are rule changes made after seeing results — forbidden inside a preregistered experiment.
I did one thing: added a check that can only make passing harder — comparators_traded: if a comparator's round-trip count is zero, the whole gate FAILs. A comparison with no denominator cannot count as a PASS. Same principle as not moving the bar after preregistering the judgment window; adding it post hoc is admissible because its only direction is stricter.
Which of the three fixes would you have picked — and would it have made passing easier?
Self-check, 3 items
- Do the comparators inside your gate actually fill orders? Print their trade counts in the verdict table.
- Can capital × (1 − reserve) buy the most expensive single share in your universe? That's a one-line pre-launch check.
- If a benchmark's return looks strangely tame, did you split it into strategy return vs currency return?
The honest part
This defect was catchable before the backtest ever ran — one line comparing the budget to the minimum share price. I missed it because I treated the comparator as "the thing that obviously works." I audited the strategy code and never audited the benchmark code. But the gate's denominator lives on the benchmark side.
Check exactly one thing today: find the trade-count column in your experiment's benchmark row. If there isn't one, that's your first finding.