Paper-only experiment. PnL/win-rate figures are generalized here; exact numbers and charts come from the bot's own daily gate report (never fabricated).
I run a paper bot where an LLM makes the trading judgment, and I evaluate it through two separate pre-registered gates. The interesting one is the early-close gate.
Two gates, two questions
- Main gate — is the bot's raw edge real? It needs a minimum sample (below it, the verdict is INSUFFICIENT), and it has a kill threshold on cumulative PnL. Below the line long enough, the bot dies.
- Early-close gate — a different question: for the same trades, does closing early beat holding to resolution? This gets its own counterfactual comparison, a minimum count, and a confidence-interval lower bound that has to sit above zero before I'd adopt early-close.
Keeping them separate matters: a bot can have a real edge (main gate) while early-closing is still not proven better than holding (early-close gate), or vice versa.
Reading the early-close gate honestly
The early-close verdict lived in AMBIGUOUS for a while — the point estimate favored closing early, but the confidence-interval lower bound stayed below zero, so the direction wasn't established. A good day would pull the lower bound up; a bad cluster would push it back down. The pre-registered rule keeps me from calling it early: adopt only when the lower bound clears zero at the required sample, not when the average merely looks favorable.
Source: the paper bot's daily gate dashboard. Bot names and dollar PnL masked; win-rate, gate verdicts (KILL/AMBIGUOUS), and the hold-vs-early-close curves are shown. Paper only.
The honest part
The temptation with an LLM-in-the-loop bot is to over-trust it because the reasoning sounds good. The gates don't care how good the reasoning sounds — they read the realized outcomes. Separating "is there an edge" from "is early-close better than holding" stops me from bundling two uncertain claims into one hopeful story. Neither gate has told me to go live, and that's a result, not a delay.