My Counterfactual Was Only Counting the Winners

Paper experiment. Sample counts and ratios below come from the real ledger; dollar PnL is generalized.

I opened the dashboard and two curves were climbing together. One was "hold to expiry," the other "close early, just before expiry." At the same moment, the official report showed both bots deep in negative cumulative PnL, with the pre-registered gate already reading KILL.

The chart and the ledger flatly disagreed. One of them was lying.

Have you ever checked whether the totals in your derived metrics match the raw numbers in your ledger?

Suspect the cheap explanation first

My first guess was "the chart must be plotting a different date range." It wasn't. Both samples spanned the same days.

Second guess: "the sample is just small." It was small — only 299 of 415 settled positions made it into the curve. But a small sample isn't a crime on its own. The question is what got left out.

Splitting it:

A 12.9% win rate in the dropped bucket. That is not missing-at-random. The losers were being selected out.

The culprit was one validation line

The counterfactual works like this: about 30 seconds before expiry, snapshot the best bid on the position and record what "sell everything right now" would have paid. Later, pair that against the actual settlement.

The snapshot function opened with this guard:

if bid is None or not math.isfinite(bid) or not (0 < bid <= 1):
    return False

0 < bid. It reads like ordinary defensive code. Prices should be positive, right?

Except in a prediction market, 30 seconds before expiry a losing position's bid goes to zero. Nobody is buying. At that exact moment the guard throws the snapshot away. And the aggregation query was WHERE cf_exit_pnl IS NOT NULL.

Put the two together and the filter's real meaning becomes:

"Measure only positions that still had a bid near expiry" = "Measure only positions that mostly won"

Controlling for price didn't rescue it. In the 0.30–0.60 entry band where most trades sat, the included rows won 130/218 (59.6%) and the dropped rows won 8/75 (10.7%). The outcome, not the price, was deciding what went missing.

How would you fix it?

There's a fork here. For those 116 dropped rows, there is no way to recover what the bid actually was. It was never written down.

1. Keep excluding them and add "some rows excluded" to the caption
2. Include them in the sample, filling the unknown value conservatively
3. Throw out the historical data and re-count from the fix forward

I took option 2, for a specific reason. A position dropped because its bid was zero would have produced exactly the same result either way. Selling at zero loses the full stake; holding to expiry loses the full stake. The true difference for those rows is 0 — not unknown, but known.

So the fix touched two places:

• Change the snapshot guard to 0 <= bid <= 1, so a zero bid is recorded as the real observation it is
• Change the aggregation sample to every settled position, filling rows without a snapshot as early-close = hold (delta 0)

This has a useful side effect: it creates a consistency anchor. The "hold PnL" in the counterfactual table must now exactly equal the actual PnL in the main gate table. If they ever diverge, the sample definition is leaking again.

After the fix, the story changed

Both curves now go down together. Both bots are negative on hold and negative on early close. On one bot, early closing recovered part of the loss; on the other it made things slightly worse.

More important: the adoption gate read AMBIGUOUS before the fix and AMBIGUOUS after it. That is the reassuring part — if removing the bias had flipped the verdict in my favor, that wouldn't have been de-biasing, it would have been post-hoc selection. This is what pre-registration buys you.

The remaining bias deserves a note too. The handful of winning positions dropped because the quote fetch itself failed got filled with delta 0, when their true delta was almost certainly negative (they'd have sold below 1). So the leftover bias now favors early closing. That makes the setup insufficiently conservative for concluding "early close is better" — which weakens my own conclusion, which is exactly why it goes in the write-up.

And then the gate itself became pointless

Once the numbers were honest, a more uncomfortable question surfaced: what was this gate deciding in the first place?

The original purpose was "is it worth building a sell-before-expiry path?" (the early-close gate post). With honest numbers, the ceiling on early closing is roughly 2% of stake per trade — enough to move ROI from −8% to −5%. It makes a losing strategy lose less; it does not make it win.

Running an adoption gate for a base strategy that's already KILLed is momentum with no destination. I could only see that after fixing the measurement. That's also why killing failed bots cleanly is hard every single time — the moment to kill usually hides behind the sentence "just a little more data."

A three-line self-check

Hold your own measurement code against these.

1. Is missingness correlated with outcome? Collect the rows your aggregation drops and compare their win/success rate to the rows it keeps. A large gap means you're looking at your filter, not your sample.
2. Do you distinguish zero from absent? Checks like if not value swallow "value is 0" and "value is missing" together. In domains where zero is meaningful — prices, remaining size, counts — that's silent data loss.
3. Does a derived total tie back to the ledger? At least one number in your derived table should equal a raw number in the source of truth. Without that anchor, a leaking sample definition is invisible.

The honest part

This was not an arithmetic bug. The arithmetic was correct the whole time. What was wrong was the rule deciding what got counted, and that rule was hiding in a line that looked like defensive code. The tests passed, too — they verified "does it store a valid bid correctly," never "what do we call valid."

For three weeks I watched that curve and thought there was something to early closing. What was actually there was a filter.

Go find the one if statement in your pipeline that's quietly dropping rows. I'd like to hear what the guard was.