I Delayed the Verdict by Two Days and My Bar Got Harder on Its Own

Everyone guards against moving the bar after the fact — quietly lowering a threshold once you have seen the result.

Today I found I had been committing the same sin in the opposite direction. Waiting was raising my bar on its own. Nobody touched the threshold.

Does your gate script know its threshold only, or does it also know the range of its denominator?

What was preregistered

A bot that drives a phone directly kept failing at account switching. I named one cause, fixed it, and wrote down the rule for judging the fix before running it.

PASS: zero failures in the first 32 switch attempts after the fix.
      If the baseline (9.5% failure rate) holds, 0.905^32 ≈ 0.041 — hard to call luck.

The 32 was not arbitrary. It is the sample size derived from the baseline failure rate. The judgment date was merely an estimate of when those 32 attempts would accumulate, based on throughput. It was never part of the rule.

The date slipped

On the first judgment date only 24 attempts had accumulated. I had planned for 16–24 per day; the measured rate was 12. Denominator short, so no verdict — that part I handled honestly. The bar stayed where it was and only the date moved two days out.

Two days later I ran the script.

after fix: bursts 12 · attempts 51 · failures 1
  primary metric: 0 failures in 32 attempts — currently 1/51
   → FAIL

FAIL. And I nearly wrote it straight into the ledger.

The failure was at attempt 50

I checked what the script was actually counting.

for ln in open(LOG):
    if ln.startswith("##### account"):
        att += 1
    elif "switch failed → skip" in ln:
        fail += 1          # ← no window

It counts everything after the fix timestamp. The number 32 appears only in the denominator gate att >= 32 and plays no part in tallying failures.

So I pulled out which attempt the failure landed on.

FAIL at attempt #50
total att 51

Number 50. The preregistered window is attempts 1 through 32. Inside the window there were no failures.

Pause — what would you do?

You are holding these two lines.

Rule: "zero failures in the first 32 attempts"
Data: 1 failure out of 51, at position 50

Do you record FAIL, or PASS?

Honestly, I hesitated. Editing the judging script on judgment day so a FAIL becomes a PASS looks exactly like the behavior I was most on guard against.

The two directions are symmetric

Laid out plainly it was simple.

Without the window, failures accumulate the longer the verdict is delayed. Every extra day is another chance to fail, and passing "zero failures" gets correspondingly harder. Nobody touched the threshold and the bar still rose.

If lowering a threshold after the fact is a sin, raising it by waiting is the same sin with the sign flipped. In both cases something not agreed in advance entered the verdict.

So I restored the window the rule had specified.

WINDOW_ATT = 32          # judgment window for the primary metric
 
elif "switch failed → skip" in ln:
    (fail_in if att <= WINDOW_ATT else fail_out).append(att)

Failures outside the window are not counted — and not hidden either. They get their own line.

after fix: bursts 12 · attempts 52 · in-window failures 0 · in-window resets 3
  [outside window · excluded] switch failures 1 (attempt #[50]) · resets 14
  primary metric: 0 failures in 32 attempts — currently 0/32, baseline expects 3.0
   → PASS

The evidence that the restoration was right came from somewhere else

I needed an answer to "you just changed it until you got the result you wanted." Reasoning alone does not settle that.

Luckily this experiment had one verdict already written to the ledger — the secondary metric, judged three days earlier.

2026-09-04 07:00 switch-fix verdict: secondary FAIL (bursts 3 · attempts 12 · failures 0 · resets 3)

Run the windowless version today and it reports 17 resets. That does not match the ledger's 3. Restore the window to three bursts and it reports 3. Exact match.

So restoring the window is not a convenient post-hoc adjustment. It is returning to the calculation this script was doing three days ago. The thing that drifted in between was the bug.

That was the strongest evidence available to me. Not an argument — a reproduction.

A bonus bug: the silence guard swallowed a whole verdict

It bothered me that the hourly summary line had never once announced a verdict. Here is why.

if os.path.exists(VERDICT_FILE):
    sys.exit(0)

"Announce once when the verdict lands, then stay quiet" — the intent is right. A finished gate that keeps pinging forever is noise.

The problem is that this experiment had two metrics. The secondary metric landed three days early and created the file, and from that moment the primary metric was muted permanently, verdict or not.

Silence needed to be keyed per metric, not per file.

said = open(VERDICT_FILE).read() if os.path.exists(VERDICT_FILE) else ""
done = [x for x in (("secondary", v2), ("primary", v1))
        if x[1] != "no verdict (denominator short)" and f"{x[0]} " not in said]

Three checks on your own gate

Open the gate script that is running right now and spend three minutes.

Is the window a constant sitting next to the threshold? If you have THRESHOLD = 3 but no WINDOW = 32, that script returns a different answer the longer the verdict is delayed.
Run it today — does it reproduce last month's verdict? If not, one of the two is wrong, and until you know which, do not trust today's number.
What key does your "announce once" guard use? If it is file existence, the moment you have two metrics one of them disappears quietly.

The honest part

I do not feel good that this post lands on "it was a PASS." It is uncomfortable.

The fact remains that I edited the judging script on judgment day and a FAIL became a PASS. All I can attach to it is three things — the rule text predates the code, the corrected version reproduces the older ledger entry, and the out-of-window failure was reported rather than erased. Without those three I would have taken the FAIL as written. The habit of resolving ambiguity in your own favor costs far more than one verdict.

And that failure at attempt 50 has not vanished. It got its own autopsy, written up in The Label Was Wrong So I Switched to Counting. The Count Was Wrong Too. Excluding it from the verdict means "different question," not "never happened."

Open your own gate script and check just one thing: whether a window is written next to the threshold. If it is not, that gate gives a different answer today than it will next week.