My Control Group Grew 4.5x Without Me — a PASS Would Have Taught Me Nothing

I deployed 227 parameter pages: 115 amount-by-rate combinations on a loan calculator, 111 percentile pages on a salary app, and one take-home pay table. The data already existed — I only gave it URLs, so the build cost was close to nothing.

Before starting, I wrote down the verdict rule.

Naver long-tail sessions (excluding home landings), 28-day total, must reach +30% over a baseline of 173 (≥225). The window opens 14 days after deploy and runs 28 days.

I pushed the window back to absorb indexing lag, and I put the definition of "long-tail" into code rather than into prose. Deciding a definition on judgment day is moving the bar after the fact.

One question first. Does your A/B threshold know what the number would have been if you had done nothing?

I looked at the numbers before the window opened

The window opens two weeks after deploy. Curious about how the channel was moving, I split sessions by app — and summed only the apps I had not touched.

Untouched apps, Naver long-tail sessions
  14 days ago:   56
  now:          251

4.5x. On apps where I did nothing.

The cause surfaced fast. Four days earlier I had done portfolio-wide SEO work: cleaning up root-path 308 redirects and adding locale links to the footer. That lifted the whole channel.

So what happens to my verdict rule? A baseline of 173 plus 30% is 225. If the channel is on its way to 4.5x, 225 gets cleared even if those 227 pages produce exactly zero sessions.

The danger is not that it might FAIL. It is that a PASS would not tell me why. A verdict you cannot learn from is a cost with no return. I had been careful with thresholds ever since a gate with no denominator; this time the rot was not in the denominator but in the comparator.

Three ways out

Which would you pick?

1. Raise the threshold. +30% can't filter the trend, so make it +80% or so.
2. Push the judgment date back until the trend settles.
3. Keep the threshold and the window; change what it compares against.

Option 1 is the most common and the worst. Raising a pre-registered bar before results land is the same sin as letting a delay make the bar harder, just pointed the other way. That number has no basis. The moment I hand-write "what it takes to beat the trend" into the threshold, my mood decides the next verdict.

Option 2 is also bar-moving. Window length is part of statistical power.

I took option 3. Put the same +30% bar on top of the trend.

TREATED_HOSTS = ("loan.ootssu.com", "salary.ootssu.com")   # the 227-page expansion
 
# Measured decomposition of the 28 days before deploy
#   treated 32 · control 127 · expanded 14  -> 173 = BASELINE (window definition checks out)
BASE_TREATED = 32
BASE_CONTROL = 127

The verdict becomes:

expected = 32 × (control_in_window / 127)
PASS  ⇔  treated_in_window ≥ 1.30 × expected

The 1.30, the Sep 22–Oct 19 window, and the long-tail definition are all untouched. Only the comparator moved, from "a fixed baseline" to "what the control group predicts." And this is not moving the bar after the fact — the judgment window has not opened yet. I did not look at outcome data and adjust; I fixed the comparator before any outcome data existed. That distinction is the whole thing.

I left the original absolute threshold in place. Pre-registered is pre-registered. But the conclusion gets read off the second metric.

I was the one contaminating the control group

Then I caught myself again. The same day I added the correction, I shipped 30 compatibility pages on a different app.

That app is in the control group. But a control group has to be "the trend among apps that did nothing." Let an app that expanded its own pages sit inside the judgment window and the control inflates, expected grows, and the bar the treated group must clear gets harder on its own. Bar-moving again, just pointed the other way.

That is no reason to freeze expansion for 40 days, though. Just subtract the expanded app from both sides.

# An app that expanded itself inside the window is not a control.
#   attach  2026-09-10, 30 compatibility pages. /ko/pair/... counts as long-tail.
EXPANDED_HOSTS = ("attach.ootssu.com",)

I corrected the baseline decomposition along with it. The control had been recorded as 141, which included this app's 14. Subtracting it from the window while leaving it in the baseline makes the numerator and denominator different populations, and the ratio breaks entirely. So the control drops to 127. 32 + 127 + 14 = 173, matching the original baseline.

The rule itself went into a code comment: every future expansion adds its host to EXPANDED_HOSTS. Otherwise the verdict quietly gets more conservative — and "quietly" means I miss it again next time.

Finally I guarded the division. If the control is 0, expected is 0 and any value passes. In that case the script returns "cannot judge." Substituting 0 for a measurement failure leaves a wrong verdict in the ledger.

Three checks for your own setup

1. Did you actually count the control group? "I didn't touch anything else" is a memory, not a number. I only learned about the 4.5x after counting.
2. Did you change something else inside the judgment window? Remove that target from the control, and fix the baseline decomposition to the same population. Subtracting from one side only breaks the ratio.
3. Is the fix landing before or after the window opens? Before is a design correction; after is moving the bar. The same code change is guilty or innocent depending on when.

The honest part

This post has no result. Judgment day is October 20 and the window has not even opened. I do not know whether those 227 pages worked.

What I fixed is not the effect but the ability to see the effect. It was not impressive work — a few constants, one host list, two commits. And precisely because it was not impressive, skipping it would have handed me a green light on October 20, no understanding of why, and a decision to build the next 227 pages on that.

If you have an experiment running right now, pull the control group number once. Pulling it on judgment day is too late.