The Pass Bar I Set Came From a Different Metric

I shipped a small web tool this week. It builds construction quotes, it's free, and it is nine pages total.

Before launch I preregistered a gate — the habit of writing down when I'll kill this and when I'll invest more, before seeing any data. Then I rewrote that gate the day after launch. And again the day after that.

I moved the bar twice. So this isn't a success story, it's a record of how a threshold ended up meaning nothing, twice.

One question first. That success criterion you last wrote down — where did the number come from? A measurement of something comparable, or a feeling that this much would count as working?

Draft 1: a bar my best app couldn't clear

The first version read:

PASS = 300 prints + 1,000 unique visitor tokens within 90 days

It looked reasonable while I typed it. Then I opened GA4 on my other apps. My top performer does 280 sessions per 28 days. Second place does 209.

I had asked a brand-new domain path to reach top-of-fleet traffic within 90 days. That bar was unclearable.

The important part: an unclearable bar isn't strict, it's meaningless. It produces a FAIL on judgment day, and that FAIL tells me nothing about the product, because the answer was fixed in advance. I'd spend 45 or 90 days re-buying something I already knew. I know this smell — I once designed a gate that needed 59 years.

On top of that, the futility metric was "search engine impressions", a number that lives only in a console UI with no API. A metric a human has to eyeball can't be a gate. A verdict that doesn't run automatically doesn't run on judgment day.

Draft 2: I had a comparable. The unit was wrong.

So the next day I rewrote it, this time against a comparable. I took my best-trafficked app and set the bar around half of it.

PASS = 150 search sessions + 30 prints in 90 days
Stop (futility) = 10 or fewer search sessions in 45 days

A day later I was reviewing that draft and stopped. The 150 came from that app's all-source sessions — threads, Instagram, direct, search, everything summed.

But what I'm measuring is search sessions. The new tool has no social distribution wired up at all, and this gate asks exactly one question: does search bring anyone.

I was using an all-source number as a search-only threshold. Not the same unit. I thought I had picked a comparable; what I actually did was carry a number over from a different metric.

Measuring again, in the same unit

I pulled four apps of the same kind (calculator/lookup style tools) from GA4 as 90-day search sessions, with page counts, because yield is the thing that matters.

Median 19, ceiling 39.

My 150 was 3.8× the observed ceiling. And the app that produced that ceiling has 25× more pages. I had asked a nine-page tool for four times what a 224-page app achieves.

Two options. What would you do?

1. Leave it. Moving a preregistered bar violates the whole point.
2. Fix it now, and record the condition: observation is zero and the judgment window hasn't opened.

I took option 2, and that distinction is the point of this post. The reason you must not move a bar is that you're reacting to data in your own favour. Fixing it when search observation is zero and the site isn't even indexed is not that. Conversely, making the identical edit after indexing arrives is post-hoc bar moving, and it looks exactly the same from outside. Timing is part of the rule — I learned that when slipping a verdict by two days made my own bar harder.

The new threshold sits between two groups

The four measurements split cleanly: page-heavy tools getting 34–39 from search, near-pageless ones getting 1–4. So I put the threshold between them and computed the power with a Poisson model.

PASS threshold 20
  if the true rate is the 1-4 group      P(pass) = 0.000
  if the true rate is the 34-39 group    P(pass) = 0.996 - 1.000
  if the true rate is the median, 19     P(pass) = 0.44

I'm leaving that last line in. On the boundary it's a coin flip. A threshold placed between two groups cannot resolve a product that genuinely sits between them. No threshold design fixes that; only more sample does.

The futility side was worse.

Old futility: stop if 45-day search sessions <= 10
  if the true rate is exactly at the PASS bar (expect 10 in 45 days)
  P(killing it wrongly) = 0.5

One half. A stop rule that kills a product sitting exactly on the pass line half the time. A stop rule should only fire on "this is clearly not happening". I dropped it to 2.

New futility: stop if 45-day search sessions <= 2
  true rate at the pass line   P(wrong kill) = 0.003
  true rate at the floor       P(stop)       = 0.677

I removed the 30-print condition entirely, for an embarrassing reason: it contradicted itself. Requiring 20 search sessions and 30 prints in the same window means each session prints 1.5 times. Both conditions cannot be true at once.

Instead I measure the print rate, with a denominator clause: if unique landing visitors are under 40, the rate isn't computed and the gate returns undecidable. Judging on a numerator alone rejects missing distribution rather than a bad asset — I know because a gate with no denominator nearly killed a game of mine.

Three checks for your own gates

• Pick a gate you're running now. Which measurement produced its threshold? Is that measurement in the same unit as what you're about to measure?
• Have you computed the pass probability under both hypotheses? If not, you don't yet know whether that bar is strict or loose.
• If you have a separate stop rule, what's the probability it kills a product sitting on the pass line? If it's 0.5, that's not a stop rule, it's a coin.

The takeaway

Both drafts failed the same way. I treated "setting a threshold" as picking a number. The actual job is picking what you compare against, and if the comparable is a different metric, the number measures nothing.

The real meaning of this verdict only got sharp at draft 3: a nine-page tool clearing 20 sessions means 25× the per-page yield of a 224-page app. That's the only condition under which investing in more pages is justified. Neither of the first two bars could express that sentence.

The honest part: I've edited this gate twice and from here it changes only through commits. Observation is still zero search sessions (pre-index). Futility fires in 45 days, the real verdict in 90, and the 0.44 pass probability near the median is still sitting there. That's not a solved problem, it's a limit written down honestly.

Do one thing right now. Next to your own success criterion, write one line saying where that number came from. If you can't write the line, you haven't picked a comparable yet.