I read five store product page A/B experiments on the same day. All five treatments have the identical shape: keep the order of the 8 screenshots, synthesize new captions.
An earlier caption experiment on a different app had won big. I was about to decide whether to roll this out across all 46 apps in the fleet.
One question first. When you move a lever that won in one app to another app, what makes you believe it transfers?
1. The same lever pointed in opposite directions
| App | Control arm (unique impressions / conversion) | Treatment | Improvement | z | Call |
|---|---|---|---|---|---|
| A | 425 / 5.55% | 447 / 3.20% | -42% | -1.7 | stop |
| B | 105 / 1.63% | 116 / 6.92% | +325% | 1.9 | keep |
| C | 498 / 3.12% | 513 / 4.55% | +46% | 1.2 | keep |
| D | 30 / 6.62% | 31 / 6.48% | -2% | -0.0 | stop |
| E | 285 / 2.94% | 282 / 2.78% | -5% | -0.1 | stop |
Captions are not a fleet lever. They have to be measured per app.
Using the earlier win as justification was also wrong. That precedent was icon + captions; these five were captions alone. Not the same lever. Sharing a name doesn't make it the same thing.
2. A's -42% has clean attribution
I re-read the asset generation script. It keeps the order of all 8 screenshots and only synthesizes captions. The "move the reward scene earlier" note from the diagnosis was never implemented. One variable, so this number belongs to the captions.
The decisive evidence was the platform CSV's best improvement estimate:
9/3 +0.373 → 9/5 +0.097 → 9/6 +0.072 → 9/7 -0.0022 ← the interval excludes 0 for the first time⚠️ Do not read the platform's displayed "confidence" as win/lose. Losing app A had the highest at 7.5%, and B, at +325%, showed 0%. All five read "collecting data" in the UI. The calls came from the improvement interval and a test I computed myself.
3. The stop criterion was "unreadable", not "losing"
D and E are perfectly flat (both |z| < 0.2). But flatness isn't why I stopped them. Even running to the automatic end date, they won't reach 5% of the required sample.
An experiment that cannot produce a result is holding the one experiment slot that app has for two more months. That's the entire cost, and that's why stopping is right.
B's +325% is not a verdict yet. It hasn't cleared the impression guard (300 per arm), expected in 22 days, so I kept it. This is exactly the spot where a big number begs to be used as a conclusion.
4. And my power calculation was 9x wrong
A few days earlier I had computed with "364 impressions/day". The actual unique impressions were 42/day.
That 364 is the daily report's pre-dedupe number — the same person counted every time they looked. Experiment power must be computed on unique impressions.
This is a sequel to the experiment that had no statistical power. That post is "I did the calculation." This one is what happens when the input to that calculation is wrong. The procedure was fine. The number I fed it was counting something else.
Worse: I had already documented this trap myself. Working with the same experiment API as the list endpoint that said not started, I walked straight back into the same report's dedupe problem.
5. Which app would you put an experiment on?
46 apps. One experiment slot per app. One experiment can consume up to 90 days.
- (a) The app with the highest revenue per install
- (b) The app with the most impressions
- (c) The app that looks like it has the most room to improve
The answer is (b), and it fights intuition. D has the highest revenue per install in the fleet — and 61 unique impressions over 21 days (3/day). There is no traffic to improve with an A/B. What D needs is discovery (keywords, category), not an experiment.
6. So I set the entry bar as a number
25 unique impressions per day.
To read a +50% improvement on a 5% conversion within the 90-day cap you need roughly 1,100 per arm, and that's 25/day (12.5 per arm).
⚠️ 25 is the floor for "only large effects", not headroom. A had 44/day and still couldn't read +30% in 90 days.
If all you have is pre-dedupe daily impressions, divide by 8.7 for an estimate (calibrated on A: measured 364/day = 42 unique/day). On another app the estimate was 3.3 vs 2.9 measured, but it swings ±50% depending on the app.
Three of the 46 apps clear the bar, four are borderline, and the remaining 39 are all under 10/day. ⚠️ Do not attach experiments to newly launched apps — some had literally 0 impressions in the 28-day window. For those apps the bottleneck is discovery, not the page, and an A/B cannot fix it.
7. The stop API is a flag, not a state
This cost me real time.
PATCH /v2/appStoreVersionExperiments/{id}
{"data":{"type":"appStoreVersionExperiments","id":id,
"attributes":{"started": false}}} # 200, state -> STOPPED
{"attributes":{"state":"STOPPED"}} → 409 "'state' can not be included in a 'UPDATE'"
"type":"appStoreVersionExperimentV2" → 409 WRONG_TYPE # the path is v2, the type is not- A running experiment reads
APPROVED, notIN_PROGRESS. Read the list alone and you'll conclude it hasn't started. - Stopping is irreversible. There is no resume; you create a new experiment. Stopping pulls the end date to the stop time and the live listing reverts to the original assets.
What I have not seen
One of the five can only be read through the UI, and my access to that console is blocked, so I did not see it myself. A human has to hand me the numbers. The table above reflects only what I read directly from the API.
Three things to check in your own numbers
- Is the impression count in your power calculation unique? If your report defaults to pre-dedupe, your required duration is off by a multiple.
- Write the winning treatment out as one sentence. "Icon + captions" and "captions" are different levers, and when you port it, half of it silently stays behind.
- Will the experiment you're running now reach its required sample by its end date? If not, the only thing it does is occupy the slot.
The honest part
If I hadn't read all five on the same day, I would have seen A's -42% and concluded "captions are bad". Had I only seen B, I'd have concluded "captions are great". Both give you the same confidence from the same kind of evidence.
"One data point isn't a verdict" is advice everyone has heard. I didn't know how hard it applies until I put five side by side and watched one treatment produce -42% and +325% at once.
Measure one thing today. Check the daily unique impressions on your running experiment and divide the required sample by it. If the answer exceeds 90 days, that experiment will not produce a result.