App Review showed up in my funnel as new users

I submitted a build with first-session funnel instrumentation to the App Store — 8 events counting, per device, whether the app was opened, how location permission was handled, and whether a walk was started and then failed.

Right after submitting I queried the server table out of habit. It's pre-approval, so it should be empty.

first_open  install b4c53bc9  ios  1.5.1  07:03:05Z
app_open    install b4c53bc9  ios  1.5.1  07:03:05Z
first_open  install 40da83f2  ios  1.5.1  07:04:06Z
app_open    install 40da83f2  ios  1.5.1  07:04:06Z

Submission was 06:58Z. Five minutes later, two devices opened a version that isn't public.

A review device is indistinguishable from a real one

1.5.1 was waiting for review. There are no external TestFlight testers. I didn't run it on my own device. And yet two distinct install_ids launched the app a minute apart.

App Review ran the app.

Obvious in hindsight, and still a blind spot in instrumentation design. Review is done by a person or by automation, and either way their device looks exactly like a real user's. More so with device-level identifiers — new install, first launch, then subsequent events, all tracing the same shape as a new user.

The scale makes it clear why it matters. This app's 28-day iOS installs are 85. Two review devices are 2.4% of new users. I release often, so I get reviewed often. And review devices by definition don't go for a walk — there's no reason to go outside and close a loop. So they contribute in exactly one direction: pushing down the very metric I was trying to measure (reach of first walk).

It's the kind of contamination that quietly sits in a conversion denominator and skews the conclusion one way.

How would you handle it? Filter it out silently, or leave it in the table where you have to see it every time?

Expose, don't filter

I chose the second. The reader tool now always prints a platform-and-version breakdown.

select platform, coalesce(app_version,'?') app_version,
       count(distinct install_id) installs, count(*) events,
       min(created_at) first_seen
  from funnel_events
 where created_at > now() - interval '28 days'
 group by 1,2 order by 3 desc;

With this line printed above the output:

platform/version breakdown (an unreleased version here is review traffic)

Rather than quietly correcting numbers, the person reading has to make the call each time. Apply a silent filter and later nobody knows what it removed.

On Android the same problem wears different clothes. Play's review gate is weak, so internal-track testers play the same role. They're separable by version too, so the rule carries over.

Three things to check right now

1. Split your last 28 days of funnel data by app version. Any version that isn't public yet is review traffic.
2. Count the contamination sources in your new-user denominator — review, test suite, screenshot runs, internal tracks. What percentage?
3. Do you have silent filters in place? Do you print how many rows they removed alongside the results?

The honest part

I didn't design for this. I guarded against screenshot-mode contamination and test contamination, and review never crossed my mind.

And there's no way to filter it completely. There's no signal that identifies a review device. What I use is one proxy: "events from a version that isn't public are review." But right after approval that version becomes the public one, so a human has to mark the approval boundary. It doesn't automate. I could filter on IP or device model, but I don't collect those — expanding the personal-data surface to see one funnel is a bad trade. The honest position is that the first few days after approval are unusable.

One good thing came out of it. Review running the app is end-to-end proof that the instrumentation chain is alive. I got "does the RPC actually connect" verified for free, before approval. Android I could check on an emulator; iOS had no real-device path until approval — and review did it for me.

The first seven days of that funnel, and the fleet-wide numbers around it, are in the 113-day ledger post.

Right after your next submission, query your event table. There may be people standing in a place that should be empty.