I made the tests pass and production gained users

I added first-session instrumentation to a map-based walking app. The server only recorded account creation and successes (walk completed, territory claimed), so where new users disappeared was entirely invisible. Opened the app and quit? Denied location permission? Walked but failed to close a loop? Indistinguishable. How I found that state is its own post → The funnel nobody measured.

I defined 8 events, built a dedicated table and a SECURITY DEFINER RPC, wired the client, and wrote 5 unit tests. Ran xcodebuild test. All 184 passed.

Then, out of curiosity, I queried the server table.

first_open        install b4c53bc9  1.5.1
app_open          install b4c53bc9  1.5.1
walk_cancelled    install 3c679dea  1.5.1
walk_cancelled    install 3c679dea  1.5.1
walk_ended        install 3c679dea  {"closed":"true"}
territory_claimed install 3c679dea  {"lobes":"2"}

Instrumentation I had not shipped had written 6 rows to production. Nobody has claimed two territories.

A question: how many rows did your test suite write to production last week? Do you have that as a number?

Two things overlapped

First, xcodebuild test actually launches the host app. Even unit tests run inside the app process, so App.init runs before any test does. The logFirstOpenIfNeeded() I'd put there went straight out over the network. The code that installs an observation sink lives in setUp() — which runs after the app is up. By ordering alone, it can never intercept that call.

Second, view-model tests use real objects. The walk-session test genuinely calls start() and close() — exactly where the instrumentation lives. The lobes: 2 on territory_claimed is the fingerprint of the figure-eight path test creating two lobes and passing. The evidence that the tests worked was itself the contaminating data.

Here's the twist: a funnel is a table that counts human behavior. Run CI once and the conversion rate moves. And it's quiet — the tests are green, the app is fine, and the table gains a user who does not exist.

The guard is XCTest detection, and the ordering is the trap

private static var isUnderTest: Bool {
    ProcessInfo.processInfo.environment["XCTestConfigurationFilePath"] != nil
        || NSClassFromString("XCTestCase") != nil
}

Get the order wrong and it breaks the other way. If isDisabled is checked before the sink, tests can no longer observe the instrumentation — everything is disabled under test, so the sink receives nothing.

static func log(_ event: Event, payload: [String: String] = [:]) {
    if let sink { sink(event, payload); return }   // observation path first
    guard !isDisabled else { return }              // then the block
    // …RPC
}

logFirstOpenIfNeeded needs one more condition. While disabled it must not even touch the firstOpen flag. If a screenshot run burns the flag, that device's real first launch is never recorded again.

guard sink != nil || !isDisabled else { return }

Android is a different language, so a different fix. The dependency is injected, so a mock in the test is enough — but a relaxed mock is mandatory.

funnel = mockk(relaxed = true)   // a real instance writes to the production table

I settled on a single acceptance criterion: after one full test run, the table row count must be 0. Verified exactly that: iOS 184 tests / Android 192 tests green, production table 0 rows.

Side trap — verifying an ipa with strings is only half true

If you grep the binary with strings to confirm "did the instrumentation make it into the build," you will misjudge it. Swift inlines literals of 15 bytes or fewer as small strings, so they never appear as binary constants. first_open and walk_started come back with zero hits while location_permission_asked and log_funnel_event show up. Short event names look missing. If you use this as an existence check, anchor on one long symbol.

Three things to check right now

1. Run your full suite, then immediately count rows in your production telemetry table. Anything other than zero is your answer.
2. Does your app entry point (init / Application.onCreate) make network calls? That runs before your tests' setUp().
3. Verify your test-disable flag isn't checked before the observation sink. If it is, your tests can't verify instrumentation at all.

The honest part

I had the screenshot-mode guard from day one. I anticipated that store-asset capture would contaminate data; I did not anticipate that tests would do the same thing. Same class of risk, and I'd only blocked one side. There turned out to be a third contamination source too → App Review counted as new users.

I also didn't find the contamination by design review. I found it by idly querying the table because I was curious about the results. Without that, it would have shipped and I'd have started the baseline with 6 ghost users mixed into day one.

10 rows total went in and I deleted all of them — two DELETE runs against a production table. It was telemetry, so nothing was lost; had it been payment or user data, the same mistake would have cost something entirely different.

When porting to Android I knew about the trap and still learned it from the compiler breaking first (a constructor gained a parameter and two tests failed to compile). In a language that doesn't catch that, I'd have been bitten again.

What that telemetry actually showed is the first-session funnel in the 113-day ledger post.

Run CI once and select count(*) on your production event table. If it isn't zero, there are people living in your funnel who do not exist.