Three Ways My Own Tests Lied to Me

I hit the same species of failure three times in one day. Every time the tests were green, and every time the problem was that the values my tests supplied were cleaner than reality.

Does your test feed what actually arrives, or what was convenient to type?

1. The stub didn't care about `this`

I built a bridge that lets a web page post messages to a native app, trying several transports in order.

const sink = host.webkit?.messageHandlers?.plotta?.postMessage;
sink(json);

All eleven tests passed. Here was the stub:

const host = { webkit: { messageHandlers: { plotta: { postMessage: (s) => seen.push(s) } } } };

An ordinary arrow function. It doesn't need this, so calling it detached works fine.

The real postMessage is bound to its receiver. Detached, it throws TypeError: Illegal invocation. My code wrapped the call in try/catch, so it was swallowed in silence and the native side waited out a 45-second timeout having received nothing.

The regression test now looks like this:

const plotta = {
  postMessage(this: unknown, s: string) {
    if (this !== plotta) throw new TypeError("Illegal invocation");
    seen.push(s);
  },
};

A stub has to be as strict as the real thing or it certifies broken code.

2. The test input was prettier than the caller

A function that folds the app's language into one of the five the web renderer speaks:

let base = identifier.split(separator: "-").first…

The tests:

XCTAssertEqual(webLocale("ko-KR"), "ko")   // passes
XCTAssertEqual(webLocale("zh-Hant"), "zh") // passes

All green. But the real caller passes this:

ReelPayloadBuilder.make(… locale: Locale.current.identifier)

Locale.current.identifier is not a BCP-47 tag — it is ICU form. ko_KR. An underscore. Splitting on "-" alone leaves "ko_kr", which matches nothing, so it falls back to English.

Every Korean and Japanese device was rendering its video in English. The test only knew the hyphenated spelling. A locale quietly drifting and shipping the wrong thing has bitten me before — my Japanese YouTube was advertising a page that didn't exist.

The amusing part: the Android version of the same function used substringBefore('-').substringBefore('_') and handled both. One platform was right, one was wrong, and both had passing tests.

3. The self-verification verified nothing

While changing a database function I added a self-check. The migration makes the function return owner_id.

IF position('owner_id' in src) = 0 THEN
  RAISE EXCEPTION 'owner_id was not added';
END IF;

Looks safe. Then I ran the rollback rehearsal before applying — and the check still passed after the rollback.

Because the original body already contained this:

JOIN plotta.users u ON u.id = t.owner_id

The string owner_id had been there all along. My check was a condition that passes whether or not the change happened. I've written about the check itself lying once before — I added verification, and the verification lied. In numbers:

loose check   position('owner_id' …)                = 284   ← passes before the change
tight check   position('''owner_id'', t.owner_id' …)  = 0     ← fails before the change

Pause here

What do the three have in common? I'd put it this way:

A test proves "it works on the values I supplied." If those differ from the values that actually arrive, passing tells you nothing.

And coverage won't catch it. In all three cases the lines did execute. They executed with the wrong input.

Three things to check

Is your mock more forgiving than the real thing? Does it throw where the real one throws — on this, on null, on out-of-order arrival, on duplicate calls? Wherever the real thing is strict and the mock is lenient, that spot is untested.
Where did your test input come from? If you copied it from documentation, be suspicious. Print the expression the real caller passes and use that value. One log line separates ko-KR from ko_KR.
Does your invariant fail before the change? This is the cheapest check there is. Run the self-verification against the pre-change state. If it passes, it is guarding nothing.

The honest part

The third one was luck. Without the habit of running a rollback rehearsal I would have assumed it passed on merit. The rehearsal exists to answer "is applying this safe" — this time it answered "is my check actually a check."

The first two weren't luck. A code review caught them, with 200 unit tests green.

When did you last watch your self-verification fail?

1. The stub didn't care about this