My apps cross-promote each other with a banner. Promo entries live in a remote JSON file, and each app renders the first entry matching its conditions.
This config broke quietly a few weeks back, so I added a test.
every app in the fleet has exactly one promo it will render
Green. It stayed green while the inventory grew from 10 entries to 21.
Which direction does your coverage test count in?
Counting the other way
The selection logic draws the first entry that passes its filters. So an entry can be permanently preempted by one above it.
I counted by entry instead of by app.
entries reaching zero apps: 8 of 2138% of the inventory was dead. And those 8 happened to be the promos for apps I had just launched.
Why the test passed
The test runs app → promo. As long as every app sees something, it is green.
You can add entries forever and it will never catch entries that nobody sees. The two directions do not imply each other.
- every app has something to show → true
- every entry has somewhere to appear → false
Two causes
Ghost targets. Some entries targeted apps that do not carry the banner at all. Seeing an app's name in a config file makes it easy to assume that app renders the banner; that is a separate question. Four were in this state.
Overlapping targets. When several entries aim at one app, the earlier one wins. One entry had four targets: two ghosts, and two already claimed by entries above it. Net reach: zero.
A fork in the road
Two ways to fix it: change the selection to rotate among matches, or keep targets disjoint so the outcome is deterministic.
Which one?
I took the second. Rotation needs state and makes debugging harder. With 1:1 disjoint targets the result is fixed regardless of order. One fallback sits at the end so an app not covered by any rule never sees an empty slot.
And I added three tests in the other direction.
every promo reaches at least one app
target lists are disjoint
every target is an app that carries the bannerA list is not a measurement
That third test matters. The source of truth for "does this app carry the banner" is not the config file — it is the call site in code.
Counting that had its own trap. The component file's own doc comment contains a usage example, so unless you exclude it, the grep reads as every app carrying the banner with the same source. I made exactly that misjudgement once.
Self-check
- Does your coverage test only check A → B? Have you counted B → A?
- If you have "pick the first match" logic, do you verify that later entries are reachable at all?
- Have you reconciled the list in your config against the actual call sites?
The honest part
I wrote that test. I added it specifically so this class of bug would not recur, and the direction I chose covered half the problem.
Same family as tests passing while the button did nothing. Green means "what I checked is true," not "the feature works."
Take one coverage test of yours and write the same sentence backwards. Checking whether it still holds takes five minutes.