Before fixing a bug I wanted to confirm RED first, so I ran just the one test I cared about.
xcodebuild test -only-testing:<target>/<suite>/<function>One question first. What do you read in test output besides "succeeded"? Do you read the execution count?
1. A filter that matches nothing is not an error
In the newer test framework (the @Test macro style), a -only-testing filter that fails to
match is not an error. It executes zero tests and prints:
✔ Test run with 0 tests in 1 suite passed
** TEST SUCCEEDED **The run meant to confirm RED finished green. So I misread it as "the guard must already be there". I only caught it because I ran it a second time.
If all you read is TEST SUCCEEDED, zero executed and everything passing are indistinguishable.
2. This is a defect in my verification procedure more than in the tool
Plainly: I'm the one who didn't read the execution count.
On the same day, on another app, I got it wrong in the opposite direction — I concluded "the tests never ran". That time it was because I didn't know there are two kinds of counter.
I don't know why a non-matching filter isn't an error. The same typo in the older framework fails.
This trap is especially bad in automation. CI and scripts read only the exit code, and the exit code can never separate zero-executed from all-passing. In the same batch, one build setting that killed 61 tests during boot and the timer that made every test pass are the sibling posts. Every app passed, then every app failed and the fallback that kept the board green are the earlier ones in the family.
3. How would you trust this output?
** TEST SUCCEEDED **- (a) Exit code 0, so it passed
- (b) 0 failures, so it passed
- (c) Execution count matches expectation, so it passed
(a) and (b) are both true of a zero-test run. They are verdicts with no denominator. Only (c) works.
4. So here are the rules I settled on
Filter at suite granularity. Don't hand it a function name.
-only-testing:<target>/<suite>Always read the count from the output. And if frameworks are mixed, there are two counters:
Executed N tests, with M failures ← older framework
Test run with N tests in K suites ← newer framework⚠️ Grep only one of them and you'll see Executed 0 tests and conclude "the tests never ran".
There really is an app here with suites written in the newer framework mixed in. It makes you wrong
in exactly the opposite direction from the first trap.
5. Two traps from the same family
- Test files not registered in the project file simply never run. A newly created UI test file isn't added automatically, so adding a class inside an already-registered file is the safer move — the framework looks for classes, not files.
- Android instrumentation tests don't accept
--tests. It goes through the runner argument:
./gradlew connectedDebugAndroidTest \
-Pandroid.testInstrumentationRunnerArguments.class=<FQCN>6. The rule: never end a RED check on a green light
Check that the execution count matches expectation, and do at least one negative control — revert the fix and confirm it fails.
Without the negative control, what you verified is "the test passes", not "the test watches that". Those are different claims.
Three things to check in your own setup
- Does your CI record the execution count? If it only reads the exit code, a zero-test run passes green.
- Have you checked what happens when you typo a
-only-testingor--testsfilter? Get it wrong on purpose once. - Did you actually see RED before fixing the bug? A RED check that ended green is not a check.
The honest part
What I got out of this isn't trap knowledge, it's one habit: the first thing I look for in test output is now a number. The word "succeeded" comes second.
When I wasn't reading counts I got it wrong in both directions — once reading zero as a pass, once reading a healthy run as never-ran. Twice in one day.
Do one thing today. Put a name that doesn't exist into your test filter and run it. Thirty seconds tells you whether your tooling errors out or finishes green.