I Added Verification, and the Verification Lied

In an earlier post I found an external API returning 200 while silently discarding my value. I renamed ten channels and not one title changed.

The fix looked obvious: write, then read back and compare. I put it in every write path across six bots — token refresh, channel settings, video upload, comment moderation. On mismatch, the code logs a VERIFY_FAIL token; a scanner greps for it and rides along on an hourly alert. Three synthetic test cases passed.

I should have felt done. One thing nagged: I had verified the checker, not the thing the checker is supposed to catch. All I had done was drop a fake VERIFY_FAIL line into a log and confirm the scanner read it.

A question for you: has your alerting pipeline ever fired on a real failure, or only on test signals?

Designing a live drill

I needed a genuine failure, not a synthetic one. I picked a channel with zero subscribers and set out to send values the platform would actually mangle.

First attempt: push channel keywords past the 500-character limit.

400  Request contains an invalid argument

It failed loudly. The drill failed, but I learned something — this field does not die quietly. Silent failure isn't everywhere; it lives in specific fields.

Second attempt sent two things at once: (1) a description with trailing whitespace and newlines, and (2) a title, which I already knew was read-only. Then I checked the result with the actual shipped verification function.

[drill] update response: 200 (no error)
  VERIFY_FAIL read-back mismatch: description ...
  ! title is ignored by the API (read-only) — set it in the web studio
[drill] verification result ok=False

Success. I had produced a real "200 but the value differs" event, the check caught it, and the scanner pulled it into the alert message:

[health] ✅ all jobs normal (exit0)
 ⚠️ read-back failures: 1
 verify_drill.log(1): VERIFY_FAIL read-back mismatch: description ...

That's where I expected the story to end.

Except the restore didn't restore

The drill script reverts to the original value at the end. Its log said:

[drill] restore FAILED · currently 554 chars

The original is 547. The restore code definitely wrote 547, and reading it back gave 554.

A moment later I read again: 547. I hadn't changed anything.

It was propagation lag. Read right after a write and you get the old value.

Why that's the bigger problem

Now the earlier "success" looked different. The description mismatch the drill caught may not have been the server trimming my value at all — it may have been my own write not visible yet. And when I added retries and ran the drill again, the description mismatch vanished and only the title problem remained.

So my verification was in this state:

Real silent failure (title) → caught ✅
Perfectly healthy write (description) → reported as a failure ❌

And the second is more dangerous than the first. Once alerts fire falsely, humans start ignoring them, and an ignored alert is no alert. You land exactly where you were before adding verification — plus the false comfort of "we have checks."

So let me ask. What would you do? Accept latency for alert accuracy, or accept false positives for immediacy?

I took the first. These alerts are the kind a human reads within a few hours, not something anyone reacts to in seconds.

Three things I changed

Retry the read-back. Up to three attempts, two seconds apart, before concluding failure. A first-attempt match passes immediately, so the healthy path costs nothing.

Make the alert line readable. Originally I dumped the mismatching values. A 547-character description truncated at 80 characters in the alert told me nothing. Now it reads:

VERIFY_FAIL read-back mismatch: description · stored 547 / expected 554 · first diff at 547 · stored: How large companies…

Two lengths and the position of the first difference. The diagnosis fits on one line.

Carry warnings all the way to the alert. Verification that only writes to a log isn't verification, because nobody reads logs. I unified every mismatch behind one token (VERIFY_FAIL), and a scanner greps the last 24 hours of logs and adds one line to the hourly heartbeat. The 24-hour window exists for the same reason — an old failure that rings forever also ends up ignored.

Three things to check

Has your alerting fired on a real failure? If only on test signals, you've verified the pipe, not the detection.
Do you trust a read immediately after a write? APIs backed by distributed storage may not show you what you just wrote.
Who gives up first when it cries wolf? If the alert can't outlast the human's patience, it's already dead.

The honest part

Three retries over six seconds is fitted to the lag I actually observed (one retry cleared it). A longer lag will bring false positives back. I didn't inflate the number without evidence, and the real delay will be in the logs when it happens.

Also, the lesson here isn't "add verification." In one day I added verification, the verification had a defect, and that surfaced only because I ran a real write instead of a synthetic test. A passing test means a test passed. It does not mean the code is right.

The discovery half of this story is in The API Returned 200 and Threw My Value Away, and another case of my own measurement lying to me that same day is in I Almost Blamed the Crawler for My Ad Rejection.

You have an alert running that has never fired on a real failure. Care to break something for real today?