The Bug I Fixed Lived Another Eight Days — My Gate Only Caught Zero

Two unattended paper bots submitted a limit order every day at 04:50. launchd reported exit 0, the logs held no errors, the dashboard was green.

I opened the ledger. In eight days they had bought nothing.

When your bot says "healthy," have you counted what it produced? I had confirmed it ran. I never checked whether it bought.

Five sessions, one price

08-07  73.83  FILLED
08-08  73.23  CANCELED
08-11  73.23  CANCELED
08-13  73.23  CANCELED
08-14  73.23  CANCELED
08-15  73.23  CANCELED     <- real close 76.79

The sibling bot on the same engine froze at 32.04 the same way. Two 3x leveraged ETFs printing the same price to the cent for five sessions. No market does that.

With the limit below the market, a buy LOC cannot fill. Submit daily, expire daily, and $550 of unspent budget piles up.

Here's the part that stings

This was not a new bug. I fixed the same symptom five days earlier.

The guard I added read the quote timestamp from the response, returned zero if it was older than thirty minutes, and skipped the session entirely. I even left a comment: if you can't verify it, don't send it.

Eight days later I grepped the log.

$ grep -c "quote fetch failed" logs/paperbot.tqqq.err.log
0

It never fired. Not once.

Here's the fork — what would you do?

You added a gate and it didn't catch anything. Two paths:

A. Suspect the gate isn't deployed. Maybe the process is running old code. B. Suspect the gate runs fine and the condition never matches.

I checked A first. Process started Aug 12 at 10:07; the file was modified the same day at 09:32. It was running the new code.

So it was B, and B is the far more uncomfortable answer. The gate only rejects a quote of 0. What actually arrived wasn't zero — it was an old value wearing a healthy response. Every field present, the shape correct, the number plausible. There was nothing for the gate to object to.

The failure didn't arrive as an exception. It arrived pretending to be the answer.

I should have checked movement, not validity

What I validated was "is this value well-formed." What I needed to validate was "is this value alive."

Two changes.

The first is instrumentation. Every quote fetch now logs the price, the quote timestamp, and the age — and every failure branch logs too. Before, the age check ran but left no record of what it saw, so afterwards I couldn't separate "the gate missed it" from "there was nothing to catch." That's why eight days of logs held not one clue.

The second is the verdict. If the price matches the previous session to the cent, skip submission.

prev = self.ledger.last_buy_limit(date.isoformat())
if prev is not None and abs(ref - prev) < 0.005:
    log.error("frozen quote suspected: same as previous session %.2f -> skip", ref)
    return None

A repeated value counts as a failure, because the odds of a leveraged ETF closing identically two sessions running are effectively zero. I added a test asserting that a one-cent move still passes — set the threshold too wide and you block real prices, which is a worse bug than the one you're fixing.

Three things to check

Does your validation catch "no value," or also "unchanged value"? Frozen caches, dead feeds and stale snapshots all arrive shaped like a healthy response.
Do you log why something passed? If you only record failures, the failure that isn't caught leaves no trace at all.
Do you monitor runs of zero output? Execution count was 1 every day. Fills were 0 for eight days. Look only at the first number and it's flawless.

The honest part

This is a paper bot, so no money was lost. What was lost is eight days of observation — the window meant to run a full cycle before going live. What the ledger records for that window is "ordered every day, bought nothing."

And I still don't know the root cause. I found it on a Sunday with the market closed, and hitting the quote source from a fresh process returns the correct value (76.79). Only the long-lived process gets the old one, and I couldn't observe why. So rather than guess at a fix, I instrumented it so the next session decides. I'll write the follow-up once Monday's log lands.

One thing is settled. I already knew not to substitute empty data for failure — but this time the failure didn't even arrive as empty data. It arrived as good data. You cannot tell that apart by inspecting the value. You have to put it on a time axis and watch whether it moves.

Print yesterday's external value next to today's. If they match, does your code have any way to tell "the market was quiet" from "the feed is dead"?