The Ledger Held 922 Topics. I Had Published Four.

I run a bot that produces one video a week without a human picking the topic. It scrapes candidates from public records, selects one, drafts a script, and publishes if the draft passes review.

Last week it produced nothing. The log said:

[discover] 94 candidates
  skip: ...
  skip: ...
[discover] no suitable candidate this run

I read that as topic exhaustion. It looked at 94 candidates and none fit — time to add more categories.

All 94 were permanently excluded that run. None of them had been judged.

Does your pipeline write "this isn't a fit" and "I couldn't read this" into the same column?

Three numbers

I opened the ledger.

922 minus 8 minus 4 leaves 910. Roughly 900 entries were excluded without ever being judged. Every week the bot printed "no suitable candidate" and burned every candidate it had looked at that run. Same family as the time every job was green with nothing to show for it — except this time the log looked fine too.

Five lines

The bug was the order of the loop.

seen.add(s); seen.add(title)   # don't retry what we tried
ext = extract(title)
if len(ext) < 500:
    continue                   # ← already in the ledger, never screened

seen.add sits above the line that fetches the body. So the flow is:

1. Record the candidate as "covered"
2. Fetch the body
3. If the body is short, skip

Anything skipped at step 3 is already dead from step 1. It will never be drawn again — not next run, not ever.

And extract() looked like this:

def extract(title):
    try:
        ...
        return p.get("extract", "")
    except Exception:
        return ""      # a network failure returns an empty string too

It swallows the exception and returns an empty string. Which means the wiki throwing a 429 and an article genuinely being thin look identical upstream: both are len(ext) < 500. One timeout permanently excludes a perfectly good topic as "poorly documented."

I had already fixed this bug once

This is the part that stings.

There is a sibling bot. Different channel, different subject, same code lineage. In August I found the exact same bug there and fixed it. The numbers then: 305 of 309 ledger entries burned without a verdict, five videos actually made.

I fixed that bot. I did not look at this one.

The clue was sitting in plain sight. This bot's HTTP header read:

UA = {"User-Agent": "aftermath-docbot/1.0 (...)"}

That is the sibling's name. The one marker in the code saying "this file was copied from over there" was right there, and nobody checked whether the bug fixed over there still lived here.

Which gives a rule: when bots share an ancestor, fixing one means grepping the others. Copied code copies its bugs.

What would you do?

You are looking at a ledger with 900 burned entries. Which do you do first?

• Add categories — grow the candidate pool and delay exhaustion. The cause stays.
• Empty the ledger — restore everything. Reasoned rejections come back too, so you re-screen the same duds every week.
• Split exclusions by grounds — separate what was judged from what was never read, and restore only the latter.

The third is right, but it requires the ledger to carry reasons. Mine didn't. So the restore criteria had to come from outside the ledger: keep only entries backed by an artifact (published, queued, rejected folder) or a logged reason, and drop the rest. 922 → 22.

The shape of the fix

Four parts.

• A fetch failure is not an empty value. A dedicated FetchFailed exception means anything we couldn't retrieve stays out of the ledger and returns as a candidate next run.
• 429 is a signal, not a failure. Honor Retry-After, up to three attempts.
• A failed screening call doesn't burn the topic either. The judging tool dying is not a defect in the subject.
• Only reasoned rejections are excluded permanently, and the reason is written to its own file. Next time this happens, the split can be made inside the ledger.

Then three regression tests, whose names are the rule:

test_fetch_failure_does_not_burn_topic
test_merit_rejection_is_recorded_and_burned
test_screen_call_failure_does_not_burn_topic

I ran it. Candidates went from 94 to 475, and the bot picked a topic and queued it.

Three checks

Point these at your own automation right now.

1. Does your exclusion ledger have a reason column? Without one you cannot later separate "judged and rejected" from "excluded by accident."
2. Does your input-fetching function swallow exceptions and return an empty value? If so, upstream outages masquerade as downstream data-quality problems.
3. Do you stamp "already processed" before processing or after? Before means every failure in between becomes a permanent exclusion.

The honest part

This bot makes no money yet, and I can't price what burning 900 topics cost — there's no way to count how many of them would have become good videos.

One thing is certain. If I had run one more grep when I fixed the sibling in August, this post would not exist. That grep would have taken thirty seconds.