I swept the self-learning loops behind my social and YouTube bots from first principles. Nothing had crashed. All green, exit code 0. But four "runs-fine-yet-learns-nothing" defects surfaced. I fixed three. The fourth — the most suspicious one — I traced through the code and then deliberately left alone. Not fixing it was the best thing this audit produced.
One question first. When a metric sits stuck at the same value for days, do you fix it first, or do you read why it was built that way first? I'm the reach-for-the-keyboard type. That day I didn't, and that was right.
The suspicious signal
Scanning a performance-feedback log, one line snagged me. A format's score on the Japanese channel: ddi = 1159. Fine on its own. The problem was that the number came out identical on 08-04 and 08-05, and barely moved to 1711 on 08-06. Every other format sat between 20 and 230 — this one was 6–8× higher.
An incremental signal should change daily. The same number three days running smelled like a cumulative value (a stock) being re-read every day. My diagnosis was already forming: "This scores by lifetime views. So a video that went viral once dominates the signal forever, and a freshly shipped format starts at zero views and can never win. The classic rich-get-richer bug." I was already prepping a --dry run to switch to delta (day-over-day) scoring.
I opened the code, and a comment stopped me
Trusting neither the label nor my own diagnosis, I opened the function that builds the score. Right at the top:
START = "2020-01-01" # aggregate over lifetime
# (so a recent-but-unposted winner doesn't get demoted
# out of a rolling window)That was past-me having already wrestled with this exact tradeoff. The cumulative aggregation wasn't a mistake — it was a choice, and a defensible one. Score on a rolling N-day window and a proven evergreen winner gets demoted simply because it wasn't posted recently. Then you stop making the thing that works. Lifetime aggregation existed to prevent precisely that.
Pause here. What would you do? Switch to flow (delta) for a fresher signal — while reintroducing the evergreen-demotion bug? Or leave the stock and accept a slow-moving number?
Why not fixing was correct
Switching to flow wouldn't kill a bug — it would trade one bias for another, and specifically for the bias past-me had explicitly rejected. Neither scheme is clean.
- Stock (cumulative): proven winners survive, but new formats start disadvantaged and the number moves slowly.
- Flow (delta): new formats compete fairly, but an evergreen winner you haven't posted lately reads as zero and gets wrongly demoted.
And crucially, ddi=1159 being identical for three days wasn't a stalled loop. The channel was stable, so its lifetime view count was stable. A stable cumulative value on top of a stable reality is not a failure — it's the expected result. I nearly mistook correct behavior for a bug.
On top of that, this score doesn't pick what to publish by argmax. A sample threshold (MIN_APP_VIEWS) strips noise, rotation appends an exploration tail, and the final pick is a reasoning committee, not a max(). The lock-in was already softened in several layers. There was nothing to fix.
And the twist on the other three
Honestly: the other three fixes were real and worth doing — removing a dead learning file, adding a sample guard, flooring the weight of conversion-oriented content. But once the audit was done, all three were re-applications of posts I'd already written. I found an open loop with no consumer in another bot, put a sample threshold on yet another, and ran into an unmeasured funnel again. Same lessons, different bots.
Classes of bugs recur as you move between bots; fixing one site leaves the sibling callers broken. That's good hygiene, but it isn't a new story. The only genuinely new thing this audit produced was the one I didn't fix.
A 3-line self-check
Before you "fix" a suspicious metric, check exactly three things.
- Did you read the comment or commit message explaining why it's built that way? (Past-you may have already reasoned about it.)
- Does your fix eliminate a bug, or does it swap one bias for another?
- Does that flat number reflect a stalled system or a stable reality? (Both look motionless.)
The honest part
This isn't "never touch it." If the channel dynamics shift, there's a day when flow becomes the right call — which is why I left the knob in place. The point is different: a first-principles audit exists to find the real bottleneck, not to run up a diff count, and this time the tool pointed at "don't break something that works."
The opposite case came out the same day: a metric that looked fine but was wrong did have to be fixed. Here, not fixing was right; there, fixing was right. A matched pair on when to trust a metric.
Sometimes the best output an audit can give you is a single diff you don't write. Picture one metric in your own pipeline that hasn't moved in days. Before you fix it, will you git blame the commit that made it that way first?