The shorts pipeline has a self-improving loop: it turns each published video's views, completion, and engagement into a reward, then auto-adjusts the next rotation's app order, category weights, and retention hints. I thought it was working well.
Mistaking noise for signal
The gate fired on "does a score file exist?" So scores from apps with two-digit view counts went straight into the ranking. If a 12-view video happened to have a high completion rate, the loop read that app as a "winner," pushed it to the top of the order, and promoted it harder. In the Korean app_perf, ddi — with fewer than 30 views — was sitting at #1, and so was zodiac.
This isn't improvement, it's overfitting. The smaller the sample, the larger the variance, and the loop mistakes that variance for skill. It's the same disease as killing strategies by out-of-sample Sharpe — if you don't validate outside the sample, you reinforce luck.
The fix: a minimum-sample brake
I added a MIN_APP_VIEWS = 30 threshold. Apps with lifetime views below it aren't taken as ranking scores; they're demoted to the exploration tail — not thrown away, but moved to the "judgment pending, keep exposing to gather sample" slot.
The threshold lives in one pure helper, _qualified(buckets, vsum, min), shared by all three producers (order, category, retention). The gate firing was also made honest: "file exists" became "a sample-qualified score exists" (_has_scores). After applying it, ddi and zodiac dropped out of the ranking.
The honest part
The most dangerous failure of an automated feedback loop isn't stopping — it's running confidently in the wrong direction. With no sample gate, the loop learns luck, reinforces luck, and calls it data. Statistical significance isn't something a dashboard decides for you — you have to state, in code, inside the pipeline, "the minimum sample at which I trust this number." When n is small, it isn't performance. It's noise.