The Pipeline Knew Which Topic Won — And Spent Half Its Output on the Loser

An unattended pipeline builds and uploads shorts every day. It has performance feedback attached: it reorders apps by views and completion rate on its own. I thought of it as a self-improving system.

While measuring channel reach today, I found that was only half true.

Is your automation actually using the scores it computes? I had verified that the numbers were calculated. I never verified that anything consumed them.

First I compared the channels

To find out which channel could actually drive app installs, I put reach per piece side by side.

The Threads account shows 156,064 cumulative views. That looks large, but it comes from replying inside other people's threads — it is not reach on our own posts. Our own posts are seen by dozens.

That explained something else. A few days ago I posted a public offer: leave a comment and get a redeem code. Zero requests came in. Not because the offer was bad, but because only dozens of people saw it.

Then the gap inside Shorts

I sorted the last 25 uploads by views.

2,958  zodiac animal — ox / snake / rooster
2,466  zodiac animal — tiger / horse / dog
  928  blood type personality
  598  MBTI compatibility
  569  zodiac animal — rat / dragon / monkey

Median of the four zodiac-animal videos: about 1,500. Median of the other 21: about 220. Same production cost, 4–10x the reach.

So I went to shift the topic mix — and stopped. The queue is generated automatically every day. The entries I hand-wrote failed to render because they lacked fields the renderer needs. That was not the place to touch.

The system already knew

I just ran the performance feedback script.

[perf] ko scores: {ddi: 1390, bloodtype: 345, animalface: 196, zodiac: 83}
[perf] ja scores: {ddi: 2009, bloodtype: 707, zodiac: 185, animalface: 158}

Zodiac animals ranked first — 4x the runner-up in Korean, first in Japanese too. What I thought I discovered today, the pipeline had been computing every single day.

So why did MBTI keep going out?

The cause was one line

Content generation runs on two tracks: personality and compatibility. One of each, per channel, per day.

The personality track picks its order like this:

order, _ = _perf(lang)                      # read the performance scores
pref = [a for a in order if a in REGISTRY] or LANG_APP_ORDER.get(lang)

The compatibility track picks it like this:

pref = LANG_APP_ORDER.get(lang)             # only a hardcoded dict

And that dict was:

LANG_APP_ORDER = {"ja": ["ddi", "mbti", "bloodtype"]}

Japanese only. Korean and English have no key, so it resolves to None and falls back to declaration order. Declaration order starts with mbti. MBTI's 16 types make 120 pairs. At one video per day that is four months of MBTI, while the topic performing 4x better waits its turn.

I can guess why Japanese was special. Someone noticed the Japanese numbers and patched it by hand. A hand-patched spot only fixes the language in front of you.

Here's the fork — what would you do?

A. Add ko and en to the dict. Five minutes, and today's winner gets used. B. Make the compatibility track read the performance scores too, from the same source the personality track uses.

A is right today and wrong next month, because when the winner changes nobody will edit the dict. That is exactly how the current state was produced — one manual patch, two forgotten languages.

I went with B. Two lines.

order, _ = R._perf(lang)                    # same source as the personality track
pref = [a for a in order if a in MATCH] or LANG_APP_ORDER.get(lang)

After the change, the next compatibility video planned for Korean, English and Japanese was the zodiac topic in all three.

The experiment is pre-registered

I would like to write "and views went up," but I don't know yet. Instead I wrote the decision rule first.

• Don't fill the queue with the winner alone — keep a control group. All-winner makes it impossible to separate a topic effect from a timing effect.
• First call in three weeks. PASS is the zodiac median at 2x the control, FAIL is under 1.3x, and anything between is recorded as "no verdict."
• The sample is four videos. I cannot rule out YouTube's recommender simply picking those four.
• I can't see impressions or CTR (analytics scope not authorized), so topic effect and thumbnail effect cannot be separated.

I also wrote down what this experiment does not answer. More views does not imply more app installs. App traffic is currently 193 people a month. If reach rises and installs don't, the bottleneck isn't the topic — it's that Shorts viewers don't install apps.

Three things to check

1. Do the code that computes your self-improvement metric and the code that consumes it read the same function? I verified the computing side and never looked at the consuming side. With two tracks, you have to check both.
2. Is there a hardcoded per-language or per-region dict? With a single key present, everything else silently takes the default. No error, no warning.
3. If your queue is consumed in order, what is at the front right now? Put combinatorial content (a 16×16 matrix) at the front and it fills months.

The honest part

This is not a story about automation breaking. Everything worked. Videos were built, uploaded, scored. The dashboard would have been green.

One line was missing between what the system knew and what it did. That line was assigning four months of output to the losing topic.

A default once wrote my conclusion for me too. When we audit automation we usually ask "is it running." Today I learned to ask "while running, what is it choosing." Print your pipeline's performance scores once, and check whether they actually select the next artifact — or get computed and thrown away.