Data that never piled up
My Shorts pipeline gets a performance score for each video and uses it as a reward signal. The original design was simple: look at which videos did well for each app, and feed that into the app's next video.
The problem showed up the moment I split that signal by app. With several apps, each one gets only a few videos, and even fewer of those have analytics attached. A per-app table took a long time to fill up enough to support any decision. It wasn't that the signal was missing. I had cut it into as many slices as I had apps, and no slice ever got big enough to mean anything.
Is your system learning in units so fine that each one only ever holds a handful of data points?
I never questioned the per-app split because it felt intuitive. Each app has different users and different material, so learning them separately seemed right. But if data arrives more slowly than your learning unit needs, your conclusions come late whether the unit is right or wrong.
What's left when you drop the app
So I changed the question. Instead of asking what works for this app, I asked what works regardless of app. That way every video counts.
That needed traits that have nothing to do with the app. I use two right now:
- Hook length bucket
- Whether the hook is a single line or multiple lines
Every video has both traits, whatever the app. If you group the same reward signal by these traits instead of by app, every app's videos land in the same table. The per-app tables can be nearly empty while the trait tables fill up much faster.
What would you do? Wait for per-app data to fill in, or stop telling apps apart and give up some precision?
What I changed
I added a new module, dna_perf.py. It does three things:
- Groups the existing reward signal by those two traits, across all apps.
- Writes the result to
hook/dna_hints.json. - Injects those hints into the prompt of the pre-publish review step.
The part I cared most about is the condition that turns it on. Until 8 videos have analytics data, it does nothing. Once there are 8, it turns on by itself. I don't have to flip anything. My call was that no hint at all beats a hint built on almost no data leaking into a prompt.
The change was 119 lines for the new module, 46 lines of tests, and a few lines wiring it into the existing scheduler and review code. It wasn't a big job. What took time wasn't the code. It was doubting the idea that everything had to be split by app.
Self-check
- Have you actually counted how many data points each learning unit (app, category, user segment) holds?
- Does your data have shared traits that still make sense once you drop that unit?
- Does your code have a threshold that keeps learned output from flowing downstream while the sample is still tiny?
The honest part
As of writing, I don't know whether this pooling actually produced better videos. The commit records the intent of getting a usable signal sooner. It isn't a measurement of results. The threshold of 8 wasn't derived statistically either. It's closer to a guess that it beats nothing. Pooling across apps assumes hook length has a similar effect in every app, and I haven't checked that assumption yet. I also don't know how much about a video two traits can really explain. I didn't solve the small-data problem. I just made the system waste a little less of the small data it has.