Verdict day was 10-08
On August 29 I pre-registered a Shorts content-type A/B test. Type B is a first-person reel showing a real measured number (territory area) from one of my apps. The gate updates once every evening, and I haven't changed a single rule since registration. Verdict day was 10-08, and the test needed 27 B samples.
On 10-05 I ran a pre-verdict check with two goals: confirm that 27 samples would actually be in by 10-08, and find out ahead of time which way the verdict would go. During the check a different problem turned up. Of my three language channels, the English one had been effectively dark since September 28.
While you wait for an experiment's verdict, when did you last check that the path your samples arrive through is still working?
27 arrives, and the result is already locked
The maturity rule is 'post date + 6 < today'. Posts through 09-26 give 21 unique B samples. The 09-29 posts mature on 10-06 for 24, and the 10-01 posts mature on 10-08 for 27. The 09-29 post (8,462,970 m²) and the 10-01 post (8,660,870 m²) show different areas from the earlier posts, so the repeat filter doesn't drop them.
I also checked Analytics lag. I re-queried three posts on 10-05 that had just matured on 10-04. Only one had changed, from 149 to 151 (+1.3%). Values are effectively final after a day, so the risk of undercounting on the boundary day was negligible.
On that day's data (A 59, B 21), the point estimate was 3.62x with a CI of [1.27, 2.48] on a log2 scale. I then added the current cumulative views of the six B posts that hadn't matured yet. These can only be at or below their 7-day values, so they're a lower bound (ko 353/120, en 2/0, ja 255/456). Result: PASS, 2.59x, CI lower bound 0.45 > 0. View counts never go down, so the 10-08 verdict couldn't fall below PASS. CONTINUE would require all six posts to be at zero, and the measured data already ruled that out.
The result was settled three days before verdict day. The real problem was the 'en 2/0' inside those numbers.
Only the English channel was flat on the floor
On 09-26 a B post got 447 views, and on 09-27 an A post got 155. After that, English per-post views went 5, 2, 0, 0, 0, 0, 1. It didn't matter whether a post was A or B, and the ko and ja channels were normal over the same days.
The daily traffic sources made it clearer. English Shorts-feed traffic was exactly zero from 09-28 through 10-02, the last day with data. That's five straight days, the longest run since August 1 (the previous longest was three days, 09-20 to 22). Old videos dropped out of the feed too, not just new ones: the whole channel library was gone. That points to suppression at the channel level, not per video. Search traffic fell along with it, from 40 a day to 8.
The B video posted on 10-03 got 0 views in English and 465 in Japanese. Same video. Studio's Reach tab said only 'not enough traffic data', which means it had no impressions at all.
Hypotheses I crossed off
- Upload settings changed: all 12 English uploads from 09-23 to 10-04 had the same category, language, tags, made-for-kids flag, license, embed setting and upload time (10:31Z). No commit from 09-24 to 28 touched the upload path.
- Policy action: channel status was normal (public, not made for kids, long uploads allowed). Studio's notices column showed '—' for every video, and there were no warnings in notifications. The videos also played fine outside YouTube.
- Cross-channel duplicate detection: I upload in a fixed ko→en→ja order every day, and ja, which goes up last, is fine. If duplication were the cause, ja would have been hit first.
- The 81 posts on another social platform that day: those link to the App Store and have nothing to do with YouTube.
One hypothesis is left, and I can't verify it through the API or Studio. Between 09-17 and 24 I reposted the same polygon four times, and the English channel may have been classified as repetitive content after that. The ko channel posted the same four, had zero feed traffic for three days (09-22 to 24), and then recovered. Only en failed to recover.
The verdict stands, but the next step is blocked
The problem hit both arms equally, so it doesn't change the verdict's direction. It does mean English samples sit at the floor in both arms. My pre-registration says 'expand to all apps only after PASS'. With PASS locked in, the planned next step was to take the B format to other apps that produce real numbers (a score, a streak, a recovery capacity).
What would you do at this point, with PASS locked: start the expansion right away, or look at the dark channel first?
I held off on expanding, for two reasons. First, if the English channel stays at zero, a third of the output goes nowhere. Second, even 3.6x only takes a post from roughly 70 views to 250. So before starting, I'll work out how many app installs that actually means.
There was nothing to fix in code for the English channel. Instead I set one threshold: if English feed traffic is still zero through 10-12, I'll consider pausing English posting. Piling up zero-view videos may only strengthen the repetition signal.
Three checks before your verdict
- Have you checked that the conclusion is the same with the remaining samples entered as zero and as their current cumulative values?
- When you split by channel or language instead of arm totals, is there a stretch where both arms sit at the floor together?
- When feed traffic hits zero, have you checked whether it's only new videos or the whole channel, old videos included?
The honest part
I still don't know why the English channel went dark. Repetitive-content classification is just the hypothesis left standing, and I couldn't confirm it through the API or Studio. The verdict is locked at PASS, but the win belongs to one app's territory reels, not to the 'first-person real number' format in general. Because of a pre-registration constraint, the B stimulus (area) kept shrinking, from 9.99 to 8.28 km². That means the effect may be underestimated, but I don't know by how much.