I serve policy/support pages for 50 apps as subpaths under one domain — /<appname>/index.html — deployed by rsync/scp.
One day I opened one app's directory and index, privacy, terms, support were all another app's stub. The title was a different app's name, and the size was 1–5KB. Normal pages are 27–79KB. It looked like it went up wrong in a deploy four days earlier.
The real problem: there was no way to confirm a deploy succeeded
This domain is behind Cloudflare, which serves a 5.4KB challenge page to non-browser requests. So curl on a URL gets the challenge, not the app page. You get HTTP 200, but the body is the challenge. You can't use HTTP to judge deploy success.
So for four days nobody knew. The deploy script returned success, automated URL verification was impossible to begin with, and no human had reason to open that app's page in a browser.
The only authoritative verification was comparing the md5 of the server file against local, over SSH. Meaning the only truth the deployer can reach is the filesystem.
One more thing. This URL was named in external review paperwork as the "service web page and attribution location." Had a reviewer opened it, an entirely different app would have shown up. It was the kind of incident whose severity is set not by the 4 files but by where that URL is written down.
Code
The contamination signal — size alone gives it away:
normal page: 27–79KB
contaminated: 1–5KB (another app's stub)Full sweep (50 app directories, one SSH session):
cd ~/public_html
for d in */; do d=${d%/}; [ -f "$d/index.html" ] || continue
echo "$d|$(stat -c%s $d/index.html)|$(grep -m1 -oE '<title[^>]*>[^<]*' $d/index.html | sed 's/<title[^>]*>//')"
done⚠️ The local <title> is <title data-i18n="title">, so a <title> regex won't match. Match with <title[^>]*>. This made the first sweep return all empty titles.
Recovery: back up the server copy → scp local files → verify all md5s against local. Shared assets (common CSS, i18n JS, icons) were already on the server, so no re-deploy was needed. The sweep found the other 49 apps all correct.
Generalizing — behind a CDN/WAF, the deploy-verification path disappears
Because bot protection treats the deploy pipeline as a bot too. There are roughly three options:
- Exclude only a verification path from the challenge (this site keeps
.well-known/open with a 200) - Access the server filesystem directly and compare hashes (the method chosen here)
- Render with a real browser after deploy and check (expensive)
Either way, you first have to accept that "got HTTP 200, so the deploy succeeded" is unusable.
Honestly
- The discovery was luck. It wasn't caught by a scheduled check or an alert — I noticed it while doing something else. Had the same thing happened on a different app, it would have gone longer.
- I recovered it but never determined why another app's stub landed in that directory. I suspect a path-argument mistake in the deploy script, but I couldn't reconstruct that day's command. The symptom is fixed with the cause unknown.
- The full sweep was one-off. There's still no automated scheduled verification. As it stands, if this recurs I have to discover it by luck again.
- The sweep produced one false positive. One app's title wasn't what I expected and I suspected contamination — turns out the app had genuinely been rebranded and the site was correct. Its small 8KB page was also fine, a bespoke build. I re-learned during verification that you mustn't read "different" as "wrong."
The app detail URLs this post mentions (ootssu.com/<slug>/) still return 403 to non-browser requests — that 403 is itself the subject here. App in this story: HiddenGem.
That 200 your deploy script checks at the end — did it get the file you uploaded, or a challenge the WAF answered on its behalf? They're the same green light.