Most of my search traffic comes from Naver, so whether its crawler is reaching my sites is a number that matters. One day I sent a plain curl at the apex domain and got back 403.
A hypothesis formed immediately. The CDN edge is blocking the crawler. That is a top-priority item — if the channel that delivers 87% of my inbound traffic is stopped at the door, no other SEO work means anything. I wrote "if it's blocked, this is priority one" into a handoff note and passed it along.
One question before we go on. When you conclude "my bot is blocked," which response did that verdict come from?
I ran more arms. They all returned the same answer
To firm up the hypothesis I added conditions.
curl -sS -o /dev/null -w '%{http_code}\n' -A 'Yeti/1.1' https://<apex>/
curl -sS -o /dev/null -w '%{http_code}\n' -A 'Googlebot/2.1' https://<apex>/
curl -sS -o /dev/null -w '%{http_code}\n' -A '<ordinary browser UA>' https://<apex>/All three returned 403. At that point most people write "confirmed." I did write it.
But look at the third line. An ordinary browser UA is also 403. If the edge were really singling out bots, a request dressed as a browser should have passed. It didn't — which means the experiment cannot distinguish who is being stopped.
So I asked the first-principles question. Is this experiment structurally capable of producing a "pass"?
It was not. CDN bot verification runs on the reverse DNS of the requesting IP, not on the UA string. A request from my laptop can never be a verified crawler, whatever I type into the header. Every arm returns the same answer. Adding arms adds zero information.
The 403 itself was also a misreading. It was not a block — it was a browser challenge page. Verified bots never see that page at all. I had read the symptom (403) and the cause (blocking) as the same thing.
The verdict came from the origin access log
Where would you look next?
- The edge dashboard's bot management screen — check the block rules.
- The search console's "crawl request" screen — see whether the crawler came.
- The origin server's access log — count status codes for that bot's UA.
Option 1 only tells you what you configured to be blocked, not what actually was. Option 2 is worse: the crawl-request screen in Search Advisor or Search Console shows submission, not what happened after submission.
I went with 3. On the origin server I filtered the access records for that window by crawler UA and counted status codes.
Yeti GET / 200
Yeti GET /sitemap.xml 200
Yeti GET /robots.txt 304Zero 403 responses. The crawler was coming in, and had never once been stopped. The edge allowlist work I had queued was a fix for a problem that never existed.
The real problem showed up after the hypothesis died
If I had stopped there, the entry would read "false alarm." But the same log answered the next question: if it isn't blocked, how often does it come?
- Naver's crawler: 3 requests/day
- Google's crawler: 18 requests/day
Six times the difference. The problem was never entry — it was crawl budget. Had the blocking hypothesis been right, the fix would have been one edge rule. The real problem is the slower, more expensive kind: raising crawl frequency through site trust and internal linking.
That number went straight into the denominator of another verdict. I have 227 new parameter pages deployed and I am waiting on indexing. At 3 crawls a day, the time it takes to walk 227 pages is the ceiling on my judgment window. The lesson from adding pages and getting fewer hits — look at yield first — becomes "look at budget first" here.
Three checks for your own setup
- Can the experiment output a "pass" at all? If every condition returns the same answer, adding conditions adds nothing. Before running it, write down what would have to be true for the opposite result to appear.
- Is the verdict based on an edge response or an origin log? The edge only reports what it gave you, never what it gave the bot. If you cannot read the log, record it as unknown rather than as a finding.
- Are you writing the status code and the cause into the same cell?
403is emitted by blocking, challenges, rate limits and misconfiguration alike. Pin a cause to a single code and you will promote a non-existent problem to priority one.
The honest part
This investigation raised traffic by exactly nothing. What it bought was two things: one unnecessary task removed from the top of the list, and a real number for the actual bottleneck.
The embarrassing part is worth recording too. I mistook "I ran three conditions" for evidential strength. I had not increased my sample — I had repeated the same bias three times. Repetition is not verification.
If you have an issue open right now that says "looks blocked," go check where that verdict came from. Is it a status code from the origin log, or a response you poked for yourself?