Why Google Keeps Failing to Crawl Your Sitemap Behind Cloudflare (And How We Fixed It)

Google Search Console sat there, week after week, refusing to read our sitemap. The status said “Couldn’t fetch” — or sometimes “Sitemap could not be read.” Bing had no problem parsing the exact same files. We resubmitted. We bumped the lastmod dates. We even considered renaming the file. Nothing changed.

If you’ve landed on this article, you’re probably stuck in the same loop. And here’s the uncomfortable truth we had to learn the hard way: the file was almost certainly fine — the problem was the CDN sitting in front of it.

The first diagnosis pointed at the wrong file

Our first real lead came from a second-opinion audit. It reported something that sounded completely plausible:

Your sitemap-index.xml is valid, but one of the child sitemaps it references — sitemap-1.xml — is not XML at all. It’s plain text. Google is strict about this and fails the entire sitemap the moment any child file is malformed.

That explanation felt right, and it sent us down a rabbit hole of checking the generator, the split logic, and the output format. There was just one problem: when we fetched sitemap-1.xml ourselves, it was perfectly valid XML.

That contradiction was the entire puzzle, hiding in plain sight.

Same URL, two different answers — that’s the real clue

Here’s what we saw when we looked at the raw response headers with curl:

curl -s -o NUL -D - "https://www.gotokit.cc/sitemap-1.xml"

HTTP/1.1 200 OK
Content-Type: text/xml
Server: cloudflare
last-modified: Mon, 17 Aug 2026 13:59:36 GMT
cf-cache-status: HIT
Age: 334

Four lines tell the whole story:

• Server: cloudflare — every request is served through Cloudflare’s edge, not straight from our origin.
• cf-cache-status: HIT — the response came from Cloudflare’s local edge cache.
• Age: 334 — that cached copy had been sitting there for minutes; with the right Cache-Control it can sit for days.
• last-modified — the timestamp of the content on the origin, which was older than the version we had just deployed.

Cloudflare — and every CDN like it — stores copies of your files on hundreds of edge nodes around the world. A visitor in New York, a crawler in Frankfurt, and your laptop in Shanghai can all hit different nodes. If one node still holds a stale copy from an old deploy, it will happily serve that old copy to whatever asks for it — including Googlebot.

That’s exactly what happened. The “plain text” version was an old, buggy output from an earlier version of our sitemap script. The “flat, minimal” version was the framework’s default sitemap that preceded our custom one. Neither existed on the origin anymore — they were ghosts living only in stale edge caches.

The two real bugs (only one was in our code)

Once we stopped trusting a single fetch location, the picture got clear. There were two distinct problems:

1. Stale Cloudflare edge cache — the main culprit. Our origin had been serving correct XML for days, but some edge nodes were still handing out pre-fix files. Google’s fetcher, spread across many regions, kept sampling those bad nodes, which is why it reported “Couldn’t fetch” even though the file looked fine from our end.

2. Duplicate URLs in sitemap-0.xml — our own code. Reviewing the generator, we found a copy-paste bug: the loop that emits topic pages (like /ai-tools/ and /pdf-tools/) had been written twice. That’s a real defect — it bloats the sitemap and slows processing — but it wasn’t what broke Google. It was noise on top of the cache problem.

That second bug is also a good reminder of why you want your sitemap to be provably clean before you ship it. If you’re hand-assembling one, run it through a validator; if you need to build or repair one fast, Gotokit’s free XML Sitemap Generator parses your URLs, flags invalid entries, and outputs a standards-compliant sitemap-index.xml with split child files — all locally in your browser.

The fix, in order

There’s no single magic step; the order matters.

1. Fix the source bug. We removed the duplicated loop so sitemap-0.xml stopped emitting every topic page twice.
2. Regenerate the files. Fresh lastmod dates, correct XML, no duplicates.
3. Purge the entire Cloudflare cache. This is the step most people miss. Purging the sitemap URL alone isn’t enough when stale copies live on nodes you don’t know about. We used purge_everything.
4. Redeploy. Replace the live files.
5. Verify from the edge, not just the origin. After purging, the same curl -D - showed a fresh last-modified and cf-cache-status: HIT now holding the correct content.

If your site runs on Cloudflare, you can trigger a full purge from the dashboard (Caching → Configuration → Purge Everything) or via the API:

curl -X POST "https://api.cloudflare.com/client/v4/zones/{ZONE_ID}/purge_cache" \
  -H "Authorization: Bearer {API_TOKEN}" \
  -H "Content-Type: application/json" \
  --data '{"purge_everything":true}'

Only after the cache is clean does it make sense to tell Google to look again.

How to know if YOUR issue is actually CDN cache

Run this same diagnosis before you keep editing sitemap files:

Check	What to look for
curl -s -o NUL -D - https://yourdomain.com/sitemap.xml	Is Content-Type application/xml or text/xml? Is Server a CDN?
cf-cache-status / x-cache header	HIT means a cached copy was served
Age header	How long that copy has been cached
last-modified	Is it the timestamp you actually deployed?

If the response differs depending on where you test from, you’ve caught it: your origin is fine, your edge cache is stale.

Why Google still shows the error after you fix it

This is the part that drives people to resubmit three times in a row. Understand today: even with the CDN fixed, Google has its own two layers of lag.

1. Googlebot’s own cache. It does not re-fetch your sitemap the instant your CDN is purged. It comes back on its own schedule.
2. Search Console’s status panel. The “Couldn’t fetch” banner reflects a past fetch and does not update in real time.

So after purging and redeploying:

• In Google Search Console, delete the old sitemap submission and resubmit https://yourdomain.com/sitemap-index.xml.
• Use URL Inspection → Request Indexing / Live Test on the index and each child sitemap to force a fresh fetch and confirm “Page fetch = Successful.”

Then wait. Google treats a sitemap as a hint, not a command — but once the format is clean and the cache is clear, it usually flips to “Success” within days.

The short version to keep

• If Bing reads your sitemap but Google can’t, don’t just blame the files.
• A CDN can serve different content to different visitors — including different search engines.
• The classic symptom is the one we had: valid XML from your machine, “not XML” from someone else’s check.
• Fix the file, then purge everything, then re-verify from the edge, then resubmit.
• A sitemap is a discovery aid, not a ranking factor — but a broken one quietly hides pages from Google.

None of this required rewriting our routing, our robots.txt, or our index file. It required looking past the file and at the network in front of it.