Google Believed the Sitemap, Not the 404

Did you know a 404 isn't always enough to get a page out of Google?

On a large website I worked with, a whole section was removed. Every URL returned 404. Intentional, clean, done.

The sitemap didn't get the memo. A technical issue in the enterprise CMS kept every deleted URL in the XML sitemap for a couple more weeks.

Impressions and clicks for that section dropped by roughly half. Then they stalled. Half of the section's visibility was still in Google, still showing up, still getting clicked. Straight into a 404.

Then the sitemap got fixed. Almost immediately, impressions and clicks for the section dropped to near zero.

Same 404s. The only thing that changed was the sitemap.

A 404 looks like an accident. A sitemap looks like intent.

Look at it from Google's side.

Google has to recrawl a URL to see the 404. On a large site, that doesn't happen for every URL on the same day.

And 404s happen by accident all the time. A deploy goes wrong. A route breaks. A CDN serves an error page for an hour.

A sitemap is the opposite. It's a file somebody maintains on purpose, and it says: these URLs exist, they matter, please index them.

Now put both signals on the same URL. The page says gone. The sitemap says important. Which one looks like the mistake?

From the outside, the 404. My read: Google kept trusting the sitemap, and the URLs stayed in the index longer than they should have.

I can't see inside Google's pipeline. But nothing else changed on that site when the sitemap was fixed, and the drop followed right after.

Keeping deleted URLs in the sitemap slowed the drop

There's advice that has been around SEO for years. When you delete pages, keep them in the sitemap for a while, ideally with a fresh lastmod, so Google recrawls them sooner and processes the 404 faster.

On paper it makes sense. On this site it did the opposite.

Keeping the URLs in the sitemap didn't speed anything up. It slowed the drop down. Removing them is what finished it.

One site, not a controlled experiment. But I'll take a real before and after over a theory. When a page is gone, it leaves the sitemap in the same release. Not next sprint. Not "once Google has seen the 404."

What a half-deleted section costs you

For those couple of weeks, the section was invisible as a product and visible as an error.

People searched, saw a result, clicked, landed on a 404 and went back to Google. Probably to a competitor. That's the worst of both worlds – you've already lost the page, and you're still spending trust on it.

Googlebot gets mixed signals too. Every time the sitemap re-advertises a dead URL, it gives Google a reason to come back to it instead of a page you actually want indexed.

And your reporting gets noisy. The Page indexing report fills up with 404s you submitted yourself. Real problems get buried under self-inflicted ones.

The worst part: nobody looks at a sitemap after it's generated. So the same mistake repeats with every removal.

Keep every sitemap file under 10–15MB

Removals aren't the only place where sitemaps quietly break. Size is the other one I keep running into on enterprise sites.

Google's official limit is 50MB uncompressed or 50,000 URLs per file. On paper, anything below that is fine.

In practice, it isn't. On multiple other enterprise sites I worked with, I've had problems with sitemaps sitting just below the limit. Push a single file toward 45MB and most of the time Google simply won't load it reliably. Files get partially read or skipped, and you're left guessing which URLs Google actually saw.

My rule is much stricter. Keep every sitemap file under 10–15MB uncompressed. Split with a sitemap index and you lose nothing.

The catch is that sitemaps don't stay the same size. Every new product, every new locale adds a bit, and nobody checks until something breaks. I watch it with sitemap monitoring in VitalSentinel, my own tool, so I see a file creeping up long before Google stops loading it.

Compression doesn't save you here. Gzip a 45MB sitemap and it shrinks to a few megabytes on the wire. The limit is measured on the uncompressed file. Gzip helps transfer. It doesn't change what Google has to parse.

Hreflang is where files explode. Every URL carries one alternate line per locale:

<url>
  <loc>https://www.example.com/de-de/products/whey-protein-1kg/</loc>
  <xhtml:link rel="alternate" hreflang="de-DE" href="https://www.example.com/de-de/products/whey-protein-1kg/"/>
  <xhtml:link rel="alternate" hreflang="de-AT" href="https://www.example.com/de-at/products/whey-protein-1kg/"/>
  <xhtml:link rel="alternate" hreflang="fr-FR" href="https://www.example.com/fr-fr/products/whey-protein-1kg/"/>
  <!-- ...one line for every locale -->
</url>

Each of those lines is roughly 110 bytes. Twenty locales and you're over 2KB per URL. That's only about 4,000–5,000 URLs per 10MB file.

At twenty locales, even Google's own 50MB limit hits at around 21,000 URLs. The 50,000 URL cap never comes into play. You hit the size problem first, usually without noticing.

My preference: move hreflang to the HTML head. Google supports all three methods – HTML, HTTP headers and sitemap – equally. In the head, the annotations live on the page they describe, and the sitemap goes back to doing one job – listing URLs. Each page gets a couple of KB heavier. The sitemap shrinks by more than 90%.

Just pick one method. Hreflang in both the head and the sitemap means two sources that will eventually disagree.

Deleting a sitemap doesn't make Google forget it

Removing a sitemap in Search Console only removes it from the report. It doesn't stop Google from fetching the file. Google already knows the URL and keeps using it for a while.

Old sitemaps from a migration are the usual suspects. Nobody remembers them, but they still list URLs from a structure that no longer exists.

If a sitemap should die, make it actually die. Return 404 or 410 on the file itself. Remove it from robots.txt and from every sitemap index that references it. Then remove it in Search Console.

How I remove pages now

If the page is gone for good, it returns 410, not 404. A 404 says "not found" – it might be back tomorrow. A 410 says "gone, permanently", and Google tends to process it a bit faster. Whichever you return, it has to stay that way.

The URL leaves the sitemap in the same deploy. If the CMS regenerates sitemaps on a schedule or caches them, that job is part of the removal, not an afterthought.

Internal links go too. Navigation, related products, breadcrumbs, footer links. As long as your own site links to a URL, Googlebot keeps finding it.

Redirect only when there's a real equivalent page. Not to the homepage, not to a parent category "just in case."

Then check the output, not the process. Fetch the live sitemap after the release and confirm none of the removed URLs are in it. Then track indexation of the removed URLs until they actually drop out, not until the next audit.

On the site above, those couple of weeks were a CMS problem, not a decision. That's exactly why sitemap cleanup belongs in the removal process from day one, not in a ticket afterwards.

A 404 tells Google a page is gone. Your sitemap decides whether Google believes it.