Sitemap hygiene: why thousands of dead product URLs are quietly wasting your crawl budget

A store selling 400 active products with a sitemap listing 6,000 URLs is not being thorough. It's asking Google to spend its limited attention on 5,600 pages that no longer sell anything. Discontinued items, expired seasonal SKUs, and out-of-stock variants pile up in sitemaps for years because nobody thinks to remove them. Google keeps crawling them anyway, because the sitemap keeps telling it to. Meanwhile the new arrivals and bestsellers you actually want indexed wait longer for their turn.

What crawl budget actually means for a mid-sized store

Crawl budget is the rough number of pages Googlebot is willing to fetch from your site in a given period, based on your site's perceived value, server response times, and crawl demand. For a small store, this sounds academic. It isn't.

Google's own documentation on crawl budget (developers.google.com) notes that crawl prioritization mostly matters for sites with large numbers of URLs relative to their authority: exactly the situation an ecommerce catalog with years of accumulated SKUs finds itself in. If your sitemap lists thousands of URLs that return 404s, redirect to nowhere useful, or serve pages Google has already decided are low value, you're spending crawl budget confirming that garbage is still garbage instead of discovering that your new product line exists.

The effect compounds. Every crawl cycle spent re-fetching a dead product page is a crawl cycle not spent noticing that you updated the price, restocked an item, or published a new collection page. For stores that update inventory frequently, that lag matters.

How dead URLs accumulate without anyone noticing

Nobody sets out to build a sitemap full of ghosts. It happens through ordinary store operations. A seasonal collection launches and ends, but the individual product pages stay live and indexed for another two years because nobody assigned removal as a task. A vendor gets discontinued and forty SKUs go permanently out of stock, but the platform's default sitemap generator keeps listing them because it lists everything by default, not just what's saleable.

Shopify, WooCommerce, and BigCommerce all auto-generate sitemaps from your product catalog, which is convenient until it isn't. None of them ask "is this product still worth Google's time?" They ask "does this URL exist?" Those are different questions, and the gap between them is where sitemap bloat lives.

This is a close cousin of the problem described in soft 404s: pages that technically load but no longer represent anything useful, quietly confusing both crawlers and customers. Stale sitemap entries are often the delivery mechanism that keeps sending Google back to those pages.

A practical audit process

You don't need special tools to start, though a monitoring setup makes ongoing maintenance easier. The audit itself is straightforward:

First, pull your current sitemap.xml (or the set of sitemap files if you have an index) and extract the full list of URLs. Most platforms expose this at yourdomain.com/sitemap.xml.

Second, cross-reference that list against your actual product catalog export. Anything in the sitemap that isn't an active, purchasable product is a candidate for removal. Pay particular attention to variant URLs for out-of-stock items, since these accumulate fastest.

Third, check each candidate's current HTTP status. A discontinued product might return a 200 (still live but unsellable), a 404, or a redirect. Each needs different handling, covered below.

Fourth, check Google Search Coverage or Index reports for how many of these stale URLs are actually indexed. This tells you the scale of cleanup needed and which pages Google still considers worth ranking, which matters for deciding whether to redirect or retire them outright.

Fifth, remove the confirmed-dead URLs from the sitemap and resubmit it in Search Console. This doesn't force deindexing on its own, but it stops actively feeding crawl demand toward those pages.

What to do with the discontinued products themselves

Removing a URL from your sitemap doesn't remove the page. If the product page still returns 200 with a "sold out" banner, Google may keep crawling and indexing it anyway based on internal links, old backlinks, or its own crawl history. Sitemap cleanup is necessary but not sufficient.

For products genuinely gone for good, a 410 (Gone) or a well-targeted 301 redirect to a relevant living category or replacement product is more useful than leaving a soft-dead page in place. Google's guidance on removing content (developers.google.com) treats 410 as a clearer, faster signal than 404 for permanently retired pages.

For products temporarily out of stock but expected back, keep the page live, keep it out of the "definitely dead" bucket, but make sure your back-in-stock notification system actually works, since the whole point of keeping the page indexed is to catch demand when it returns.

Watch for the trap covered in canonical tag mistakes: redirecting a discontinued product to an unrelated page, or canonicalizing it incorrectly, can do more damage than just leaving it as a clean 410.

Building sitemap maintenance into your regular routine

One-time cleanups drift back into bloat within a year if nobody owns the process going forward. Treat sitemap maintenance as part of your product lifecycle, not a separate SEO project.

When a product is discontinued, the same workflow that updates inventory status should trigger sitemap regeneration and, where relevant, a 410 or redirect. When a seasonal collection ends, schedule its removal at the same time you schedule its launch. Quarterly, do a lighter version of the audit above to catch anything the automated workflow missed, particularly variant-level URLs that platforms sometimes handle inconsistently.

If your catalog changes often, and most do, this is exactly the kind of recurring operational task that benefits from automated checks rather than a calendar reminder someone eventually ignores. Cassian™'s sitemap checks flag when sitemap size, structure, or content drifts unexpectedly, which is a useful backstop even if the underlying cleanup work still needs a human decision about what to keep.

Keep the map current

A clean sitemap is a small, unglamorous piece of ecommerce SEO that pays off through compounding attention: Google spends its limited crawl budget on pages that can actually convert, rather than re-confirming that last year's discontinued items are still gone. Build the removal step into your product lifecycle, check it quarterly, and it stays clean without becoming its own project.