Blocking AI crawlers is costing stores money: a sane robots.txt policy for 2026

Somewhere in the last year, a well-meaning developer or a default theme setting quietly added a handful of AI crawler names to your robots.txt file, disallowed. It felt like the responsible thing to do: keep the bots out, protect the content, avoid whatever GPTBot was going to do with your product descriptions. The unintended consequence is that when a shopper asks an AI assistant to find them a pair of waterproof hiking boots under $150, your store cannot be recommended, because the crawler that would have read your catalog was never allowed in.

Why this matters more in 2026 than it did in 2023

When AI crawlers first showed up in server logs, the natural reaction was suspicion. They didn't send traffic the way Googlebot did, they consumed bandwidth, and nobody was quite sure what the trained models would do with scraped text. Blocking them felt free: no visible downside, one less thing to worry about.

That calculus has changed. Shopping is increasingly mediated by AI assistants that browse, compare, and in some cases complete purchases on a shopper's behalf. If a crawler can't read your product pages, your store is invisible to that entire channel, no matter how good your prices or SEO are. This isn't a future hypothetical. It's already happening for stores whose robots.txt quietly excludes the crawlers that feed these systems, as we cover in every Shopify store is now readable by AI shopping agents and ranking first no longer means being recommended.

The fix isn't blanket permissiveness either. Some crawlers exist purely to scrape content for resale or to feed models with no benefit back to you or your customers. A sane policy distinguishes between the two categories instead of treating 'AI crawler' as one undifferentiated threat.

Which crawlers actually matter for shopping visibility

A few names come up repeatedly in the context of AI-driven shopping and answer engines. This isn't exhaustive, and bot names and behaviors change, but as of 2026 these are worth knowing:

GPTBot (OpenAI) crawls content that can inform ChatGPT's responses and shopping features. OAI-SearchBot is a separate OpenAI crawler specifically tied to search and browsing features inside ChatGPT. Google-Extended controls whether your content can be used for Google's AI features like AI Overviews, separate from standard Googlebot indexing. PerplexityBot powers Perplexity's answer engine, which increasingly surfaces product comparisons and links. ClaudeBot (Anthropic) crawls for Claude's training and, increasingly, its browsing capabilities.

Each of these publishes documentation on its behavior and how to control it via robots.txt, and it's worth checking the current documentation directly (search each company's developer or crawler documentation pages) rather than relying on a list that will age. The pattern that matters is this: crawlers tied to live, user-facing AI shopping or search features are the ones you want reading your product pages. Crawlers whose only stated purpose is bulk training data collection with no consumer-facing retrieval are lower priority, and blocking them costs you less.

What a sane robots.txt policy looks like

Rather than a single universal file (bot behavior changes too often for that to age well), think in terms of three tiers.

Allow, deliberately: crawlers tied to shopping and answer-engine features that could send you customers or recommendations. This typically includes crawlers explicitly documented as powering search, browsing, or shopping features, since those are the ones standing between your product page and a shopper's question.

Allow with limits: general-purpose AI crawlers used mainly for model training. There's a reasonable argument for allowing these too, since being part of a model's training data may influence how it describes and recommends products later, even outside live browsing. If you're cautious about bandwidth or content reuse, this is the tier where a disallow is defensible rather than automatically costly.

Block: crawlers with no disclosed purpose, aggressive scrapers that ignore crawl-delay, or bots explicitly known for content theft rather than any retrieval or training function tied to a named AI product. This is the traditional use case robots.txt was built for, and it still applies.

The practical difference from your current file is probably smaller than it feels. Most stores don't need to invite every bot with 'AI' in its name. They need to stop reflexively blocking the ones that matter for shopping visibility while keeping the block list for bots that offer nothing in return.

Robots.txt is necessary but not sufficient

Getting the allow list right solves the access problem, not the comprehension problem. A crawler that's allowed in still needs clean, current product data to work with: accurate prices, real stock status, complete attributes, structured data that says what the product actually is. This is the same groundwork covered in AI shopping agents are placing real orders, and it matters more than the robots.txt line item itself. Letting a crawler in to read stale or broken data doesn't help you; it just means an AI assistant reads wrong information faster.

It's also worth remembering that robots.txt is a voluntary protocol. Well-behaved crawlers from major AI companies respect it. Others don't, and no file will stop them. So treat robots.txt as one layer of a policy, not the whole policy. Pair it with feed hygiene, correct meta robots tags on individual pages where needed, and periodic checks that your sitemap is actually accurate, since a crawler that's allowed in but can't find your pages via a broken sitemap ends up in the same dead end. How to tell if your store's sitemap is quietly broken covers that failure mode in detail.

Finally, revisit the file periodically rather than setting it once. New crawlers appear, existing ones change their documented purpose, and a policy that made sense in early 2025 may be quietly excluding you from a channel that didn't exist yet when you wrote it.

The bottom line

A robots.txt file that was defensively locked down in 2023 is quietly excluding your store from a growing share of how people shop in 2026. The fix isn't to open every door; it's to know which crawlers connect to real shopping and answer-engine features, allow those deliberately, and keep the block list for scrapers that give you nothing back. Review the file, check it against current crawler documentation, and treat it as part of your ongoing SEO hygiene rather than a settings toggle you flip once and forget.