Should You Block AI Crawlers? A Merchant Decision Guide

Ask a room of media executives whether to block AI crawlers and you'll get a fast yes. Ask us — we work with merchants doing $100K to $10M — and you'll get a fast no, with two narrow exceptions covered below. The robots.txt advice circulating right now was written by publishers, for publishers, whose business AI answers genuinely threaten. A store's economics are different in one way that flips the entire decision: you don't sell the content on your pages. You sell the thing the content describes.

The publisher-vs-merchant asymmetry

Start with why publishers block, because their logic is sound — for them. A publisher's product is the page itself. When an assistant reads the article and answers the reader's question directly, the reader never visits, never sees an ad, never subscribes. Extraction substitutes for the visit, so blocking GPTBot is a rational negotiating position: it protects the asset and creates leverage for licensing deals.

A merchant's page is not the product. It's a brochure for the product. When ChatGPT reads your product page and tells a shopper "this jacket is waterproof, runs true to size, and ships free over $75" — nothing was stolen. That's your pitch, delivered by the most trusted advisor the shopper has. The assistant can't fulfill the order; the transaction still routes back to you. Extraction, for a merchant, is distribution.

AI crawler

A bot operated by an AI company that fetches web pages to feed model training (GPTBot, CCBot), a live answer index (OAI-SearchBot, PerplexityBot), or both. Distinct from user-triggered fetchers like ChatGPT-User, which retrieve a specific page on demand when an assistant needs it mid-conversation.

Every blocking decision should be run through that filter. If the page's value is captured by reading it, protect it. If the page's value is captured when someone buys what it describes, put it in front of every reader you can find — human or machine.

Who's actually knocking: the seven crawlers that matter

The user-agent zoo looks intimidating, but for commerce it reduces to seven names and one distinction: crawlers that feed training data (what future models know) versus crawlers that feed live answer indexes (what assistants can look up today). You want to be in both.

Most AI companies actually run a pair: a bulk crawler for ingestion, plus a user-triggered fetcher — ChatGPT-User, Claude-User, Perplexity-User — that retrieves a page live when a shopper asks about it in conversation. Blocking the fetchers is the worst self-inflicted wound on this list: it means that at the exact moment a shopper asks an assistant about your product, the assistant can't read your page and answers from stale or third-party data instead.

Should you block GPTBot?

No — for almost every merchant, keep GPTBot open. Here's the reasoning, not just the verdict. GPTBot feeds training data, which means it shapes what future ChatGPT models know about your category before any live search happens. Brands present in the corpus become part of the model's baseline understanding — the names that come to mind, so to speak, when a shopper asks for "a durable diaper bag that doesn't look like one." Brands absent from the corpus start every conversation as strangers.

The honest caveats: training pipelines are opaque, there's a lag between crawl and model release, and presence in a corpus guarantees nothing. But weigh the sides. The cost of allowing GPTBot on a store is approximately zero — some bandwidth, prices that were already public. The cost of blocking is self-removal from the knowledge base of every future model that hundreds of millions of shoppers consult. That's a bad trade at any traffic level.

The two legitimate exceptions:

• Contractual constraints. You license or syndicate exclusive editorial content with obligations that restrict redistribution. Block on those paths specifically.
• Proprietary content businesses. You sell paid courses, patterns, or research alongside physical products. Disallow the paid-content paths — never the whole store.

Does blocking AI crawlers hurt sales?

The honest answer: not visibly, not this quarter — and that's exactly what makes the mistake cheap to make and expensive to keep. Assistant-referred revenue is still a small share for most mid-market stores, so flipping a block on today produces no dent in this month's chart. There's no alarm. Nothing appears to break.

What breaks is invisible. Off-site assistants — ChatGPT, Claude, Gemini, Perplexity, Google AI Overviews — recommend from what they can read, and they're forming their durable picture of every category right now. When a shopper asks for "the best option under $200" and you're blocked, you're simply not in the consideration set. No impression is logged. No report shows the recommendation you didn't appear in. We covered how those recommendation decisions actually get made in how AI assistants pick products to recommend — every mechanism in that piece starts with the assistant being able to read you.

One scoping note: this decision governs off-site discovery. Shopify's own Brand Agents — shipped in the Spring '26 edition, currently Plus only — read your catalog directly and don't care about your robots.txt. The crawler decision is about everyone else: the assistants your next customer is already talking to.

The welcome-list robots.txt

Welcome-list robots.txt

A robots.txt posture that explicitly allows AI crawlers across discovery content — product pages, collections, guides, policies — while disallowing transactional and private paths. The deliberate opposite of the publisher default-deny.

Here's the copy-paste block. Grouped user-agent lines share one rule set, which keeps it readable and easy to maintain:

• # --- AI crawler welcome list ---
• User-agent: GPTBot
• User-agent: OAI-SearchBot
• User-agent: ChatGPT-User
• User-agent: ClaudeBot
• User-agent: Claude-User
• User-agent: PerplexityBot
• User-agent: Perplexity-User
• User-agent: Google-Extended
• User-agent: Amazonbot
• User-agent: CCBot
• Allow: /
• Disallow: /cart
• Disallow: /checkout
• Disallow: /account

Remember what robots.txt does and doesn't do: it tells crawlers what they may read; it does nothing to help them understand what they read. Pair the welcome list with an llms.txt file that points assistants at your highest-value pages, and with structured data so the pages they fetch are machine-legible. Access, orientation, comprehension — three layers, and this article is only the first.

What to protect anyway

Welcoming AI crawlers is not the same as running an open buffet. Four things still deserve a fence:

• Transactional paths. Cart, checkout, and account pages have zero discovery value, leak session noise, and don't belong in any index or corpus. Disallow them for everyone.
• Faceted and search URLs. Internal search results and filter-parameter pages are an infinite URL space. Disallow them as you already do for Googlebot, or crawlers burn their visit on duplicates instead of products.
• Server load. The major AI crawlers are generally well-behaved but can be bursty. If one gets aggressive, rate-limit it at the CDN — don't block it outright. You cap the cost and keep the discovery.
• Your expectations. Allowing crawlers means your prices and specs are readable by anyone, competitors included. But be honest about the counterfactual: scrapers that want your pricing use headless browsers that ignore robots.txt entirely. Blocking compliant bots removes only the well-behaved readers — the ones that send customers back.

CCBot deserves one extra sentence. Common Crawl is an open corpus, so allowing it means anyone — model trainers, researchers, competitors — can pull your pages from the archive. Our call is still allow, because that single corpus feeds more training pipelines than any one company's crawler, and reach is the whole point. But it's the one line on the list where a reasonable merchant might land differently.

Roll it out and verify

The whole change is an hour of work plus a few weeks of watching. Do it in this order:

1. 1

   Audit the current state

   Load your live /robots.txt and read it. Then — more important — check your CDN and WAF bot-protection settings. Cloudflare and similar services ship toggles that block AI crawlers at the network layer, where your robots.txt never gets a vote. A welcome list behind a WAF block is a welcome mat behind a locked door.

2. 2

   Ship the welcome list

   Add the block above (robots.txt.liquid on Shopify), and remove any AI-bot Disallow rules you didn't consciously choose. Keep the transactional Disallows.

3. 3

   Confirm the crawlers arrive

   Over the next two to four weeks, sample server or CDN logs for the user-agents on the list. If a bot claiming to be GPTBot looks suspicious, verify against the operator's published IP ranges — impersonation is common.

4. 4

   Test the outcome, not the config

   Config says what should happen; only the assistants say what does. Run the AI Citation Check, ask ChatGPT and Perplexity real category questions, and see whether you appear. For the full methodology, use our AI citation audit guide.

Merchants who block AI crawlers are optimizing for a business model they don't have. Treat these bots as your newest distribution channel — because that's what they are — and spend your defensive energy on the paths that actually need it. If you want the whole channel handled end to end — access, catalog data, structured markup, measurement — that's what our Commerce GEO service does.