Shopify robots.txt and AI Crawlers: What to Allow and What to Block

Shopify’s default robots.txt names no AI crawler, so the crawlers behind ChatGPT, Claude and Perplexity can read your products unless a custom robots.txt.liquid template turns them away. AI crawlers come in two kinds: those that fetch pages to answer people or build AI search results, and those that collect pages to train AI models. The companies give the two kinds separate names, so you can opt out of training with GPTBot, ClaudeBot and Google-Extended and still appear in AI answers. AccessComply reads your robots.txt on every scan and flags a block on the search crawlers that honour it. It never edits the file.

Two kinds of AI crawler, with separate names

OpenAI, Anthropic, Perplexity, Amazon and Google each publish the names their crawlers answer to in robots.txt. The names separate two kinds of work, and this is what each company says about its own.

Crawlers that answer people or build AI search results:

Names that cover AI training:

Three things follow. Blocking a training name does not block the same company’s answer or search crawler. robots.txt is a request, and the fetchers that act for a person may not follow it: OpenAI says its rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Amazon says Amzn-User may not follow every directive. And changes take time: OpenAI says its systems can take about 24 hours to adjust after you change robots.txt.

What Shopify’s default robots.txt allows

Shopify says its default robots.txt works for most stores. As served on Shopify’s own Dawn demo store on 25 September 2026, it has one group of rules for all crawlers, which opens product, collection, page and blog pages and closes the admin, cart, checkout and account pages, plus a group for Google’s ad crawler. It names no AI crawler, so every AI crawler follows the general group and may read your products.

The file also opens with notes addressed to AI agents, pointing them to the store’s /agents.md and to Shopify’s commerce endpoints for catalog, cart and checkout.

Where a block on AI crawlers comes from

A block appears when someone adds a custom robots.txt.liquid template to the theme, in the code editor, as a new template of type robots. Shopify calls this an unsupported customization that its support team can’t help with, and warns that incorrect use can result in loss of all traffic. A block on one crawler is two lines:

User-agent: OAI-SearchBot
Disallow: /

A template written to stop AI training can name the answer crawlers too, and then those assistants are asked to stay away from every page.

Opt out of AI training and stay in AI answers

Keep Shopify’s default rules and add groups for the training names only. Shopify’s developer guide to robots.txt recommends keeping its Liquid objects for the default rules, which Shopify updates regularly, and blocks a crawler by adding a group after them:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

OpenAI says disallowing GPTBot means a site’s content should not be used in training. Anthropic says restricting ClaudeBot excludes the site’s future materials from its training datasets. Google says Google-Extended does not affect inclusion or ranking in Google Search. None of the three stops OAI-SearchBot, Claude-SearchBot, Claude-User or PerplexityBot. Amazon’s page puts training under Amazonbot and Alexa answers under Amzn-SearchBot and Amzn-User.

What AccessComply checks in your robots.txt

On every scan, including the free scan on this site, AccessComply reads your store’s robots.txt and asks one question for each crawler of ChatGPT, Claude, Perplexity and Alexa that honours it (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Amzn-SearchBot): may it fetch a product page or the home page?

• If neither, the finding names the assistants and the exact crawler names that are blocked.
• A narrower rule, such as one that closes search results pages, is not reported.
• Training names such as GPTBot, ClaudeBot, Google-Extended and Amazonbot are never reported. Blocking them is a reasonable choice, and it keeps the assistants’ search crawlers able to read your store.
• The fetchers that act for one person, such as ChatGPT-User, are not reported either: their owners say robots.txt may not apply to them, so a block there does not do what it says.
• A robots.txt the scan could not fetch counts as not measured, never as a failure.

If the block is deliberate, leave it. Otherwise remove those groups from robots.txt.liquid, or delete the template to go back to Shopify’s default; save a copy of the file first.

Why the app never edits robots.txt

Which crawlers may read your store is a business decision, and Shopify warns that a wrong robots.txt can cost a store all of its search traffic. So AccessComply reports what the file says and leaves the change with you.

Products also reach assistants through Shopify Catalog, which Shopify builds from your product records. Shopify’s documented way to keep a product out of AI channels is the Unlisted status, which also hides it from sitemaps, search engines and your store search. Shopify product data for AI shopping covers the fields those channels read.

The AI shopping topic collects the related guides. See pricing for plans.

Sources

• OpenAI, Overview of OpenAI crawlers.
• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity documentation, Perplexity crawlers.
• Amazon Developer, Amazonbot.
• Google Search Central, Google’s common crawlers.
• Shopify Help Center, Editing robots.txt.liquid.
• Shopify developer documentation, Customize robots.txt.
• Shopify Help Center, Shopify Catalog and product discovery for agentic storefronts.
• Shopify’s default robots.txt as served on its Dawn demo store, read 25 September 2026.