AI crawlers: which ones to allow

Search, user-triggered, training: each assistant sends several robots. What each one does, what blocking it costs you, and how to set robots.txt on Shopify and WooCommerce.

Updated on October 9, 2026 · by the skoup team

Four families of robots

An AI assistant does not send one robot to your store, but several. Each has a role, described by its vendor, and that role tells you what blocking it costs.

Search robots build the index the assistant looks in for its answers. Blocked, they can no longer cite your pages.
User-triggered robots read a page when a user asks a question or pastes a link. Several vendors warn that robots.txt may not apply to them.
Training robots collect content for the next models. Blocking them does not remove your pages from search answers.
Search engines (Googlebot, Bingbot, Applebot): their index also serves the assistants, such as Google's AI Overviews or Siri at Apple.

The table, vendor by vendor

This table follows each vendor's public documentation, read on October 9, 2026. The links are at the bottom of the page.

Robot	Vendor	Role	robots.txt according to the docs	Effect of blocking it
`OAI-SearchBot`	OpenAI	ChatGPT search	Respected	The site no longer appears in ChatGPT search answers
`ChatGPT-User`	OpenAI	User-triggered	"May not apply"	No effect on search, according to OpenAI
`GPTBot`	OpenAI	Training	Respected	Content excluded from training
`Claude-SearchBot`	Anthropic	Search	Respected	Content not indexed, reduced visibility according to Anthropic
`Claude-User`	Anthropic	User-triggered	Respected	Claude no longer reads your pages to answer
`ClaudeBot`	Anthropic	Training	Respected	Future content excluded from training
`PerplexityBot`	Perplexity	Search, no training	Respected	The site no longer appears in Perplexity's results
`Perplexity-User`	Perplexity	User-triggered	"Generally ignores"	Not documented
`Googlebot`	Google	Search engine, AI Overviews and AI Mode included	Respected	Page missing from Google, and so from its AI answers
`Google-Extended`	Google	Token: Gemini training and grounding	Respected (no separate robot)	Content excluded from training and from grounding in Gemini
`Storebot-Google`	Google	Google Shopping	Respected	Affects every Google Shopping surface
`Bingbot`	Microsoft	Bing search engine	Respected	Page missing from Bing
`Applebot`	Apple	Search engine: Spotlight, Siri, Safari	Respected; follows Googlebot if not named	Page missing from Apple's search
`Applebot-Extended`	Apple	Token: training opt-out	Respected (crawls no page)	Content excluded from training, still in search
`meta-webindexer`	Meta	Meta AI search	Respected	Content not cited in Meta AI answers
`meta-externalagent`	Meta	Training and indexing	Respected	Content excluded from these uses
`meta-externalfetcher`	Meta	User-triggered	"May bypass"	Not documented
`Amzn-SearchBot`	Amazon	Search (Alexa), no training	Respected	Content not eligible for Alexa's search
`Amazonbot`	Amazon	Amazon products and training	Respected	Content excluded from these uses
`Amzn-User`	Amazon	User-triggered	"May not follow all directives"	Not documented
`MistralAI-Index`	Mistral	Search, no training	Not documented	Not documented
`MistralAI-User`	Mistral	User-triggered	Governs which sites it visits	Mistral no longer reads your pages on request
`MistralAI-Training`	Mistral	Training	Respected	Content excluded from training
`DuckAssistBot`	DuckDuckGo	AI answers, no training	Respected, effective within 72 hours	Content missing from DuckDuckGo's AI answers

Two cases deserve one more line.

Google has no separate AI robot for its search. Its AI Overviews and AI Mode read Googlebot's index: according to Google, robots.txt rules for Googlebot are the control, and nosnippet, max-snippet or noindex limit what is reused.

Google-Extended is not just a training opt-out. According to Google, this token also governs the grounding of the Gemini app's answers in the Google Search index. Blocking it changes nothing in Google Search, but can keep your pages out of Gemini's answers.

What blocking costs a store

A store lives on shopping questions: "which sunscreen for sensitive skin", "which bike to commute to work". To answer them with current prices and stock, the assistant relies on what it reads on the web.

A blocked search robot: your product pages can no longer be cited by that assistant. It is the costliest block, and often an accident: a User-agent: * rule that is too broad, an app, a hosting setting.
A blocked user-triggered robot: when a customer pastes the link to one of your products to ask about it, the assistant cannot read it.
A blocked training robot: your pages stay citable. The choice is yours, with the Google-Extended exception above.

Allow every search and user-triggered robot in the list. They decide whether your products can appear in answers.
Decide on training, knowing what it does. Blocking GPTBot or ClaudeBot does not remove you from ChatGPT's or Claude's answers. If Gemini matters to you, keep Google-Extended allowed.
Keep private pages closed to every robot: cart, account, checkout, internal search.
Copy those private pages into every named group. Under the robots.txt standard (RFC 9309), a robot that has its own User-agent group ignores the * group. A User-agent: OAI-SearchBot group that only holds Allow: / also opens your cart to it.
Check after every change of theme, app or host, and make sure /robots.txt answers 200. Under the same standard, a file returning a server error (5xx) means everything is disallowed; a missing file (404) means everything is allowed.

The firewall trap

robots.txt is not the only filter. A firewall or a CDN can turn a robot away before it even reads your file, and nothing shows in robots.txt. Google says so: crawling must be allowed in robots.txt "and by any CDN or hosting infrastructure".

Cloudflare is the most frequent case:

on July 1, 2025, Cloudflare announced it would block AI crawlers by default on new domains;
on July 1, 2026, it split those robots into three categories: search, agent and training. Since September 15, 2026, on new domains, the agent and training categories are blocked by default on pages that display ads, and search stays allowed. A robot that does both search and training follows the most restrictive rule.

If your store sits behind Cloudflare, open its security settings and check what applies to AI robots, on top of your robots.txt.

Content-Signal: stating the use, not just the access

Content-Signal is a robots.txt line proposed by Cloudflare in September 2025 (contentsignals.org). It states what you accept being done with your pages, with three values, yes or no:

search: building a search index and showing links with short excerpts;
ai-input: feeding the content to a model at answer time (grounding, generated search answers);
ai-train: training or fine-tuning a model.

For example: Content-Signal: search=yes, ai-train=no. Cloudflare adds it in this form to the robots.txt files it manages. These are preferences, not a technical barrier, as Cloudflare itself writes. For a store, ai-input=no works against the goal: it asks assistants not to use your pages to answer.

Editing robots.txt

On Shopify

Shopify generates a default robots.txt. To change it, create the robots.txt.liquid template in the theme's Templates folder, from the code editor. This template replaces the generated file. Shopify recommends using its Liquid objects rather than writing the file as plain text, and saving the current robots.txt before publishing to compare the two.

On WooCommerce and WordPress

WordPress serves a virtual robots.txt as long as no file exists at the site's root. Three ways to change it:

with Yoast SEO: Yoast SEO, then Tools, then File editor;
with Rank Math: Rank Math SEO, then General Settings, then Edit robots.txt;
with a physical robots.txt file placed at the root: it replaces the virtual file, and it is then the one to edit.

The delays

A change is not read right away. OpenAI says about 24 hours, Meta up to 24 hours, DuckDuckGo 72 hours.

Check your file

Our AI robots.txt checker reads your file the way these robots do, robot by robot, and suggests the group to add when a search robot is blocked. It is free, with no account.

In skoup, this check is part of each brand's technical audit. A blocked search robot is an error there, a blocked training or user-triggered robot a warning. The audit also reads Content-Signal and checks that your product pages can be read without JavaScript. An open robots.txt is still only a prerequisite: knowing whether your products are actually cited means reading the answers, which the guide Track your products in ChatGPT describes.

Sources

Pages read on October 9, 2026: OpenAI, Anthropic, Perplexity, Google, crawlers, Google, AI features, Apple, Meta, Amazon, Mistral, DuckDuckGo, Cloudflare 2025, Cloudflare 2026, Cloudflare, Content Signals, Shopify, Yoast, Rank Math, RFC 9309.