# AI crawlers: which ones to allow

> Search, user-triggered, training: each assistant sends several robots. What each one does, what blocking it costs you, and how to set robots.txt on Shopify and WooCommerce.

Updated on October 9, 2026 · by the skoup team

## Four families of robots

An AI assistant does not send one robot to your store, but several. Each has a role, described by its vendor, and that role tells you what blocking it costs.

- **Search robots** build the index the assistant looks in for its answers. Blocked, they can no longer cite your pages.
- **User-triggered robots** read a page when a user asks a question or pastes a link. Several vendors warn that robots.txt may not apply to them.
- **Training robots** collect content for the next models. Blocking them does not remove your pages from search answers.
- **Search engines** (Googlebot, Bingbot, Applebot): their index also serves the assistants, such as Google's AI Overviews or Siri at Apple.

## The table, vendor by vendor

This table follows each vendor's public documentation, read on October 9, 2026. The links are at the bottom of the page.

| Robot                  | Vendor     | Role                                             | robots.txt according to the docs          | Effect of blocking it                                          |
| ---------------------- | ---------- | ------------------------------------------------ | ----------------------------------------- | -------------------------------------------------------------- |
| `OAI-SearchBot`        | OpenAI     | ChatGPT search                                   | Respected                                 | The site no longer appears in ChatGPT search answers           |
| `ChatGPT-User`         | OpenAI     | User-triggered                                   | "May not apply"                           | No effect on search, according to OpenAI                       |
| `GPTBot`               | OpenAI     | Training                                         | Respected                                 | Content excluded from training                                 |
| `Claude-SearchBot`     | Anthropic  | Search                                           | Respected                                 | Content not indexed, reduced visibility according to Anthropic |
| `Claude-User`          | Anthropic  | User-triggered                                   | Respected                                 | Claude no longer reads your pages to answer                    |
| `ClaudeBot`            | Anthropic  | Training                                         | Respected                                 | Future content excluded from training                          |
| `PerplexityBot`        | Perplexity | Search, no training                              | Respected                                 | The site no longer appears in Perplexity's results             |
| `Perplexity-User`      | Perplexity | User-triggered                                   | "Generally ignores"                       | Not documented                                                 |
| `Googlebot`            | Google     | Search engine, AI Overviews and AI Mode included | Respected                                 | Page missing from Google, and so from its AI answers           |
| `Google-Extended`      | Google     | Token: Gemini training and grounding             | Respected (no separate robot)             | Content excluded from training and from grounding in Gemini    |
| `Storebot-Google`      | Google     | Google Shopping                                  | Respected                                 | Affects every Google Shopping surface                          |
| `Bingbot`              | Microsoft  | Bing search engine                               | Respected                                 | Page missing from Bing                                         |
| `Applebot`             | Apple      | Search engine: Spotlight, Siri, Safari           | Respected; follows Googlebot if not named | Page missing from Apple's search                               |
| `Applebot-Extended`    | Apple      | Token: training opt-out                          | Respected (crawls no page)                | Content excluded from training, still in search                |
| `meta-webindexer`      | Meta       | Meta AI search                                   | Respected                                 | Content not cited in Meta AI answers                           |
| `meta-externalagent`   | Meta       | Training and indexing                            | Respected                                 | Content excluded from these uses                               |
| `meta-externalfetcher` | Meta       | User-triggered                                   | "May bypass"                              | Not documented                                                 |
| `Amzn-SearchBot`       | Amazon     | Search (Alexa), no training                      | Respected                                 | Content not eligible for Alexa's search                        |
| `Amazonbot`            | Amazon     | Amazon products and training                     | Respected                                 | Content excluded from these uses                               |
| `Amzn-User`            | Amazon     | User-triggered                                   | "May not follow all directives"           | Not documented                                                 |
| `MistralAI-Index`      | Mistral    | Search, no training                              | Not documented                            | Not documented                                                 |
| `MistralAI-User`       | Mistral    | User-triggered                                   | Governs which sites it visits             | Mistral no longer reads your pages on request                  |
| `MistralAI-Training`   | Mistral    | Training                                         | Respected                                 | Content excluded from training                                 |
| `DuckAssistBot`        | DuckDuckGo | AI answers, no training                          | Respected, effective within 72 hours      | Content missing from DuckDuckGo's AI answers                   |

Two cases deserve one more line.

**Google has no separate AI robot for its search.** Its AI Overviews and AI Mode read Googlebot's index: according to Google, robots.txt rules for Googlebot are the control, and `nosnippet`, `max-snippet` or `noindex` limit what is reused.

**`Google-Extended` is not just a training opt-out.** According to Google, this token also governs the grounding of the Gemini app's answers in the Google Search index. Blocking it changes nothing in Google Search, but can keep your pages out of Gemini's answers.

## What blocking costs a store

A store lives on shopping questions: "which sunscreen for sensitive skin", "which bike to commute to work". To answer them with current prices and stock, the assistant relies on what it reads on the web.

- **A blocked search robot**: your product pages can no longer be cited by that assistant. It is the costliest block, and often an accident: a `User-agent: *` rule that is too broad, an app, a hosting setting.
- **A blocked user-triggered robot**: when a customer pastes the link to one of your products to ask about it, the assistant cannot read it.
- **A blocked training robot**: your pages stay citable. The choice is yours, with the `Google-Extended` exception above.

## What we recommend to a store

1. **Allow every search and user-triggered robot** in the list. They decide whether your products can appear in answers.
2. **Decide on training**, knowing what it does. Blocking `GPTBot` or `ClaudeBot` does not remove you from ChatGPT's or Claude's answers. If Gemini matters to you, keep `Google-Extended` allowed.
3. **Keep private pages closed** to every robot: cart, account, checkout, internal search.
4. **Copy those private pages into every named group.** Under the robots.txt standard (RFC 9309), a robot that has its own `User-agent` group ignores the `*` group. A `User-agent: OAI-SearchBot` group that only holds `Allow: /` also opens your cart to it.
5. **Check after every change** of theme, app or host, and make sure `/robots.txt` answers 200. Under the same standard, a file returning a server error (5xx) means everything is disallowed; a missing file (404) means everything is allowed.

## The firewall trap

robots.txt is not the only filter. A firewall or a CDN can turn a robot away before it even reads your file, and nothing shows in robots.txt. Google says so: crawling must be allowed in robots.txt "and by any CDN or hosting infrastructure".

Cloudflare is the most frequent case:

- on July 1, 2025, Cloudflare announced it would block AI crawlers by default on new domains;
- on July 1, 2026, it split those robots into three categories: search, agent and training. Since September 15, 2026, on new domains, the agent and training categories are blocked by default on pages that display ads, and search stays allowed. A robot that does both search and training follows the most restrictive rule.

If your store sits behind Cloudflare, open its security settings and check what applies to AI robots, on top of your robots.txt.

## Content-Signal: stating the use, not just the access

Content-Signal is a robots.txt line proposed by Cloudflare in September 2025 ([contentsignals.org](https://contentsignals.org/)). It states what you accept being done with your pages, with three values, `yes` or `no`:

- `search`: building a search index and showing links with short excerpts;
- `ai-input`: feeding the content to a model at answer time (grounding, generated search answers);
- `ai-train`: training or fine-tuning a model.

For example: `Content-Signal: search=yes, ai-train=no`. Cloudflare adds it in this form to the robots.txt files it manages. These are preferences, not a technical barrier, as Cloudflare itself writes. For a store, `ai-input=no` works against the goal: it asks assistants not to use your pages to answer.

## Editing robots.txt

### On Shopify

Shopify generates a default robots.txt. To change it, create the `robots.txt.liquid` template in the theme's Templates folder, from the code editor. This template replaces the generated file. Shopify recommends using its Liquid objects rather than writing the file as plain text, and saving the current robots.txt before publishing to compare the two.

### On WooCommerce and WordPress

WordPress serves a virtual robots.txt as long as no file exists at the site's root. Three ways to change it:

- with **Yoast SEO**: Yoast SEO, then Tools, then File editor;
- with **Rank Math**: Rank Math SEO, then General Settings, then Edit robots.txt;
- with a **physical `robots.txt` file** placed at the root: it replaces the virtual file, and it is then the one to edit.

### The delays

A change is not read right away. OpenAI says about 24 hours, Meta up to 24 hours, DuckDuckGo 72 hours.

## Check your file

Our [AI robots.txt checker](https://www.skoup.ai/en/tools/ai-robots-txt-checker/) reads your file the way these robots do, robot by robot, and suggests the group to add when a search robot is blocked. It is free, with no account.

In skoup, this check is part of each brand's technical audit. A blocked search robot is an error there, a blocked training or user-triggered robot a warning. The audit also reads Content-Signal and checks that your product pages can be read without JavaScript. An open robots.txt is still only a prerequisite: knowing whether your products are actually cited means reading the answers, which the guide [Track your products in ChatGPT](https://www.skoup.ai/en/guides/track-products-in-chatgpt/) describes.

## Sources

Pages read on October 9, 2026: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/guides/bots), [Google, crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), [Google, AI features](https://developers.google.com/search/docs/appearance/ai-features), [Apple](https://support.apple.com/en-us/119829), [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers), [Amazon](https://developer.amazon.com/amazonbot), [Mistral](https://docs.mistral.ai/robots), [DuckDuckGo](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot), [Cloudflare 2025](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), [Cloudflare 2026](https://blog.cloudflare.com/content-independence-day-ai-options/), [Cloudflare, Content Signals](https://blog.cloudflare.com/content-signals-policy/), [Shopify](https://shopify.dev/docs/storefronts/themes/seo/robots-txt), [Yoast](https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo), [Rank Math](https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/), [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html).
