SEO & Online Marketing

llms.txt, GPTBot & Co.: Should my store control AI crawlers?

Nov 19, 2026·Author: Stephan Dunger

llms.txt, GPTBot & Co.: Should my store control AI crawlers?

Every website should control its AI crawlers—the question is, how? The much-discussed llms.txt is a community suggestion with no binding effect; the robots.txt file still determines which bots are actually allowed to crawl the site—and, in case of doubt, the bot rule in the CDN. This post explains which AI crawlers exist, how to selectively allow or block them, and what llms.txt can realistically achieve.

At a Glance

  • llms.txt (llmstxt.org) is a Markdown file in the root directory that offers curated content to AI systems—a community proposal, not a standard adopted by OpenAI, Google, or Anthropic.
  • Binding control is handled via robots.txt directives for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and others—plus CDN/WAF rules as a hard boundary.
  • Both allowing and blocking are legitimate —what matters is a conscious decision rather than a random configuration based on generic bot lists.
  • The body of research on the benefits of llms.txt is limited (as of November 2026)—low effort, uncertain return.

What is llms.txt?

The llms.txt is a proposal from the AI community (llmstxt.org, launched in September 2024): a Markdown file in the website’s root directory that provides AI systems with a curated overview—who the provider is, what content is important, and where the machine-readable entry points are located. In addition, the proposal includes an llms-full.txt file that bundles all content in full text. The idea: Instead of a language model having to work its way through navigation, cookie banners, and template code, it is served the essence of the website directly. Important to note: None of the major AI providers has formally committed to evaluating the file—it is an offer, not a protocol.

Does llms.txt really control what AI crawlers do?

No. The llms.txt file can offer content, but it cannot prohibit anything—it’s the storefront, not the bouncer. The actual control of AI crawlers takes place on two other levels: First, via the robots.txt file, where the documented bots of the major providers can be addressed by user-agent. Second, through CDN and firewall rules (Cloudflare, etc.), which technically enforce access restrictions—this is relevant because the robots.txt is a convention that not every bot demonstrably adheres to. If you configure both properly, you don’t need llms.txt for control; if you create it as well, you have nothing to lose.

Which AI crawlers should I be aware of?

The most important documented user-agents, sorted by purpose (as of November 2026—the providers’ lists are constantly growing):

User-AgentOperatorPurpose
GPTBotOpenAITraining future GPT models.
OAI-SearchBotOpenAISearch index for ChatGPT search—provides answers with source links.
ChatGPT UsersOpenAIRetrieves individual pages in real time when a user requests them.
ClaudeBotAnthropicCrawling for Claude, including for training purposes.
Google-ExtendedGoogleNot a dedicated crawler, but a control token: determines whether Gemini is allowed to use the content for training future models and as a source for responses in the Gemini app and in Vertex AI (grounding).
PerplexityBotPerplexitySearch index of the Perplexity answer engine.
Meta-ExternalAgentMetaTraining of the Meta AI models.
CCBotCommon CrawlOpen Web Archive—feeds many third-party training datasets.

One detail is often misunderstood: Google-Extended affects Gemini outside of Search—specifically, the training of future models and its use as a source for answers in the Gemini app and in Vertex AI (grounding). The AI Overviews and AI Mode in Google Search rely on the regular Googlebot—so if you block Google-Extended, you won’t disappear from the AI answers in Search, but you will no longer appear as a source in the Gemini app. We’ve outlined what changes there in our post on AI Mode and AI Overviews.

How do I block or allow GPTBot & Co.?

Using standard robots.txt directives, on a per-user-agent basis. A common approach for online stores: allow response and search crawlers (visibility), block training crawlers (content protection):

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Allow response and search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

For Google-Extended, the following applies: The block also removes the store as a source from the Gemini app; AI responses from Google Search remain unaffected. The explicit “Allow” directive is usually technically redundant (what isn’t prohibited is allowed), but it documents the deliberate decision. Next, check the second level: Many CDNs and firewalls maintain generic “AI bot” or “bad bot” lists that also include response crawlers—a robots.txt allowance is useless if the WAF rejects the same bot with an HTTP 403. Both levels must reflect the same decision. The effort involved is manageable—a quarter of an hour that can easily be squeezed in even during the week leading up to Black Friday.

Allow or block—which is right for my store?

There are two sides to this consideration. Visibility is an argument in favor of allowing access: Anyone who wants to appear as a source in AI responses must let the response and search crawlers in—the basics of this are covered in the article on GEO fundamentals; the same applies to future shopping bots: if you block them, you’re invisible. Arguments for blocking include content protection (not providing editorial content for training free of charge) and server load, since AI crawlers can generate significant traffic volumes. The pragmatic middle ground for most online stores: allow search and answer crawlers, but evaluate training crawlers separately. One thing is especially wrong—failing to make this decision at all and leaving it up to a pre-provided bot list.

Does llms.txt improve AI visibility?

Honest answer: It’s unclear. There are currently no reliable studies showing an effect of llms.txt on citation frequency or AI traffic (as of this writing); analyses of server logs show only sporadic access to the file among many operators, and no major provider has officially committed to evaluating it. This comes at very little cost: An llms.txt file can be created and maintained in an hour. Our assessment: Create one, but don’t expect miracles—visibility comes from the content, structured data, and crawler access. Our own llms.txt, currently in use, shows what such a file looks like.

What the XONIC Shop System Offers for This

AI crawler control has been included since version 4.8.2 and precisely separates the two levels discussed in this post: The AI_BOT_ALLOW switch specifically allows response and search crawlers, while AI_TRAINING_ROBOTS_DISALLOW automatically writes exclusions for training crawlers into the robots.txt file—without any manual editing of the file. Added to this is the AI knowledge export, which makes curated store content available in a machine-readable format—essentially the concept behind llms.txt, but structured. Product and content pages provide the citable foundation right out of the box: Product JSON-LD including the merchant feed, FAQPage microdata, and NewsArticle JSON-LD with an author profile. The feature overview provides a summary.

Frequently Asked Questions About llms.txt and AI Crawlers

The robots.txt file controls access: It allows or prohibits crawlers from reading the website and is an established standard. The llms.txt file offers the following: It presents curated content to AI systems, but cannot enforce anything—and no major provider has committed to evaluating it.

Yes—providers use separate user agents for this. If you block GPTBot (training) but allow OAI-SearchBot along with ChatGPT-User (search and live retrieval), you’ll remain quotable in ChatGPT search results. For Google, the following applies: Google-Extended handles training and grounding for Gemini (Gemini app, Vertex AI), not the AI responses in Google Search.

The robots.txt file is a convention, not a technical block. Major providers document their bots and state that they respect the directives; this does not apply to undocumented crawlers. If you need a strict restriction, you should also implement CDN or firewall rules.

No, as long as the regular Googlebot remains allowed—blocking GPTBot, ClaudeBot, or Google-Extended does not affect standard Google indexing. However, anyone who blocks all bots across the board also risks affecting search results themselves; therefore, always review bot lists in detail.

Sources

Stephan Dunger
About the author

Stephan Dunger

CTO · XONIC Solutions GmbH · With the company since 2012

Stephan Dunger is one of the brains behind the XONIC store system. He has been developing the platform together with the team since 2012 - from the database to the interfaces to the checkout.

A passionate programmer, technical mind and consultant at the same time: with his in-depth knowledge of store systems and e-commerce, Stephan combines the depth of a developer with an eye for the big picture. Together with the XONIC team, he shapes the technical direction, consistently thinks about functions from the retailer's perspective and advises on customized solutions.

The result is software with a face: customers don't get an anonymous provider, but a direct line to the people who develop XONIC. Pragmatic, fast and at eye level.

Customer testimonials

write review
Never miss a thing.

Legal updates and new features straight to your inbox.

Subscribe to the newsletter