AI Crawlers, robots.txt and llms.txt: What to Allow and Why

AI crawlers are automated bots operated by AI companies and search engines that visit websites to collect content, either to train AI models, to build search indexes for AI powered answers, or to fetch pages when a user asks an AI assistant a question. You control their access mainly through your robots.txt file, which tells each crawler which parts of your site it may visit. The llms.txt file is a newer, unofficial proposal for giving AI systems a concise, structured summary of your site. Understanding the difference between these crawlers, and making deliberate choices about them, is now a basic part of managing your visibility in AI search.

This guide is for website owners, marketers, and developers who want to decide which AI crawlers to allow, which to block, and what to expect from llms.txt. I will explain the main types of AI crawlers, why they matter in 2026, how robots.txt works for AI bots, how to make sensible access decisions, what llms.txt is and is not, common mistakes, a practical example, and a checklist. Crawler access is the technical foundation of generative engine optimization: if AI systems cannot reach your content, nothing else you do for AI visibility can work.

What Are AI Crawlers?

A crawler, also called a bot or spider, is software that automatically visits web pages and reads their content. Search engines have used crawlers for decades. AI companies now operate crawlers too, and they generally fall into three categories.

Training crawlers

These collect publicly available web content that may be used to train or improve AI models. Blocking them affects whether your future content is used for training, but it does not usually remove information the model has already learned.

Search and retrieval crawlers

These build indexes that AI search features use to find and cite sources in real time. Blocking them can reduce or eliminate your chances of appearing as a cited source in that platform’s answers.

User initiated fetchers

These visit a specific page when a user asks an AI assistant to look something up or to read a particular URL. They behave more like a browser acting on a person’s behalf than a crawler building an index, and some providers state that these fetchers may not follow robots.txt in the same way as automated crawlers.

The Main AI Crawlers and Tokens

The list below reflects publicly documented user agents and tokens at the time of writing. Providers add and change crawlers regularly, so always check each company’s current documentation before making decisions.

The most important distinction for most businesses is between crawlers that support search visibility and crawlers used mainly for training. Blocking a training crawler and blocking a search crawler have very different consequences.

Why Do AI Crawlers Matter in 2026?

AI assistants and AI search features have become real discovery channels. When someone asks ChatGPT, Perplexity, Copilot, or Google’s AI features a question, the answer often cites web pages. Those citations depend on the platform being able to crawl and index your content. A site that blocks search related AI crawlers, intentionally or not, may simply be absent from those answers. That makes crawler access a direct factor in AI visibility.

At the same time, many businesses have legitimate concerns about their content being used to train AI models without compensation, or about server load from aggressive crawlers. Publishers in particular often want to limit training use while keeping search visibility. The good news is that most major providers separate these functions, so you can make granular decisions rather than choosing between everything and nothing.

There is also a practical risk that many site owners are unaware of: security plugins, firewalls, and content delivery networks increasingly block AI crawlers by default or through bot protection settings. Some CDN providers have introduced options to block AI crawlers automatically for new sites. Unless someone checks, a business can lose AI search visibility without ever deciding to.

How robots.txt Works for AI Crawlers

The robots.txt file sits at the root of your domain, for example example.com/robots.txt. It contains groups of rules, each starting with a User-agent line that names a crawler, followed by Allow and Disallow lines that specify paths. Well behaved crawlers read the file before crawling and follow the rules that apply to them.

Here is a simple example that allows search related AI crawlers while blocking some training crawlers:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

A few important points about how robots.txt behaves:

• It is a set of instructions, not a lock. Reputable crawlers follow it, but it does not technically prevent access. Content that must be private needs authentication.
• Crawlers follow the most specific group. A crawler that finds a group naming it specifically follows that group and ignores the general asterisk group.
• Blocking crawling is not the same as removing from an index. For Google Search, use noindex or other controls to keep pages out of results; robots.txt only controls crawling.
• Changes take time. Crawlers cache robots.txt, so changes may take a day or more to take effect.

How to Decide Which AI Crawlers to Allow

Step 1: Define your goals

Most businesses that sell products or services want maximum visibility wherever buyers research. For them, allowing search related crawlers is usually the right choice. Publishers whose revenue depends on people reading content on their site may want to limit training use and think carefully about AI search trade offs. Businesses with proprietary content, such as paid courses or research, may want stricter controls on those sections.

Step 2: Separate search from training

Decide on search crawlers and training crawlers separately. A common approach for businesses focused on visibility is to allow search and retrieval crawlers such as Googlebot, Bingbot, OAI-SearchBot, and PerplexityBot, and to decide on training crawlers according to your comfort level. There is no universally correct answer for training crawlers; it is a business decision.

Step 3: Protect sensitive sections

Block areas that should not be crawled by any bot, such as admin areas, internal search results, staging environments, and customer account pages. Genuinely private information should be protected by login, not just robots.txt.

Step 4: Check your firewall, CDN, and security settings

Review bot management settings in your CDN, hosting control panel, and security plugins. Look for rules that block AI crawlers, challenge unknown bots, or rate limit aggressively. Make sure these settings match the decisions in your robots.txt. A technical SEO audit should always include this check.

Step 5: Verify with server logs

Server logs show which crawlers actually visit, how often, and what responses they receive. If a crawler you intended to allow is receiving 403 errors, something is blocking it. If an unfamiliar bot is consuming significant resources, you can investigate and decide whether to block it.

Step 6: Document and review

Write down which crawlers you allow and block, and why. Review the decision every six months, because new crawlers appear and providers change how they operate.

What Is llms.txt?

llms.txt is a proposed convention, published in 2024, for placing a Markdown file at the root of a website, for example example.com/llms.txt, that gives language models a concise overview of the site: what it is, and links to the most important pages or documentation, ideally with short descriptions. Some proposals also suggest providing clean Markdown versions of key pages.

The idea is appealing, especially for documentation heavy sites such as software products. A clean summary could help AI tools find the most relevant information without parsing complex HTML.

However, it is important to be accurate about its status. llms.txt is not an official web standard, and at the time of writing major AI providers have not confirmed that their search or training systems rely on it. Google has indicated that it does not use llms.txt for Search. Adoption has been strongest among developer tools and documentation sites, where coding assistants and similar tools may read it when directed to.

My recommendation is practical. If you have extensive documentation or a complex product, creating an llms.txt file is a low cost experiment that may help some tools. For most business websites, it is optional. It should never replace the fundamentals: crawlable pages, clear content, strong internal linking, and accurate structured data as described in my schema markup guide.

A simple llms.txt file might look like this:

# Example Consulting

> Example Consulting is a digital marketing consultancy specializing in SEO, answer engine optimization, and paid search for B2B companies.

## Key pages

- [Services](https://www.example.com/services/): Overview of consulting services
- [SEO strategy guide](https://www.example.com/seo-strategy-guide/): Our framework for building an SEO strategy
- [Contact](https://www.example.com/contact/): How to get in touch

Beyond Access: Making Content Usable

Allowing crawlers is only the first step. Content must also be readable and useful once they arrive. Important information should be present in HTML rather than only in images, PDFs, or content loaded by complex scripts. Pages should load reliably and quickly, because some crawlers have tight time limits; my Core Web Vitals guide covers performance. And your business should be described clearly and consistently, as covered in my guide to entity SEO and the Knowledge Graph.

Crawler Decisions by Business Type

There is no single policy that suits every website. These are the patterns I most often recommend.

Service businesses and B2B companies

For businesses whose content exists mainly to attract and inform potential customers, visibility is usually the priority. Allowing search related crawlers from Google, Microsoft, OpenAI, Perplexity, and other major platforms makes sense. Many of these businesses also allow training crawlers, reasoning that being well represented in future models is more valuable than restricting their marketing content. Others prefer to block training use on principle. Either choice is reasonable if it is deliberate.

Publishers and content businesses

Publishers whose revenue depends on page views, subscriptions, or licensing face a harder trade off. AI answers may reduce visits, yet being cited can still bring audiences and brand recognition. Many publishers block training crawlers while allowing search crawlers, and some negotiate licensing agreements directly with AI companies. These decisions deserve input from leadership, not only from the technical team.

Ecommerce stores

Product information, pricing, and availability are exactly what shoppers ask AI assistants about. Allowing search related crawlers helps products appear in AI shopping and research answers. Protect checkout, account, cart, and internal search pages from crawling, and keep product feeds and structured data accurate.

Sites with proprietary content

Paid courses, premium research, and member content should sit behind authentication. Public marketing pages about those products can remain open to crawlers so that people can discover them, while the valuable content itself stays protected.

Common Mistakes

• Blocking all bots by accident. Broad firewall or CDN bot rules can block search related AI crawlers without anyone noticing.
• Confusing training and search crawlers. Blocking OAI-SearchBot when you only meant to block GPTBot removes you from ChatGPT search results.
• Assuming Google-Extended affects AI Overviews. According to Google, it does not affect inclusion in Google Search, including Google AI Overviews.
• Using robots.txt for privacy. Sensitive content needs authentication, not just a Disallow rule.
• Relying on llms.txt instead of fundamentals. An unofficial file cannot compensate for inaccessible or unclear content.
• Syntax errors. A misplaced rule or typo can block far more than intended.
• Never checking logs. Without logs, you cannot confirm what crawlers actually experience.

Best Practices

• Make a deliberate, documented decision for each major AI crawler.
• Treat search related crawlers and training crawlers as separate decisions.
• Keep robots.txt simple, tested, and version controlled.
• Align CDN, firewall, and plugin settings with your robots.txt decisions.
• Review server logs to confirm crawler behavior.
• Protect genuinely private content with authentication.
• Consider llms.txt as an optional extra, especially for documentation sites.
• Review crawler policies every six months as providers change.

Practical Example: A B2B Software Company

Consider a B2B software company whose leadership noticed that competitors appeared in ChatGPT and Perplexity answers about their category while their own product rarely did. This is an illustrative scenario. The team assumed the problem was content quality.

A crawler review would start with robots.txt, which turned out to allow all bots. The server logs, however, showed that requests from OAI-SearchBot and PerplexityBot were receiving 403 errors. The cause was a bot protection setting in the company’s CDN that challenged unrecognized crawlers, which had been enabled during a security project the previous year. Googlebot and Bingbot were allowed because they were on the CDN’s verified list, so traditional SEO had never shown a problem.

The fix would involve adjusting the CDN rules to allow the search related AI crawlers the company wanted, while keeping protection against abusive bots. The team would also decide to block GPTBot and CCBot for training purposes, document the policy, and add an llms.txt file pointing to its documentation and key product pages. After the change, logs would confirm successful crawls, and the team would monitor AI answers over the following weeks.

Access alone would not guarantee mentions, but it removes a barrier that made them nearly impossible. The broader work of earning visibility is covered in my guide to getting your brand mentioned in ChatGPT.

AI Crawler and robots.txt Checklist

• Current robots.txt reviewed and tested.
• Decision documented for each major AI crawler and token.
• Search related crawlers allowed if AI search visibility is a goal.
• Training crawler decisions made deliberately.
• Admin, staging, and private areas protected appropriately.
• CDN, firewall, hosting, and security plugin settings checked.
• Server logs confirm allowed crawlers receive successful responses.
• Important content available in crawlable HTML.
• Sitemap referenced in robots.txt.
• llms.txt considered where it adds value.
• Policy reviewed every six months.

Frequently Asked Questions

Should I block AI crawlers?

It depends on your goals. If you want visibility in AI search tools, allow their search related crawlers. Training crawlers are a separate decision based on how you feel about your content being used to train models.

Does blocking GPTBot remove me from ChatGPT?

Not from ChatGPT search, which uses OAI-SearchBot. GPTBot relates to training. Blocking OAI-SearchBot is what would affect whether your site appears in ChatGPT search answers.

Does Google-Extended control AI Overviews?

No. According to Google, Google-Extended does not affect inclusion in Google Search, including AI Overviews. Search snippet controls such as nosnippet affect how content can appear there.

Is llms.txt required?

No. It is an unofficial proposal, and major AI providers have not confirmed relying on it. It can be a useful extra for documentation heavy sites.

Will blocking AI crawlers remove my content from existing models?

Generally not. Blocking affects future crawling. Information already included in a model’s training data is not removed by changing robots.txt.

Can AI crawlers slow down my website?

Occasionally. Aggressive crawling can add server load, particularly on smaller hosting plans. If logs show a specific bot causing problems, you can rate limit it through your CDN or firewall, or block it in robots.txt if it does not support your goals. Blocking every AI crawler to solve a load problem usually costs more visibility than it saves.

How do I know which bots visit my site?

Check your server access logs or analytics tools that record bot traffic. Many hosting control panels and CDNs also provide bot reports.

Conclusion

AI crawlers now determine whether your content can be used in some of the fastest growing discovery channels. Understand the difference between training crawlers, search crawlers, and user initiated fetchers. Make deliberate, documented decisions in robots.txt, and make sure your CDN, firewall, and security plugins agree with those decisions. Verify with server logs, protect private content properly, and treat llms.txt as an optional extra rather than a solution.

Access is the foundation. Once the right crawlers can reach your content, the work of being understood and cited, through clear content, strong entities, and credible mentions, can begin to pay off. The relationship between these layers is explained in SEO vs AEO vs GEO.

Not Sure What Your Site Allows?

If you are unsure which AI crawlers can access your website, or suspect a firewall or plugin is blocking them, I can review your robots.txt, server logs, and bot settings and recommend a policy that matches your goals. Contact me through DigitalKetan.com to discuss your AI visibility.