I think about getting found on Google like boxing. We can spend all the time we want preparing to compete, but if our pages aren't even in Google's index, we're not in contention. We're not even in the ring.
The same business question applies when customers turn to ChatGPT or Perplexity: can those services find and use the information we've put on our website? They don't all work from one database or discover content in the same way. But if the aim is to attract customers through search, we need to give those services a route to our public pages. Otherwise, what are we doing all this work for?
We need to get in the game. Getting found gives us a chance to compete; it doesn't guarantee that we'll be recommended or win the customer's business.
Part of the answer depends on how your website treats automated visitors. Some help people discover your business. Others collect material for AI training, check whether your website works, or attempt to abuse it. Blocking everything can interfere with discovery. Allowing everything can permit uses you never intended.
Your robots.txt file helps you express those choices. Understanding its limits matters just as much as knowing what to put in it.
These decisions belong in the website project from the beginning. As you plan the pages, content and customer journey, you also need to decide how search services can discover that work. Bot access connects website design and development with the ongoing SEO work that follows launch.
What is a bot? What is a crawler?
A bot is software that performs tasks automatically. In this context, “robot” means a software program, not a machine walking through your office. Bots can answer questions, monitor a website, submit forms or request web pages. Their purpose determines whether their activity is useful or harmful. Cloudflare’s explanation of bots
A crawler, also called a spider, is a type of bot that systematically visits web pages. It discovers URLs through sources such as links and sitemaps, retrieves content, and may return later to look for changes. Search engines use crawlers to gather material for their indexes. Other organisations crawl the web to build datasets. Cloudflare’s explanation of web crawlers
Crawling, indexing and appearing in a search result are separate steps. Letting a search crawler visit your page does not guarantee that the page will be indexed, rank well or appear in an AI answer. Google explicitly makes no guarantee of inclusion in its AI search features. Google’s AI search guidance
Why business owners should pay attention
Bot access affects whether search services can discover your work and refresh what they know about it. It also affects server workload and how other organisations collect your content.
For a local service business, preventing a useful crawler from reading service pages can undermine the reason those pages exist. For a publisher selling original research, unrestricted collection may conflict with how the business earns revenue. Neither business should adopt a policy simply because a hosting dashboard offers a button labelled “Block AI.”
There is also a measurement problem: an automated page request is not a potential customer. Review bot activity separately from human visits, enquiries and purchases. A crawler visiting every day does not prove that it recommends your business. Cloudflare distinguishes useful automation from abusive activity in its bot overview.
The practical question is which automated uses support your business, and which you want to restrict.
Where bot access fits into web design and SEO
During website design and development, agree which pages should be public, how visitors will reach them and what information needs protection. Carry those decisions into the site's navigation, page delivery and crawl rules. They belong alongside the design and content brief, with someone responsible for checking the finished result.
Before launch or a redesign goes live, review the production site's robots.txt, indexing directives and sitemap together. Confirm that development restrictions have not carried over to public pages. If page addresses change, check the redirect plan and internal links as well. Protect any preview environment with proper access controls; a robots.txt restriction alone does not make it private.
After launch, this becomes part of technical SEO and ongoing search optimisation. Check whether search engines can reach important pages, investigate indexing problems and review access again when hosting, security settings or the website platform changes. Keep the business's decisions about AI search and training in that review.
For example, a new service page can look correct in a browser while a firewall challenges the crawler trying to retrieve it. The design review confirms that customers can use the page; the technical SEO review checks whether search services can access it. Both checks belong in the handover, with a named person responsible for follow-up.

Search, live retrieval and training are different uses
Search crawlers gather content that a service can find later. A live fetcher visits a page while responding to a user’s request. Training collection gathers material that may help develop a model.
Another term you will encounter is grounding: supplying external information to a model when it produces an answer. Reading your current opening hours to answer a customer’s question is different from using your website to train a future model. Microsoft describes training and grounding as separate uses in its publisher guidance.
Some providers give you separate controls for these uses. Others connect more than one use to the same setting. That is why “allow search and block training” needs a platform-by-platform check.
A practical chart of bots and controls
These are examples a business website owner is likely to encounter, rather than a complete directory. The names in backticks are identifiers used in crawler rules. Google-Extended is a policy token, not a separate crawler making requests under that name.
| Bot or control | Operator and purpose | What allowing or blocking it means |
|---|---|---|
Googlebot | Google’s main search crawler. | Allow access to support Google Search discovery; blocking prevents it from reading the affected content. |
Googlebot-Image, Googlebot-Video, Googlebot-News | Google’s specialised media and news crawl controls. | Restrictions can affect the corresponding search surfaces. News is a control token without its own separate HTTP user-agent string. |
bingbot | Microsoft’s search crawler. | Access supports Bing discovery; Microsoft also has separate policies governing AI uses of collected content. |
OAI-SearchBot | OpenAI’s ChatGPT search crawler. | Blocking opts pages out of ChatGPT search answers, although navigational links may remain. |
Claude-SearchBot | Anthropic’s search crawler for Claude. | Blocking can reduce visibility and the accuracy of search results involving your site. |
PerplexityBot | Perplexity’s search crawler. | Access supports search results and links. Perplexity says it is not used to collect content for foundation-model training. |
ChatGPT-User | OpenAI fetcher for certain user requests. | OpenAI says robots.txt rules may not apply to these user-initiated visits. |
Claude-User | Anthropic fetcher responding to users. | Anthropic says its bots honour robots.txt; blocking limits retrieval on a user’s behalf. |
Perplexity-User | Perplexity fetcher responding to users. | Perplexity says it generally ignores robots.txt for these requests. |
GPTBot | OpenAI’s model-development crawler. | Disallowing signals that content should not be used for foundation-model training; search has a separate control. |
ClaudeBot | Anthropic’s model-development crawler. | Disallowing signals exclusion of future material from training datasets. |
Google-Extended | Google control for specified Gemini training and grounding uses. | Blocking restricts both covered uses; it does not affect Google Search inclusion or rankings. |
CCBot | Common Crawl’s open web dataset crawler. | Blocking stops compliant future collection by this crawler, including collection that could later support AI research. |
Provider references: Google, Microsoft, OpenAI, Anthropic, Perplexity, and Common Crawl.
Other bots check uptime, audit broken links, generate link previews or collect commercial data. Abusive bots may submit spam or attempt account attacks. A name alone does not establish trust: user-agent strings can be forged. Google provides crawler verification guidance, and Common Crawl also warns about impersonators in its CCBot documentation.
What robots.txt does
Robots.txt is a plain-text file placed at a site’s root, such as https://example.com/robots.txt. It tells cooperating crawlers which paths they may fetch. Its core syntax follows the Robots Exclusion Protocol, documented in RFC 9309.
The basic instructions are:
| Instruction | Meaning |
|---|---|
User-agent: | Names the crawler or policy token the following rules address. |
User-agent: * | Supplies fallback rules for crawlers without a matching specific group. |
Disallow: | Identifies a path the matching crawler should not request. |
Allow: | Permits a path, including an exception inside a disallowed area. |
Sitemap: | Gives the full URL of a sitemap; this is a widely supported addition to the core protocol. |
# | Starts a comment for people reading the file. |
Google documents these directives and its matching behaviour in its robots.txt specification.
The file is publicly readable. Listing /private/ does not make that folder private. Passwords and access controls must protect confidential content. Nor does a crawl restriction reliably remove a URL from search: Google can still discover the address through links elsewhere. Google’s robots.txt introduction

Example: permit search crawling and restrict selected training uses
This starting example allows general crawling outside two illustrative folders and blocks four named crawlers or controls. It does not opt out of every AI provider or every possible training source. Google-Extended also carries the grounding tradeoff described below.
Merge it with your site’s existing requirements before use. Replace the example sitemap URL with the real sitemap address and adapt the folder paths.
# General crawl rules, including search crawlers
User-agent: *
Disallow: /admin/
Disallow: /private/
# Restrict selected training/data-collection uses.
# Google-Extended also restricts covered Gemini grounding.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
In this example, search crawlers use the wildcard group. Public pages outside the two disallowed paths remain crawlable without an explicit Allow: /. The named group disallows its matching crawlers or covered uses across the site.
Be careful with templates that give named search bots Allow: /, then put /admin/ restrictions only in the wildcard group. Google uses the most specific matching user-agent group, rather than adding the wildcard group’s rules to it. The named crawler can therefore miss restrictions you intended it to follow. Put any necessary restrictions in each applicable specific group, or keep a simpler fallback structure. Google’s group-matching rules
The consequences of allowing or blocking
Search access creates eligibility
Allowing a search crawler gives the service an opportunity to read your public content. It does not promise a ranking, citation or enquiry. Blocking it can prevent that service from reading updated details or including your content through that route.
Google’s AI Overviews and AI Mode use Google Search eligibility. Google says pages must be indexed and eligible for a search snippet to appear as supporting links; there is no special AI text file required. Google’s AI features guidance
Google-Extended has a visibility tradeoff
Blocking Google-Extended does not change Google Search inclusion or ranking. However, the control covers both future Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. A business choosing that restriction must accept losing eligibility for those covered grounding uses. Google’s description of Google-Extended
Training restrictions have a limited scope
OpenAI separates GPTBot from OAI-SearchBot, so you can express a training opt-out while retaining search access. Anthropic likewise separates ClaudeBot from its search and user-request bots. Those controls do not amount to a promise that no copy of your work can ever reach any model. OpenAI crawler controls, Anthropic crawler controls
Robots.txt communicates instructions to cooperating systems. It cannot force an abusive scraper to comply, erase copies already collected, or reverse completed model training. The underlying protocol explicitly says these rules are not access authorisation. Robots Exclusion Protocol
Live fetching needs its own decision
Do not assume that all visits made on a user’s behalf follow the same rules. OpenAI says robots.txt may not apply to ChatGPT-User, while Perplexity says Perplexity-User generally ignores it. Anthropic says its bots, including Claude-User, honour the file. These visits can arise from a question; they are not limited to someone pasting a URL. OpenAI, Perplexity, Anthropic
Crawling, indexing and security need different controls
| Your goal | Appropriate control | Important limit |
|---|---|---|
| Ask a cooperating crawler not to fetch a path | robots.txt | Compliance is voluntary. |
| Keep a public page out of search indexes | A supported noindex directive in page metadata or an HTTP header | The crawler needs access to read the directive. |
| Keep information confidential | Authentication and server-side access control | A robots.txt entry provides no confidentiality. |
| Reduce abusive automated requests | Firewall, rate limits and bot-management rules | Overbroad rules can also block useful crawlers. |
Google explains the distinction between crawl restrictions, indexing and private content in its robots.txt guide. Perplexity’s firewall guidance illustrates why robots.txt permission alone may not give a bot access through a security layer.
Check the whole route to your content
After changing a policy, verify the actual behaviour:
- Open the live
/robots.txtfile and confirm that it contains your intended rules. Check relevant subdomains separately; a file applies to its own host, protocol and port. - Test representative public URLs and restricted paths for each important crawler. Preserve any necessary access to page resources.
- Check CDN and firewall logs for blocked or challenged crawler requests. Verify claimed bot identities using the provider’s documented method before granting exceptions.
- Inspect whether the public page delivers readable content. Making essential information available in the initial HTML reduces dependence on a crawler rendering JavaScript.
- Use Google Search Console and its robots.txt report and URL Inspection to investigate Google access and indexing. Monitor customer outcomes separately from bot requests.
These checks combine Google’s robots.txt scope rules and AI search technical guidance with the access checks described in Perplexity’s crawler documentation.
For checks beyond crawler access, see Technical SEO: Fix the Website Problems That Cost You Inquiries, which covers other website problems that can interrupt a customer's path to contacting you.
Watch the rollout of shared training preferences
As of November 19, 2026, shared AI training preferences are still developing. Google has announced support for a shared robots.txt policy token, __usage_ai-training, with implementation planned for early 2027. Microsoft says its new training and grounding preferences will take effect as it crawls from February 1, 2027. Do not confuse an announced control with one every provider already enforces. Google’s announcement, Microsoft’s implementation timetable
For your next website review, ask whoever manages the site to document which search services can access public pages, which training uses are restricted, and how those decisions are enforced. Keep that record with your hosting configuration and revisit it when providers change their policies. It gives your business a deliberate position instead of an inherited setting nobody remembers choosing.
If you're planning a new site or a redesign, explore Salient AI's web design and development services. If you already have a website and need to understand what is affecting its search visibility, our SEO services provide a starting point for reviewing technical access, content and priorities.
Written with AI-assisted research and editorial production. Technical documentation checked November 19, 2026.
