AI Crawlers on Healthcare Websites: What to Allow

A healthcare website should allow the AI crawlers that power search and citations, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and treat training crawlers such as GPTBot, ClaudeBot and Meta-ExternalAgent as a separate choice that OpenAI, Anthropic and Meta each let you make by name in robots.txt (OpenAI; Anthropic; Meta). Crawlers, not people, make most AI bot traffic: in Curve's own logs for curvecompliance.com, September 29 to October 25, 2026, Meta-ExternalAgent alone made 46.2% of 140,760 AI fetches, while fetches triggered by a person asking an assistant made 9.6%. Curve Compliance signs a BAA on every plan, and its free website scan lists the pixels and trackers each page loads.

Book a call. Curve Compliance replaces browser ad pixels with one script and sends conversions server-side, and Curve's team does the setup. Book a call with Curve.

What we measured on curvecompliance.com

The table counts AI bot fetches of www.curvecompliance.com, grouped by bot name, from Curve's own logs for curvecompliance.com, September 29 to October 25, 2026. These are aggregates only, with nothing per visitor. It is one healthcare marketing site, not a clinic, so treat it as a signal rather than a benchmark.

Bot	Fetches	Distinct paths	Last 7 days (Oct 19 to 25)
meta-externalagent	65,009	1,278	9,660
OAI-SearchBot	26,176	810	7,363
Bingbot	10,371	1,555	1,577
Bytespider	9,891	4,100	2,559
ChatGPT-User	9,804	637	2,654
PerplexityBot	4,673	802	1,417
Amazonbot	4,569	1,365	919
Claude-User	3,657	344	1,258
GPTBot	2,771	714	2,003
ClaudeBot	2,089	851	391
8 others	1,750
Total	140,760

Three things stand out in Curve's own logs for curvecompliance.com, September 29 to October 25, 2026:

• Meta's AI crawler leads. Meta-ExternalAgent made 65,009 fetches, 46.2% of the total. Meta says it crawls "for use cases such as training foundation AI models or improving products by indexing content directly" (Meta).
• People asking assistants are a small share. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User) made 13,573 fetches, or 9.6%.
• GPTBot is speeding up. It went from 31 fetches (October 12 to 18) to 2,003 (October 19 to 25).

Three kinds of AI bots

Each operator splits its bots by job, and the split decides what robots.txt can do for you.

Operator	Trains AI models	Search and citations	Fetches for a user's question
OpenAI (docs)	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic (docs)	ClaudeBot	Claude-SearchBot	Claude-User
Meta (docs)	Meta-ExternalAgent (also product indexing)	Meta-WebIndexer	Meta-ExternalFetcher
Perplexity (docs)	Perplexity says PerplexityBot "is not used to crawl content for AI foundation models"	PerplexityBot	Perplexity-User

OpenAI puts it plainly: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" (OpenAI). Anthropic says restricting ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets" (Anthropic). Meta says "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses" (Meta). Google works differently: Google-Extended is "a standalone product token" for Gemini training and grounding, and it "does not impact a site's inclusion in Google Search" (Google).

Which AI bots skip robots.txt

OpenAI, Perplexity and Meta say their user-triggered fetchers may not follow robots.txt. Anthropic says its bots do. In their own words:

• ChatGPT-User. "Because these actions are initiated by a user, robots.txt rules may not apply." (OpenAI)
• Perplexity-User. "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." (Perplexity)
• Meta-ExternalFetcher. "The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user." FacebookExternalHit "might bypass robots.txt when performing security or integrity checks" (Meta).
• Claude-User. Anthropic says its bots honor "industry standard directives in robots.txt," and that disabling Claude-User "may reduce your site's visibility for user-directed web search" (Anthropic).

These fetches are people asking an assistant about you or your topic. In Curve's own logs for curvecompliance.com, September 29 to October 25, 2026, they were 9.6% of AI fetches. For how those answers turn into visits, see tracking ChatGPT and Perplexity referrals.

Robots.txt is a request, not a lock

Google says "The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them," and that to keep information secure "it's better to use other blocking methods, such as password-protecting private files on your server" (Google). Patient portals, intake answers and anything with patient details belong behind a login, never on a public URL that robots.txt merely asks bots to skip. For what counts as PHI on a page, see what counts as PHI on a website.

Meta's November 20 terms: crawlers and test credentials

Meta's updated Commercial Terms say "These updates will go into effect on November 20, 2026." The new clause reads in full: "You authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs (and any content available through them) that are associated with your products or services, or that are made available through your or users' use of Meta Company Products and services, and to periodically pull updates from such domains, web pages, or URLs in order to provide, develop, improve, research, and maintain advertising and business products (including business, commercial and related products to help you create and deliver ads and other commercial content, provide users information about your products and services, and connect you with users who may be interested in your products and services) and maintain the integrity and security of Meta Company Products." (Meta)

The clause does not mention robots.txt, does not list which pages, and does not say how Meta will create or use test credentials. Meta's crawler page lists Meta-ExternalAds, which "crawls the web for use cases such as improving advertising and other business-related products and services," and says "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt" (Meta). If you advertise on Meta and need to know how the two fit together, talk to your counsel. For the same update's effect on Meta's business AI, see Meta Business AI for clinics.

Before November 20, check the pages your Meta ads lead to:

• List the trackers on each landing page. Curve's free website scan shows which pixels and trackers a page loads.
• Keep ad pixels off logged-in pages. HHS says "Tracking technologies on a regulated entity's user-authenticated webpages generally have access to PHI" (HHS).
• Review every sign-up and login page. The clause covers creating test credentials, so know which of your pages let anyone create an account.
• Keep PHI out of URLs and page titles. See UTM parameters and PHI.

OAI-AdsBot and ad landing pages

If you run ChatGPT Ads, OpenAI says "OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models" (OpenAI). Its ad policies add: "Ads or landing pages that cannot be reviewed or evaluated by our systems are not eligible to run." (OpenAI) So keep ad landing pages open to OAI-AdsBot even if you block GPTBot. For each AI platform's ad rules, see healthcare ads in AI assistants.

Crawlers are not agents

A crawler reads pages. An agent acts on them. Meta says Muse works "by giving every person their own computer and browser in the cloud," and "Today Muse buys things, makes appointments, and resolves support questions through a connector or directly on the business's website" (Meta, October 6, 2026). OpenAI says ChatGPT Work's Cloud browser "uses Web Bot Auth to sign outbound HTTP requests," with a Signature-Agent header pointing to chatgpt.com (OpenAI). For what agent bookings mean for conversion tracking, see when an AI agent books the appointment.

A robots.txt for a clinic

A starting point if you want AI assistants to cite you but want future content kept out of training sets. Check each operator's page before you copy it, since bot names change, and note that blocking Meta-ExternalAgent also covers the product indexing Meta describes. Blocking Google-Extended also opts your pages out of "grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps," so drop that row if you want Gemini to draw on your site (Google). For llms.txt, see llms.txt for healthcare sites.

User-agent line	Rule	What it covers
User-agent: GPTBot	Disallow: /	Training (OpenAI)
User-agent: ClaudeBot	Disallow: /	Training (Anthropic)
User-agent: meta-externalagent	Disallow: /	Training and product indexing (Meta)
User-agent: Google-Extended	Disallow: /	Gemini training and grounding (Google)
User-agent: OAI-SearchBot	Allow: /	ChatGPT search (OpenAI)
User-agent: Claude-SearchBot	Allow: /	Search (Anthropic)
User-agent: PerplexityBot	Allow: /	Search (Perplexity)
User-agent: meta-webindexer	Allow: /	Meta AI citations (Meta)

Each row is one group in your robots.txt file: the User-agent line, then its rule on the next line.

How Curve Compliance helps

Crawlers read your pages; ad platforms read what your tags send. Curve's free website scan lists the pixels and trackers on a page, so you can see what loads on condition and booking pages; see how to check which trackers your site loads. Curve Compliance replaces browser ad pixels with one script and sends conversions server-side to Meta, Google Ads, TikTok, Microsoft Advertising, LinkedIn and ChatGPT Ads.

Each platform receives only a fixed list of fields, events can use neutral names, and contact identifiers are off by default on Meta, Google Ads and TikTok. Curve checks every event for PHI-like patterns and strips PHI before data reaches an ad platform. Curve signs a BAA on every plan and is SOC 2 Type II, and Curve's team does the setup, typically live in about a week.

Sources, checked October 26, 2026: OpenAI, Overview of OpenAI Crawlers; OpenAI, ChatGPT Work's Cloud browser allowlisting; OpenAI ad policies; Meta Web Crawlers; Meta Commercial Terms, effective November 20, 2026; Meta, A New Way for Businesses and Personal Agents to Work Together; Anthropic, Does Anthropic crawl data from the web; Perplexity Crawlers; Google's common crawlers; Google, Introduction to robots.txt; HHS, Use of Online Tracking Technologies; Curve's own logs for curvecompliance.com, September 29 to October 25, 2026 (aggregates).

Frequently Asked Questions

Which AI crawlers should a healthcare website allow?

Allow the search bots you want to be cited by, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer (OpenAI; Anthropic; Perplexity; Meta). Training crawlers such as GPTBot, ClaudeBot and Meta-ExternalAgent are your call, and each can be blocked by name in robots.txt.

Should a clinic block GPTBot?

It is a choice about training, not search. OpenAI says "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models," and that each bot setting is independent (OpenAI).

What is meta-externalagent?

Meta's AI crawler, which "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly" (Meta). In Curve's own logs for curvecompliance.com, September 29 to October 25, 2026, it made 46.2% of AI fetches.

Does blocking GPTBot remove me from ChatGPT search?

No. Search uses OAI-SearchBot, and OpenAI says "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links" (OpenAI).

Which AI bots ignore robots.txt?

By their own docs, ChatGPT-User, Perplexity-User and Meta-ExternalFetcher may not follow it, because a person asked for the fetch (OpenAI; Perplexity; Meta).

Does robots.txt stop Meta crawling under its new terms?

Meta's crawler page shows how to disallow Meta-ExternalAgent and Meta-ExternalAds, while Meta-ExternalFetcher may bypass robots.txt (Meta). The Commercial Terms that take effect on November 20, 2026 add an authorization to crawl advertisers' domains and "create test credentials," and do not mention robots.txt (Meta).

Can robots.txt keep patient pages private?

No. Google says robots.txt "cannot enforce crawler behavior" and recommends "password-protecting private files on your server" (Google).

Does Curve Compliance sign a BAA?

Yes. Curve Compliance signs a BAA on every plan, checks every event for PHI-like patterns and strips PHI before data reaches an ad platform. Curve's team does the setup, typically live in about a week.

Talk to Curve Compliance

Book a call. See which pixels and trackers your pages load today and how Curve Compliance sends conversions server-side with PHI removed before anything is sent to an ad platform, under the BAA it signs on every plan. Curve's team does the setup, typically live in about a week. Book a call with Curve.