Which AI Bots Read Your Website?

Tony CarnevaleNovember 21, 202612 min read
Bots that ran the page scriptTracked websites · 90 days
BotShare
Meta AIMeta · Reading it to train a model40%
Google Ads botGoogle · Checking your ads16%
Meta-WebIndexerMeta · Indexing it for AI search8.6%
Sogou SpiderSogou · Search engine3.3%
LighthouseGoogle · Uptime or speed check0.6%
OAI-AdsBotOpenAI · Checking your ads0.3%
AhrefsBotAhrefs · SEO tool0.2%
Google-AgentGoogle · Looked your site up for someone0.1%
GPTBot, ClaudeBot, PerplexityBotRead pages without running scriptsNot counted

Venbit Intelligence Research, 2026. Share is of all bot visits that ran the page script.

The named bots that ran the page script on tracked websites in 90 days, with each one's share of all bot visits. The text-only AI crawlers have no count.

The short answer

In a recent Venbit Intelligence Research study, Meta's crawler, Meta AI, made 40% of the bot visits recorded on tracked small business websites, and reached more than nine in ten of them. GPTBot, ClaudeBot and PerplexityBot read pages without running scripts, so no analytics tool counts them and only a server log shows their reads. Your robots.txt decides which of them may read you.

What to know

  • Meta AI, Meta's training crawler, made about 40% of the bot visits on tracked websites over 90 days, and reached more than nine in ten of them.
  • Add Meta-WebIndexer and Meta's two crawlers come to 48% of every bot visit our script recorded.
  • GPTBot, ClaudeBot and PerplexityBot read a page's text without running its scripts, so GA4, Venbit Intelligence and every other script-based tool see none of them.
  • Google's ad checker came second with 16% of bot visits, reading the pages that ads send people to.
  • Training crawlers and the crawlers that fetch a page for a person asking a question have separate names, and robots.txt can block one group and keep the other.
  • AI assistants sent visitors to more than nine in ten tracked websites in the same 90 days, which is the cost of shutting every AI crawler out.

On tracked small business websites, the AI bot that reads the most is Meta AI, Meta's crawler, with about 40% of all bot visits. That is what Venbit Intelligence Research found when it named every bot that ran the page script on those websites over 90 days. The names people ask about, GPTBot and ClaudeBot, are missing from that count for a technical reason this page explains. To see which crawlers your own site lets in, enter your domain below.

Check your site

See which AI crawlers your robots.txt lets in

Enter your domain. The check reads your robots.txt once and shows, for 14 AI and search crawlers, whether each one may read your home page.

Checks 14 crawlers from OpenAI, Anthropic, Google, Meta, Microsoft, Perplexity, Apple, Amazon, ByteDance and Common Crawl.

Reads robots.txt once and judges the home page for each crawler by RFC 9309: its own rules first, the rules for every crawler when it has none, and the longest matching rule wins. A firewall or a host's bot setting can turn a crawler away before it reads the file, and this check cannot see that. Meta AI figure: Venbit Intelligence Research, tracked websites, autumn 2026. The domain you enter goes to our server for this one read and is not stored.

The rows are sorted by the job each crawler does. A crawler that opens your page because someone asked ChatGPT about you does different work from one collecting pages to train a model, and robots.txt lets you answer each by name. A green row means the file allows that crawler in. It says nothing about whether the crawler has come yet.

Only one crawler on that list leaves a trace in analytics. Meta AI runs a page's scripts the way a browser does, so our tracker records it, which is why its row carries a line from the research. The others read your text and leave without running anything.

Meta AI made 40% of bot visits

That surprised me. I expected OpenAI or Google at the top of any list of AI crawlers, because those are the names in the news. On these websites Meta outread every other AI company we can see by a wide margin, and Meta-WebIndexer turned up on nearly every tracked website, the widest reach of any crawler that names itself.

Share of bot visits by name, 90 days

  • Meta AI39.8%

    Meta, reading pages to train a model.

  • Google Ads bot16.4%

    Google, checking ad landing pages.

  • Meta-WebIndexer8.6%

    Meta, indexing pages for AI search.

  • Sogou Spider3.3%

    Sogou, a search engine.

  • Lighthouse0.6%

    Google, a speed test.

  • OAI-AdsBot0.3%

    OpenAI, checking ads.

  • AhrefsBot0.2%

    Ahrefs, an SEO tool.

  • Google-Agent0.1%

    Google, an agent acting for a person.

Venbit Intelligence Research, bots that ran the page script. Named crawlers seen on enough tracked websites to describe; unnamed scripts and fleets left out. Shares are of all bot visits.

Second place goes to Google's ad checker, which loads the pages that ads send people to. It reached about four in five tracked websites, and none of its visits counted as a person. The guide on why ad clicks do not match website visits covers what else separates a click from a visit.

Further down sit Sogou Spider, the crawler of a Chinese search engine, and two of OpenAI's and Google's newer agents. OAI-AdsBot checks ads. Google-Agent is the name Google's agents use when they act on a person's request. The guide to traffic from China and Singapore explains the wave of bots that arrives from those two countries.

CrawlerPages read in two weeks
Meta AI10,399
Google Ads bot3,845
Meta-WebIndexer2,911
Sogou Spider793
Lighthouse184
WebPageTest166
AhrefsBot5
Pages read by crawlers that ran the page script, a two-week count in November 2026. Venbit Intelligence Research, tracked websites.

The fortnight shows how unevenly Meta AI reads. It loaded 10,399 pages in those 14 days on fewer than half the tracked websites, although it reached more than nine in ten of them over the full 90 days. Meta-WebIndexer read far fewer pages and spread them across nearly every tracked website. One crawler goes deep on some sites and skips the rest for weeks, and the other checks in almost everywhere. OpenAI's search crawler ran our script too few times in the fortnight for us to publish a count.

GPTBot and ClaudeBot do not run your scripts

Every analytics tool you have heard of, Google Analytics included, counts a visit when a small script on the page runs in the visitor's browser and reports back. Venbit Intelligence works the same way. A crawler that downloads the page's text and leaves without running that script is invisible to all of them, and GPTBot, ClaudeBot and PerplexityBot read that way. Vercel tested this on its own network in December 2024 and found that none of the major AI crawlers rendered JavaScript.

AI searchLast 30 days
AIVisitsLeadsPages read
ChatGPTOpenAI · last seen 3m ago26▲ 2500%00
Visitors ChatGPT sent
Seattle, WALanded on Website Maintenance And Security page · Windows3m ago
Portland, ORComputer3d ago
The newest 2 of 26 visits.
What OpenAI read

None of OpenAI's crawlers loaded your pages in this period.

Meta AIMeta · last seen 1h ago0090▲ 1025%
Other botsNot AI assistants · 4 bots——18

GPTBot, ClaudeBot and most other AI crawlers read pages without running scripts, so the pages read here are the ones loaded the way a browser does.

The AI search card on the demo site with ChatGPT opened: it sent 26 visits, and What OpenAI read stays empty because GPTBot reads without running the script.

That empty line under What OpenAI read is the card telling the truth. ChatGPT sends people, and OpenAI's crawlers leave no mark on the page's script, so the card names the people and states that it saw no reads. We publish no count for GPTBot or ClaudeBot on this page for the same reason. A figure from a script-based tool would only cover crawlers that render pages, and those two do not.

Meta AI is visible because it loads pages like a browser and runs what is on them. Google's ad checker does the same. Those two rendering crawlers account for most of the named bot visits in our data, while the text-only crawlers sit outside it entirely.

  • Analytics scripts

    GA4 and Venbit Intelligence see crawlers that run the page, such as Meta AI and Google's ad checker. Text-only crawlers never appear.

  • Your server's access log

    Every request is a line, GPTBot and ClaudeBot included. This is where their reads show, with the page and the time of each one.

  • Search Console

    The Crawl stats report, under Settings, lists Googlebot's requests to your site by day, by response and by file type.

For a count of GPTBot on your own site, ask your host for the raw access log and search it for the name. Hosting control panels such as cPanel offer it for download. The robots.txt check above answers the other half of the question, which is whether those crawlers are allowed in at all.

AI crawlers train, index or fetch for a person

AI companies run more than one crawler, and each does a different job. Venbit Intelligence labels every AI crawler with one of four: reading the site to train a model, indexing it for the company's own AI search, looking it up for someone who asked, or checking an ad. Robots.txt follows the same split, because each crawler answers to its own name in the file.

Name in robots.txtRun byJob
ChatGPT-UserOpenAIOpens a page when someone asks ChatGPT
Claude-UserAnthropicOpens a page when someone asks Claude
OAI-SearchBotOpenAILists pages in ChatGPT search
PerplexityBotPerplexityBuilds Perplexity's search index
GooglebotGoogleGoogle Search
BingbotMicrosoftBing search
GPTBotOpenAICollects pages to train OpenAI's models
ClaudeBotAnthropicCollects pages to train Claude
meta-externalagentMetaMeta AI's crawler, labelled as training
CCBotCommon CrawlBuilds an open web archive
BytespiderByteDanceByteDance's crawler, labelled as training
AmazonbotAmazonCollects pages for training
Google-ExtendedGoogleNo crawler: a switch for Gemini training
Applebot-ExtendedAppleNo crawler: a switch for Apple's model training
The crawlers the check above reads, by job, as Venbit Intelligence labels them and as their owners describe them.

OpenAI is explicit that its settings are separate. Its crawler page says a Disallow for GPTBot means a site's content should not be used to train its models, that OAI-SearchBot decides whether a site appears in ChatGPT search, and that each setting is independent of the others. The same page says ChatGPT-User acts for a person, so robots.txt rules may not apply to it.

Google draws the same line with a token. Google-Extended never arrives as a visitor: Google says it has no user agent string of its own and works only as a control in robots.txt, deciding whether Gemini may train on pages Google has crawled. Google also states that it does not affect inclusion or ranking in Google Search. Apple says Applebot-Extended does not crawl pages either, so both rows in the tool are switches.

Blocking Googlebot to stay out of AI training takes you out of Google Search with it. Google-Extended is the switch for Gemini.

Crawl-to-referral ratios come from a big network

The statistic quoted most about AI crawlers is a ratio of pages crawled to visitors sent. It traces to Cloudflare, which published it in July 2025 from requests across its network. For the week of 19 to 26 June 2025, Cloudflare counted about 70,900 HTML page requests from Anthropic's crawlers for every visit a Claude link sent, and ratios as low as 0.1 to 1 for Mistral.

Three details matter before you apply that number to a small site. It counts requests across Cloudflare's whole network, not one website. It also combines each company's crawlers, training and live fetches together. Cloudflare notes that Claude's app sends no referrer, so visits from it go uncounted and the ratio comes out higher than the truth by an amount nobody knows.

48%
of bot visits came from Meta's two crawlers
93%
of AI assistant visits came from ChatGPT
92%
of tracked websites got a visit from an AI assistant
Venbit Intelligence Research, tracked websites, 90 days

We cannot build Cloudflare's ratio for OpenAI or Anthropic from a script, because their crawlers never run it. Meta is the one company where both halves are visible to us, and the result is lopsided. Its crawler made 58,428 visits, while visits from the Meta AI assistant were too few, on too few websites, for us to publish a figure. ChatGPT, whose crawler we cannot see, sent visitors to more than nine in ten of the tracked websites, and the guide on how much traffic comes from ChatGPT breaks that down by assistant.

Decide whether to block AI crawlers

The case for letting them in is that a crawler reading your pages is how an assistant learns what you do. AI assistants sent visitors to more than nine in ten of the websites in our research, and ChatGPT visitors turn into leads at a rate worth comparing with Google's. A plumber who blocks every AI crawler leaves ChatGPT to describe the business from whatever other sites say about it, if it mentions the business at all.

Allow AI crawlers

  • Assistants can open your pages when someone asks about a service you sell.
  • ChatGPT search and Perplexity can list your pages and link to them.
  • Your own words reach the answer in place of a directory's summary of you.

Block AI crawlers

  • Training crawlers are asked to leave your pages out of future models.
  • Fewer automated requests reach your server, which your host can measure.
  • Assistants have to describe you from other sites, or leave you out.

Robots.txt has a middle setting built in. You can keep the crawlers that answer questions and run AI search, and say no to the ones that collect pages for training. Here is how I would set it up on a small business site.

  1. 1

    Run the check above

    Enter your domain and read the four groups. A site with no robots.txt allows everything, which is where a new website starts.

  2. 2

    Keep the answer and search crawlers open

    ChatGPT-User, Claude-User, OAI-SearchBot and PerplexityBot are the ones that put your pages in front of a person who asked. Leave them allowed if you want assistants to quote you and link to you.

  3. 3

    Block training crawlers by name

    Add a group for each one you want out: the line User-agent: GPTBot, then Disallow: / under it. Repeat it for ClaudeBot, Amazonbot, meta-externalagent and the other names in the check's training group, such as CCBot and Bytespider, and use Google-Extended and Applebot-Extended for Google's and Apple's training.

  4. 4

    Leave Googlebot and Bingbot alone

    Blocking either one takes you out of that search engine. Google-Extended is the switch for Gemini training, and Google says it has no effect on Search.

  5. 5

    Check the firewall too

    Robots.txt is a request, and the standard says its rules are not a form of access authorization. A host's bot setting or a firewall rule can turn a crawler away before it reads the file, so look there as well if your server log shows a crawler that the file allows never arriving.

See which AI reads your own site

Venbit Intelligence puts all of this on one card. The AI search section of the Analytics tab lists each assistant by name with the visits and leads it sent, beside the period before. Open a row to see the people that assistant sent, the pages they landed on, and what that company's crawlers loaded, each crawler labelled with what it came for.

AI searchLast 30 days
AIVisitsLeadsPages read
ChatGPTOpenAI · last seen Sep 27, 202626▲ 2500%00
Meta AIMeta · last seen 1h ago0090▲ 1025%
Visitors Meta AI sent

Nobody clicked through from Meta AI in this period.

What Meta read
Meta AIReading it to train a model90 pages1h ago
Other botsNot AI assistants · 4 bots——18

GPTBot, ClaudeBot and most other AI crawlers read pages without running scripts, so the pages read here are the ones loaded the way a browser does.

The AI search card on the demo site with Meta AI opened: nobody clicked through from Meta AI, and its crawler read 90 pages to train a model.

Bots that are not AI assistants sit in their own row at the foot of the card: search engines, SEO tools, ad checkers and scripts. None of them counts toward visits or leads. The Feed carries a Bot filter post that groups the bots kept out of your numbers by kind, and the guide on how much of your traffic is bots shows how each one is caught.

Bot filter · Kept out of your numbers

Last 30 days · Sep 21 to Oct 21 · Worth knowing

6,447 visits from bots were kept out of your numbers.

That is 43% of everything that reached your site. Uptime monitors made 4,233, and AI crawlers made 949. Every figure on this page leaves these out.

8,510people · 57%

6,447bots · 43%

Uptime monitors4,233

AI crawlers949

Search engines and ad checkers853

SEO tools338

Link previews74

How we know
The Feed's Bot filter post on the demo site: 6,447 bot visits kept out of the count in 30 days, 949 of them from AI crawlers.

If you work in Claude or ChatGPT, Venbit MCP hands the assistant the same AI search report: the assistants that sent people, the crawlers that read the site with the pages each read, and the other bots. You can ask it in plain words which AI has been reading you this month.

How we measured this

Every figure on this page comes from Venbit Intelligence Research: small business websites tracked in the Venbit Approved System, from 21 August to 18 November 2026. A bot visit is a visit where something loaded the page and ran our script, and the tracker judged it automation and kept it out of the owner's numbers. The crawler's name comes from the user agent it sent, read at the moment the visit was recorded, and the page counts cover the last two weeks of that window.

The limits matter more on this topic than on most. These counts cover only crawlers that run scripts, so they say nothing about how many pages GPTBot, ClaudeBot or Googlebot read on these sites. A crawler seen on too few tracked websites to describe gets no figure here, which is why OpenAI's search crawler has none. Treat the numbers as a careful sample of small business websites and not a census.

See which AI reads your own site

Venbit Intelligence names every crawler that loads your pages and what it came for, beside the assistants that sent you people and leads. One line to install, free for 14 days, no card needed.

In Venbit Intelligence

More website analytics guides

Every question in this series, from Website Analytics for Small Business Owners.

Leads inbox4
Ad performance4
SEO tracking3
Visitor behaviour4
Feed and notifications3
Goals and measurement4
Privacy, AI and teams3
Venbit Intelligence Research19

Questions & Answers

  • Your server's access log is the only place to find out. GPTBot reads a page's text without running its scripts, so GA4 and Venbit Intelligence never record it. That is why GPTBot is missing from the bots that ran the script on our tracked websites. The check on this page shows whether your robots.txt allows it.

  • Meta's. On tracked websites over 90 days, Meta AI made about 40% of the bot visits we recorded, and Meta-WebIndexer another 8.6%. These shares cover crawlers that run page scripts, so GPTBot and ClaudeBot, which do not, are missing from them.

  • Block the ones you object to, by name, and keep the rest. Training crawlers such as GPTBot and ClaudeBot can be refused without touching ChatGPT-User, Claude-User, OAI-SearchBot or PerplexityBot, which open and list pages for people asking questions. AI assistants sent visitors to more than nine in ten of our tracked websites, so shutting all of them out has a cost.

  • No. OpenAI says each of its settings is independent. GPTBot covers model training, OAI-SearchBot decides whether your pages appear in ChatGPT search, and ChatGPT-User opens a page for a person, where OpenAI says robots.txt rules may not apply. ChatGPT sent 93% of the AI assistant visits our tracked websites received over 90 days.

  • Google Analytics counts a visit when its script runs in a browser, and most AI crawlers download the text and leave without running scripts. GA4 also filters known bots out automatically. The bots a script can see are the ones that render pages: on our tracked websites those made 39% of recorded visits, with Meta AI the largest.

  • It is Meta's AI crawler, the one Venbit Intelligence names Meta AI and labels as reading pages to train a model. GPTBot skips a page's scripts and this crawler runs them, so analytics can see it. It made about 40% of the bot visits on our tracked websites over 90 days, more than any other bot. Block it with User-agent: meta-externalagent.

More guides with a tool

See all 23 guides