Venbit Intelligence Research, 2026. Share is of all bot visits that ran the page script.
The short answer
In a recent Venbit Intelligence Research study, Meta's crawler, Meta AI, made 40% of the bot visits recorded on tracked small business websites, and reached more than nine in ten of them. GPTBot, ClaudeBot and PerplexityBot read pages without running scripts, so no analytics tool counts them and only a server log shows their reads. Your robots.txt decides which of them may read you.
What to know
- Meta AI, Meta's training crawler, made about 40% of the bot visits on tracked websites over 90 days, and reached more than nine in ten of them.
- Add Meta-WebIndexer and Meta's two crawlers come to 48% of every bot visit our script recorded.
- GPTBot, ClaudeBot and PerplexityBot read a page's text without running its scripts, so GA4, Venbit Intelligence and every other script-based tool see none of them.
- Google's ad checker came second with 16% of bot visits, reading the pages that ads send people to.
- Training crawlers and the crawlers that fetch a page for a person asking a question have separate names, and robots.txt can block one group and keep the other.
- AI assistants sent visitors to more than nine in ten tracked websites in the same 90 days, which is the cost of shutting every AI crawler out.
On tracked small business websites, the AI bot that reads the most is Meta AI, Meta's crawler, with about 40% of all bot visits. That is what Venbit Intelligence Research found when it named every bot that ran the page script on those websites over 90 days. The names people ask about, GPTBot and ClaudeBot, are missing from that count for a technical reason this page explains. To see which crawlers your own site lets in, enter your domain below.
Check your site
See which AI crawlers your robots.txt lets in
Enter your domain. The check reads your robots.txt once and shows, for 14 AI and search crawlers, whether each one may read your home page.
Checks 14 crawlers from OpenAI, Anthropic, Google, Meta, Microsoft, Perplexity, Apple, Amazon, ByteDance and Common Crawl.
Reads robots.txt once and judges the home page for each crawler by RFC 9309: its own rules first, the rules for every crawler when it has none, and the longest matching rule wins. A firewall or a host's bot setting can turn a crawler away before it reads the file, and this check cannot see that. Meta AI figure: Venbit Intelligence Research, tracked websites, autumn 2026. The domain you enter goes to our server for this one read and is not stored.
The rows are sorted by the job each crawler does. A crawler that opens your page because someone asked ChatGPT about you does different work from one collecting pages to train a model, and robots.txt lets you answer each by name. A green row means the file allows that crawler in. It says nothing about whether the crawler has come yet.
Only one crawler on that list leaves a trace in analytics. Meta AI runs a page's scripts the way a browser does, so our tracker records it, which is why its row carries a line from the research. The others read your text and leave without running anything.
Meta AI made 40% of bot visits
That surprised me. I expected OpenAI or Google at the top of any list of AI crawlers, because those are the names in the news. On these websites Meta outread every other AI company we can see by a wide margin, and Meta-WebIndexer turned up on nearly every tracked website, the widest reach of any crawler that names itself.
Share of bot visits by name, 90 days
- Meta AI39.8%
Meta, reading pages to train a model.
- Google Ads bot16.4%
Google, checking ad landing pages.
- Meta-WebIndexer8.6%
Meta, indexing pages for AI search.
- Sogou Spider3.3%
Sogou, a search engine.
- Lighthouse0.6%
Google, a speed test.
- OAI-AdsBot0.3%
OpenAI, checking ads.
- AhrefsBot0.2%
Ahrefs, an SEO tool.
- Google-Agent0.1%
Google, an agent acting for a person.
Second place goes to Google's ad checker, which loads the pages that ads send people to. It reached about four in five tracked websites, and none of its visits counted as a person. The guide on why ad clicks do not match website visits covers what else separates a click from a visit.
Further down sit Sogou Spider, the crawler of a Chinese search engine, and two of OpenAI's and Google's newer agents. OAI-AdsBot checks ads. Google-Agent is the name Google's agents use when they act on a person's request. The guide to traffic from China and Singapore explains the wave of bots that arrives from those two countries.
| Crawler | Pages read in two weeks |
|---|---|
| Meta AI | 10,399 |
| Google Ads bot | 3,845 |
| Meta-WebIndexer | 2,911 |
| Sogou Spider | 793 |
| Lighthouse | 184 |
| WebPageTest | 166 |
| AhrefsBot | 5 |
The fortnight shows how unevenly Meta AI reads. It loaded 10,399 pages in those 14 days on fewer than half the tracked websites, although it reached more than nine in ten of them over the full 90 days. Meta-WebIndexer read far fewer pages and spread them across nearly every tracked website. One crawler goes deep on some sites and skips the rest for weeks, and the other checks in almost everywhere. OpenAI's search crawler ran our script too few times in the fortnight for us to publish a count.
GPTBot and ClaudeBot do not run your scripts
Every analytics tool you have heard of, Google Analytics included, counts a visit when a small script on the page runs in the visitor's browser and reports back. Venbit Intelligence works the same way. A crawler that downloads the page's text and leaves without running that script is invisible to all of them, and GPTBot, ClaudeBot and PerplexityBot read that way. Vercel tested this on its own network in December 2024 and found that none of the major AI crawlers rendered JavaScript.
None of OpenAI's crawlers loaded your pages in this period.
GPTBot, ClaudeBot and most other AI crawlers read pages without running scripts, so the pages read here are the ones loaded the way a browser does.
That empty line under What OpenAI read is the card telling the truth. ChatGPT sends people, and OpenAI's crawlers leave no mark on the page's script, so the card names the people and states that it saw no reads. We publish no count for GPTBot or ClaudeBot on this page for the same reason. A figure from a script-based tool would only cover crawlers that render pages, and those two do not.
Meta AI is visible because it loads pages like a browser and runs what is on them. Google's ad checker does the same. Those two rendering crawlers account for most of the named bot visits in our data, while the text-only crawlers sit outside it entirely.
Analytics scripts
GA4 and Venbit Intelligence see crawlers that run the page, such as Meta AI and Google's ad checker. Text-only crawlers never appear.
Your server's access log
Every request is a line, GPTBot and ClaudeBot included. This is where their reads show, with the page and the time of each one.
Search Console
The Crawl stats report, under Settings, lists Googlebot's requests to your site by day, by response and by file type.
For a count of GPTBot on your own site, ask your host for the raw access log and search it for the name. Hosting control panels such as cPanel offer it for download. The robots.txt check above answers the other half of the question, which is whether those crawlers are allowed in at all.
AI crawlers train, index or fetch for a person
AI companies run more than one crawler, and each does a different job. Venbit Intelligence labels every AI crawler with one of four: reading the site to train a model, indexing it for the company's own AI search, looking it up for someone who asked, or checking an ad. Robots.txt follows the same split, because each crawler answers to its own name in the file.
| Name in robots.txt | Run by | Job |
|---|---|---|
| ChatGPT-User | OpenAI | Opens a page when someone asks ChatGPT |
| Claude-User | Anthropic | Opens a page when someone asks Claude |
| OAI-SearchBot | OpenAI | Lists pages in ChatGPT search |
| PerplexityBot | Perplexity | Builds Perplexity's search index |
| Googlebot | Google Search | |
| Bingbot | Microsoft | Bing search |
| GPTBot | OpenAI | Collects pages to train OpenAI's models |
| ClaudeBot | Anthropic | Collects pages to train Claude |
| meta-externalagent | Meta | Meta AI's crawler, labelled as training |
| CCBot | Common Crawl | Builds an open web archive |
| Bytespider | ByteDance | ByteDance's crawler, labelled as training |
| Amazonbot | Amazon | Collects pages for training |
| Google-Extended | No crawler: a switch for Gemini training | |
| Applebot-Extended | Apple | No crawler: a switch for Apple's model training |
OpenAI is explicit that its settings are separate. Its crawler page says a Disallow for GPTBot means a site's content should not be used to train its models, that OAI-SearchBot decides whether a site appears in ChatGPT search, and that each setting is independent of the others. The same page says ChatGPT-User acts for a person, so robots.txt rules may not apply to it.
Google draws the same line with a token. Google-Extended never arrives as a visitor: Google says it has no user agent string of its own and works only as a control in robots.txt, deciding whether Gemini may train on pages Google has crawled. Google also states that it does not affect inclusion or ranking in Google Search. Apple says Applebot-Extended does not crawl pages either, so both rows in the tool are switches.
Blocking Googlebot to stay out of AI training takes you out of Google Search with it. Google-Extended is the switch for Gemini.
Crawl-to-referral ratios come from a big network
The statistic quoted most about AI crawlers is a ratio of pages crawled to visitors sent. It traces to Cloudflare, which published it in July 2025 from requests across its network. For the week of 19 to 26 June 2025, Cloudflare counted about 70,900 HTML page requests from Anthropic's crawlers for every visit a Claude link sent, and ratios as low as 0.1 to 1 for Mistral.
Three details matter before you apply that number to a small site. It counts requests across Cloudflare's whole network, not one website. It also combines each company's crawlers, training and live fetches together. Cloudflare notes that Claude's app sends no referrer, so visits from it go uncounted and the ratio comes out higher than the truth by an amount nobody knows.
- 48%
- of bot visits came from Meta's two crawlers
- 93%
- of AI assistant visits came from ChatGPT
- 92%
- of tracked websites got a visit from an AI assistant
We cannot build Cloudflare's ratio for OpenAI or Anthropic from a script, because their crawlers never run it. Meta is the one company where both halves are visible to us, and the result is lopsided. Its crawler made 58,428 visits, while visits from the Meta AI assistant were too few, on too few websites, for us to publish a figure. ChatGPT, whose crawler we cannot see, sent visitors to more than nine in ten of the tracked websites, and the guide on how much traffic comes from ChatGPT breaks that down by assistant.
Decide whether to block AI crawlers
The case for letting them in is that a crawler reading your pages is how an assistant learns what you do. AI assistants sent visitors to more than nine in ten of the websites in our research, and ChatGPT visitors turn into leads at a rate worth comparing with Google's. A plumber who blocks every AI crawler leaves ChatGPT to describe the business from whatever other sites say about it, if it mentions the business at all.
Allow AI crawlers
- Assistants can open your pages when someone asks about a service you sell.
- ChatGPT search and Perplexity can list your pages and link to them.
- Your own words reach the answer in place of a directory's summary of you.
Block AI crawlers
- Training crawlers are asked to leave your pages out of future models.
- Fewer automated requests reach your server, which your host can measure.
- Assistants have to describe you from other sites, or leave you out.
Robots.txt has a middle setting built in. You can keep the crawlers that answer questions and run AI search, and say no to the ones that collect pages for training. Here is how I would set it up on a small business site.
- 1
Run the check above
Enter your domain and read the four groups. A site with no robots.txt allows everything, which is where a new website starts.
- 2
Keep the answer and search crawlers open
ChatGPT-User, Claude-User, OAI-SearchBot and PerplexityBot are the ones that put your pages in front of a person who asked. Leave them allowed if you want assistants to quote you and link to you.
- 3
Block training crawlers by name
Add a group for each one you want out: the line User-agent: GPTBot, then Disallow: / under it. Repeat it for ClaudeBot, Amazonbot, meta-externalagent and the other names in the check's training group, such as CCBot and Bytespider, and use Google-Extended and Applebot-Extended for Google's and Apple's training.
- 4
Leave Googlebot and Bingbot alone
Blocking either one takes you out of that search engine. Google-Extended is the switch for Gemini training, and Google says it has no effect on Search.
- 5
Check the firewall too
Robots.txt is a request, and the standard says its rules are not a form of access authorization. A host's bot setting or a firewall rule can turn a crawler away before it reads the file, so look there as well if your server log shows a crawler that the file allows never arriving.
See which AI reads your own site
Venbit Intelligence puts all of this on one card. The AI search section of the Analytics tab lists each assistant by name with the visits and leads it sent, beside the period before. Open a row to see the people that assistant sent, the pages they landed on, and what that company's crawlers loaded, each crawler labelled with what it came for.
Nobody clicked through from Meta AI in this period.
GPTBot, ClaudeBot and most other AI crawlers read pages without running scripts, so the pages read here are the ones loaded the way a browser does.
Bots that are not AI assistants sit in their own row at the foot of the card: search engines, SEO tools, ad checkers and scripts. None of them counts toward visits or leads. The Feed carries a Bot filter post that groups the bots kept out of your numbers by kind, and the guide on how much of your traffic is bots shows how each one is caught.
Last 30 days · Sep 21 to Oct 21 · Worth knowing
6,447 visits from bots were kept out of your numbers.
That is 43% of everything that reached your site. Uptime monitors made 4,233, and AI crawlers made 949. Every figure on this page leaves these out.
8,510people · 57%
6,447bots · 43%
Uptime monitors4,233
AI crawlers949
Search engines and ad checkers853
SEO tools338
Link previews74
If you work in Claude or ChatGPT, Venbit MCP hands the assistant the same AI search report: the assistants that sent people, the crawlers that read the site with the pages each read, and the other bots. You can ask it in plain words which AI has been reading you this month.
How we measured this
Every figure on this page comes from Venbit Intelligence Research: small business websites tracked in the Venbit Approved System, from 21 August to 18 November 2026. A bot visit is a visit where something loaded the page and ran our script, and the tracker judged it automation and kept it out of the owner's numbers. The crawler's name comes from the user agent it sent, read at the moment the visit was recorded, and the page counts cover the last two weeks of that window.
The limits matter more on this topic than on most. These counts cover only crawlers that run scripts, so they say nothing about how many pages GPTBot, ClaudeBot or Googlebot read on these sites. A crawler seen on too few tracked websites to describe gets no figure here, which is why OpenAI's search crawler has none. Treat the numbers as a careful sample of small business websites and not a census.
See which AI reads your own site
Venbit Intelligence names every crawler that loads your pages and what it came for, beside the assistants that sent you people and leads. One line to install, free for 14 days, no card needed.
In Venbit Intelligence
More website analytics guides
Every question in this series, from Website Analytics for Small Business Owners.
Leads inbox4
Ad performance4
SEO tracking3
Visitor behaviour4
Feed and notifications3
Goals and measurement4
Privacy, AI and teams3
Venbit Intelligence Research19
- Average Website Visitors per Month for a Small Business
- What Is a Good Conversion Rate for a Small Business Website?
- What Time of Day Do Website Leads Come In?
- Which Day of the Week Brings the Most Website Leads?
- How to Tell If a Website Lead Is Spam
- How Much Website Traffic Comes From ChatGPT?
- Do ChatGPT Visitors Convert Better Than Google Visitors?
- Which AI Bots Read Your Website? (you are here)
- Why Ad Clicks Don't Match Website Visits
- Mobile vs Desktop Website Traffic for Small Business
- How Long Do Website Visitors Take to Contact a Business?
- Average Time on Site for a Small Business Website
- Average Pages per Visit for a Small Business Website
- How Far Do Visitors Scroll on a Website?
- Why Is My Website Getting Traffic From China and Singapore?
- What Is Direct Traffic and Why Is It So High?
- Where Does Small Business Website Traffic Come From?
- How Many Visitors Reach Your Contact Page?
- New vs Returning Visitors for a Small Business Website
Tony Carnevale
Founder, Venbit
Tony started Venbit in Mill Creek, WA in 2011 and still does the work: web design, SEO, and the website analytics product the company builds and runs for its own clients. The numbers and screens in his articles come from that product, not from a content mill.
Sources
- Venbit Intelligence Research: websites tracked in the Venbit Approved System. Bot visits that ran the page script. Method and limits on the research page.
- Cloudflare, The crawl before the fall of referrals: understanding AI's impact on content providers (July 2025, crawl-to-referral ratios for 19 to 26 June 2025)
- OpenAI, crawlers and user agents (GPTBot, OAI-SearchBot, ChatGPT-User)
- Google Search Central, Google's common crawlers (Google-Extended)
- Apple Support, About Applebot (Applebot-Extended)
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-User)
- IETF, RFC 9309: Robots Exclusion Protocol (September 2022)
- Vercel, The rise of the AI crawler (December 2024)