llms.txt: Do AI Bots Actually Read It? 13 Days of Logs

👁 117 reads

The site has an llms.txt file: 230 lines, 170 links to key pages, generated automatically by WordPress. I reviewed 13 days of server logs, from October 24 to November 5, 2026: 1,832,096 requests in total. During that period, bots identifying themselves as OpenAI, Anthropic and Perplexity made more than 61,000 requests. How many times did any of them fetch llms.txt? Zero.

I’m writing this because llms.txt is often presented as the first step in “getting recommended by ChatGPT.” My logs suggest otherwise. Here are the numbers, how I counted them and the commands you can use to check your own logs.

What is llms.txt?

It is a proposal by Jeremy Howard, published on September 3, 2024, at llmstxt.org. The file sits in a site’s root directory at /llms.txt and is plain Markdown text. The only required element is an H1 heading with the site’s name. It can be followed by a short description in a blockquote and H2 sections with links to pages a language model should read first. The idea is to provide a shortcut: instead of working through menus, ads and scripts, a model gets an easy-to-process contents list.

My file starts with “# [site name]” and includes a site description, an “About the source” section with the author and methodology, and groups of links to tools, live service status, prices and comparisons. There is also an llms-full.txt file containing the full text. It is 7.34 MB and 80,795 lines long.

llms.txt is not a standard from any organization. It is not in the documentation for OpenAI, Anthropic or Perplexity bots. Google has addressed it directly, as I explain below.

What the logs showed

The server records every request, including its IP address, date, path, response code and browser signature, known as the user agent. I extracted all requests for /llms.txt, /llms-full.txt and /robots.txt from the daily archives, then counted AI bot requests to the entire site separately.

There were 95 requests for llms.txt and 22 for llms-full.txt. SEOJuice, an SEO auditing tool, fetched the file most often, with 26 requests. Next were agentwebmap-bot (15), the code-docs-corpus dataset crawler (7), Googlebot (4), several site auditors and a dozen or so regular browsers, probably people in the industry checking what the file looks like.

The raw results also showed one request each from GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, along with several other bots. That looked promising until I checked the timestamps. They all came from one Polish IP address over 13 seconds, on October 30 between 00:03:13 and 00:03:26, each with a different bot signature. It was an access test I ran while tidying up the file generator. A real GPTBot does not come from a home internet connection in Poland.

[30/Oct/2026:00:03:13] "GET /llms.txt" 200 "... GPTBot/1.2; +https://openai.com/gptbot"
[30/Oct/2026:00:03:14] "GET /llms.txt" 200 "... OAI-SearchBot/1.0; +https://openai.com/searchbot"
[30/Oct/2026:00:03:16] "GET /llms.txt" 200 "... ClaudeBot/1.0; +claudebot@anthropic.com"
[30/Oct/2026:00:03:18] "GET /llms.txt" 200 "... PerplexityBot/1.0; +https://perplexity.ai/..."
(18 signatures, one IP address, 13 seconds)

After excluding that burst, no OpenAI, Anthropic or Perplexity bot fetched llms.txt. I also checked the other way around: I took the official IP address lists published by OpenAI and Perplexity and filtered requests for llms.txt. Not one request came from those addresses.

AI bots visit the site, but not llms.txt

It’s not that AI bots don’t know about the site. Over the same 13 days:

Bot	What it does	Requests to the site	Requests for llms.txt
OAI-SearchBot	ChatGPT search index	19,921*	0
ClaudeBot	Anthropic data collection	18,621	0
PerplexityBot	Perplexity search index	11,289*	0
GPTBot	Data for training OpenAI models	4,184	0
ChatGPT-User	Visits a site at a chat user’s request	2,219*	0
Claude-User	The equivalent in Claude	2,563	0
Googlebot	Google index	137,287	4

*Only requests from official IP ranges published by OpenAI and Perplexity. ClaudeBot, Claude-User, GPTBot and Googlebot counts are based on the bot signature, without IP verification. Period: October 24-November 5, 2026; server logs.

OAI-SearchBot visited 3,484 different URLs, and PerplexityBot visited 3,774. Both discover pages through sitemaps and links, just as Googlebot does. Why would they need a list in llms.txt if they can read the original pages?

One thing surprised me. Of OAI-SearchBot’s 19,921 requests, only 5,259 were for pages. The rest were for CSS files, scripts and 7,364 requests to my own analytics counters, which run in the browser. The takeaway: ChatGPT’s search crawler renders pages with JavaScript. If you track clicks with your own script, filter it out or it will inflate your numbers.

Which bots fetch pages for chat users?

ChatGPT-User is the most interesting. This signature appears when someone asks a question in ChatGPT and the model visits a site to answer it. Over 13 days, there were 2,219 such requests from OpenAI IP addresses, covering 790 different pages. The most-visited pages, aside from the homepage (388 requests), were: election dates (56), an AI calendar (22), a ChatGPT price page (18) and a ChatGPT Plus versus Pro comparison (14).

A pattern stands out: chatbots fetch pages when a question involves specific information that changes, such as dates, prices or plan comparisons. I saw the same thing in Bing data discussed in an article about how Copilot chooses sources. The model doesn’t need my summary. It needs a page where the current figure appears near the top, along with the date it was checked.

Watch out for fake bots

Perplexity-User made 540 requests in the logs, none from Perplexity IP addresses. All were looking for files such as /.env, /.env.production or /config/gcp-credentials.json. These are scanners impersonating AI bots, hoping administrators will let them through without checking. Of 2,782 requests signed as ChatGPT-User, 563 also did not come from OpenAI.

Practical advice: don’t trust a request just because its user agent says “GPTBot.” Check the IP address against the official list. If you can’t verify it, treat the signature as a claim, not proof.

What Google says

In its guide to optimizing for AI search, Google says you don’t need to create AI-specific files or Markdown to appear in search, including generative features, because Google Search does not use them. It addresses llms.txt separately: you can maintain one, but it won’t help or hurt your visibility in Google because Google ignores it. That passage was added to the documentation in June 2026. A year earlier, Google’s Gary Illyes had said the same thing publicly.

My four Googlebot requests for llms.txt don’t change that. Googlebot fetches any URL it finds in links, so it found this one too.

My take: I’m keeping the file, but I won’t pay for it

I’m not removing llms.txt. It’s generated automatically, causes no harm, and is read by SEO tools and a few smaller crawlers. If OpenAI or Anthropic ever start using it, it will be ready.

But if someone sells you an “llms.txt implementation” as a way to get recommended in ChatGPT, you’re overpaying. In my logs, the file is for auditors, not models. An hour spent making sure your page has an up-to-date price, date and answer in its first paragraph will do more.

The logs suggest these are the things that matter:

• Let the right bots in. OAI-SearchBot and PerplexityBot determine whether you appear in chat search engines. If you block them in robots.txt or your hosting firewall, your site won’t be in their indexes. Whether to allow GPTBot, which collects training data, is a separate decision.
• Keep your site fast and accessible. ChatGPT-User visits on demand during a conversation. If your page is slow or shows a “prove you’re human” screen, it won’t make it into the answer.
• Publish information that changes. The pages chatbots fetched most often answered questions about dates, prices or differences between plans. Models don’t have that information in memory, so they need to retrieve it.
• Measure instead of guessing. Server logs are free and show which pages bots read, including pages fetched at users’ request.

How to check your own logs

You’ll need raw access logs from your hosting provider. They’re usually in a logs directory in the control panel or available as downloadable archives. These three commands are enough to get started:

# 1. Who fetches llms.txt (IP address, date, signature)
zcat access.log.gz | grep '"GET /llms' | awk '{print $1, $4, $NF}'

# 2. Count AI bot requests to the entire site
zcat access.log.gz | grep -c 'OAI-SearchBot'
zcat access.log.gz | grep -c 'ChatGPT-User'

# 3. Which pages ChatGPT reads at users' request
zcat access.log.gz | grep 'ChatGPT-User' | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Finally, compare IP addresses with OpenAI’s published lists (the chatgpt-user.json, searchbot.json and gptbot.json files on openai.com) and Perplexity’s list. Without this step, you’ll count scanners too, as I did in my first pass.

Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →

Sources and date checked

Logs and files checked November 6, 2026.

• Server access logs, October 24-November 5, 2026 (1,832,096 requests); my analysis
• llmstxt.org - the llms.txt proposal by Jeremy Howard, September 3, 2024
• Google Search Central - guide to optimizing for AI search, including llms.txt
• OpenAI - information about OAI-SearchBot, GPTBot and ChatGPT-User, including IP address lists
• Perplexity - information about PerplexityBot and Perplexity-User
• Anthropic - information about ClaudeBot, Claude-User and Claude-SearchBot
• Search Engine Roundtable, July 24, 2025 - Gary Illyes: Google does not support llms.txt

Kinetyra