Google Will Never Read Your llms.txt. Build It Anyway

Lighthouse shipped a new agentic audit a few weeks ago. One of the things it checks is whether you have an llms.txt file.

The same month, Google's own guidance on AI search listed llms.txt under the things you don't need to do.

One company. Telling you to build the file, and telling you not to bother with it. Same month. No wonder everyone's confused.

The confusion clears the moment you stop treating llms.txt as a search thing. It was never a search thing.

Here's where I land. Google and AI search will never use llms.txt for ranking or citations. Not now, not later.

But agentic tools have a real reason to read it today. Build the file expecting the first group to care, and you've built it for the wrong audience.

Why Google will never use it for search

Two reasons. Both are baked into how search works.

First, it's the easiest cloaking vector to come along in years.

llms.txt is a separate file you fully control. It sits next to the HTML real users see. Nothing stops you from writing a polished, keyword-stuffed version for the machine and serving something completely different to humans.

That's cloaking with markdown syntax.

Google spent two decades learning not to trust anything a site owner hands it directly. A file like this would get gamed within a week.

Second, even if Google wanted to use it, it would have to verify every claim against the live page anyway.

The moment you're checking llms.txt against the real HTML, the file has bought you nothing. It's an extra fetch, extra parsing, extra infrastructure across billions of pages. All to produce a claim you then re-confirm against the source you already crawled.

Nobody bolts a costly extra step onto the crawl to read a file they can't trust.

You don't have to trust me on this. The crawlers you'd most want reading it are the ones ignoring it.

Googlebot and Bingbot don't use it for ranking, AI answers, or anything else. The AI crawlers you'd hope would lean on it – GPTBot, ClaudeBot, PerplexityBot – barely request it.

Grep your server logs for llms.txt and see for yourself. The crawlers you wish cared about it don't.

This file is not how you win AI search. Nothing about it is.

Who it's actually for

Agentic tools. The thing it was designed for from day one.

Claude Code, Codex, Cursor, Windsurf, Antigravity. None of these have an index like Google or Bing sitting behind them.

Point one at a site it doesn't know, and it has to work out the structure on the fly. Parse HTML full of nav, ads, and JavaScript. Burn tokens doing it. Half the time it grabs the wrong section or misreads the layout.

A clean llms.txt hands the tool a map. Here's what exists, here's where it lives, here's the plain markdown version.

It saves time and tokens. It cuts the parsing mistakes that come from reading messy HTML. That's the entire value, and it's a real one.

The API case is the clearest one

If you ship an API, this is where llms.txt most obviously earns its place.

Publish a curated index of your endpoints and reference docs. Now a tool can learn what your API does and load only the page it needs, instead of scraping your whole site to find it.

My Claude Code knows your endpoints immediately. It doesn't crawl twelve pages of landing copy to reach the auth docs.

This isn't theory. The big docs sites developers point these tools at already do it.

They maintain llms.txt for their own developer docs because their users feed those files to coding agents all day. That's the file being used as designed. An agent reading a docs site to do a job, not a search crawler reading yours to rank you.

It's the clearest example, not the only one. Any site an agent has to read and navigate can benefit from a map. Docs and APIs are just where the payoff is obvious today.

Skip llms-full.txt for most sites

llms-full.txt concatenates your entire documentation into one giant markdown dump.

That makes sense for a docs-heavy or API product, where an agent benefits from ingesting everything at once and the whole thing fits in a context window.

For a normal site, it's a maintenance burden and a duplicate-content liability. Serve a full-text version that drifts from your HTML and you're back in the cloaking problem.

If your API docs aren't enormous, llms-full.txt can be worth it. If you're a regular site with no developer audience, I don't see the point.

How to handle it

If you've got developer docs or an API, ship a curated llms.txt:

• Keep it small – well under 50 KB.
• Curate it – don't dump your sitemap into it.
• Keep it accurate and maintained.
• Add a noindex header so the file doesn't end up ranking on its own.
• Return it cleanly – a broken response is the only thing the Lighthouse audit penalizes. A missing file just reads as not applicable.

The further your site sits from docs and APIs, the weaker the case. A brochure site no agent is navigating doesn't need one.

But this isn't developer-only. If agents have to read and act on your content, a clean map still helps.

The pattern, again

Same conclusion as everywhere else in technical SEO right now. No magic file makes Google trust you more. No text file wins you AI citations.

The fundamentals do the work. A file you wrote about yourself was never going to replace them.

llms.txt is fine. It's cheap, it's not hard, and it helps the tools that need a map.

Just build it for the agent reading your site, not for the search engine that was never going to read it.