llms.txt: What It Actually Does (and Why Google Says It Does Nothing)
llms.txt might be the most oversold file in SEO right now. Half the industry is shipping it as an "AI visibility hack"; Google has stated, in writing, that it does nothing for Search. Both camps quote real facts. The confusion comes from asking the wrong question, not "does llms.txt work?" but "work for whom?"
Here's the honest breakdown: what the file is, who actually reads it, what Google said, and whether the 15 minutes it takes to create one is worth it. (Spoiler: usually yes, but not for the reason the hype suggests.)
TL;DR
- llms.txt is a markdown file at your site root that gives AI systems a curated map of your most important pages. Proposed by Jeremy Howard (Answer.AI) in September 2024.
- Google confirmed in 2026 that it does not use llms.txt, not for rankings, not for AI Overviews. It is not a ranking factor and never was.
- No major LLM provider has committed to it, but in practice Perplexity and Claude retrieve it, and AI coding assistants (Cursor, Copilot) actively depend on it. Stripe, Cloudflare, and Anthropic all ship one.
- It is not a robots.txt for AI, it can't block crawlers or protect content. It's a menu, not a lock.
- Verdict: near-zero cost, small but real upside outside Google, essential for documentation sites. Do it in 15 minutes, then spend your real effort on content and structure.
What llms.txt Actually Is
The spec is deliberately simple: a plain markdown file at https://yourdomain.com/llms.txt containing an H1 with your site name, a one-line blockquote summary, and H2-grouped lists of your key pages with brief descriptions:
# Krawl
> Technical SEO implementation at scale, Core Web Vitals,
> migrations, redirects, and schema for large websites.
## Services
- [Core Web Vitals](https://krawl.digital/core-web-vitals/): Template-level performance fixes
- [Website Migrations](https://krawl.digital/website-migration/): SEO-safe replatforms and domain moves
## Guides
- [Migration Checklist](https://krawl.digital/website-migration/seo-checklist/): Pre- and post-launch stepsThe rationale: LLMs have small context windows and struggle with nav-heavy, script-heavy HTML. A curated markdown map lets an AI agent find your canonical explanation of something without crawling and parsing your whole site.
The spec has a companion, llms-full.txt, which inlines the complete content of your key pages into one giant markdown file. That's a documentation-site tool, ignore it unless you run docs.
What it is not: a robots.txt for AI. It doesn't block anything, doesn't control training-data usage, doesn't protect content. Crawl control still lives in robots.txt user-agent rules. llms.txt only offers, it never restricts.
What Google Actually Said
Google's Search Central documentation states plainly that you don't need AI-specific files, markup, or markdown to appear in Google Search, including its generative AI features. Googlers had signalled this informally for a year (John Mueller compared llms.txt to the keywords meta tag, a file that describes what you claim to be about, which is exactly why search engines learned to ignore such things), and the formal documentation ended the debate.
So, unambiguously: llms.txt will not improve your rankings, your AI Overviews visibility, or anything else inside Google's ecosystem. Anyone selling it as a Google SEO tactic is selling folklore.
Who Does Read It
Google isn't the whole picture, and this is where the file earns its keep:
| Consumer | Behaviour |
|---|---|
| Google Search / AI Overviews | Ignores it (confirmed) |
| Perplexity | Retrieves it when exploring sites |
| Claude (Anthropic) | Retrieves it; Anthropic publishes their own |
| ChatGPT / OpenAI | No commitment; browsing behaviour inconsistent |
| AI coding assistants (Cursor, Copilot, etc.) | Actively use it, the strongest real-world adoption |
| AI agents fetching your site for a user | The growth case: a clean map beats parsing your DOM |
Note the pattern: adoption is strongest where an AI tool needs to navigate and retrieve your content on demand, and weakest where content is ingested through traditional crawling (Google). Companies that live off developer documentation, Stripe, Cloudflare, Anthropic, ship llms.txt because their users' AI tools genuinely consume it.
Should You Add One? The Honest Decision Table
| Your situation | Verdict |
|---|---|
| Documentation or developer-focused site | Yes, clearly. Your users' AI tooling reads it today; add llms-full.txt too |
| SaaS, agency, B2B services site | Yes, 15 minutes for a small, growing channel (AI agents and answer engines) |
| Content site / publisher | Optional. Slight upside with Perplexity-style retrieval; zero Google effect |
| E-commerce at scale | Low priority. Product feeds and schema markup matter vastly more |
| Anyone expecting a rankings lift | No, that's not what this file does |
The cost-benefit is lopsided in a useful way: the cost is nearly zero (one static file, no code, no risk), so even a small probability of benefit clears the bar. Just keep the effort proportionate, llms.txt is the cheapest item on an AI-visibility checklist, not the strategy. The strategy is being the site AI assistants actually cite.
Implementation in 15 Minutes
- List your 10–20 highest-value pages, the ones you'd want an AI to cite or an agent to find. Not every page; it's a curation, not a sitemap.
- Write the markdown file per the format above: H1, blockquote summary, H2 sections, one-line description per link. Descriptions matter, they're what the LLM uses to pick a link. (Our free llms.txt generator builds a spec-compliant file in two minutes.)
- Host it at the root:
yourdomain.com/llms.txt, served as plain text. Root placement, exact filename. - Keep it maintained. A stale llms.txt pointing at dead URLs is worse than none. Regenerate it whenever your key URLs change (fold it into your release or migration checklist).
- Check it occasionally in server logs, hits from AI user agents on
/llms.txtare your evidence of real consumption.
Frequently Asked Questions
Does llms.txt help SEO?
Not Google SEO, Google has confirmed it doesn't use the file for Search or its AI features. It can help visibility in tools that do live retrieval (Perplexity, Claude, AI agents and coding assistants), which is a separate, smaller, but growing channel.
Is llms.txt the same as robots.txt?
No, they're opposites. robots.txt restricts, it tells crawlers what they may not fetch. llms.txt offers, it suggests what's most worth reading. It has no blocking power whatsoever.
Can llms.txt hurt my site?
The only failure mode is neglect: linking dead or outdated URLs misdirects the tools that do read it. There's no penalty, no ranking risk, no downside beyond staleness.
What's the difference between llms.txt and llms-full.txt?
llms.txt is the curated index (links + descriptions). llms-full.txt inlines the entire content of those pages into one markdown file so an AI can load everything in one request. The full version is mainly useful for documentation sites.
Want the Other 95% of Machine-Readable Visibility?
llms.txt is the cheap 5%. Structured data that search engines and LLMs actually parse at scale is where we do our work.
LinkedIn