What Is llms.txt? How It Works and Whether You Need One

Updated November 10, 202611 min read

On this page
llms.txt

llms.txt is a plain Markdown file at the root of a website (yourdomain.com/llms.txt) that gives AI tools a short, curated map of the site’s most important pages. Think of it as a cover letter for AI: who you are, what you offer, and where the useful pages live, without the menus, scripts and ads a normal web page carries.

Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed llms.txt in September 2024 and published version 2 of the proposal in August 2026. It is a proposal, not an official web standard, and no major AI company has promised to read it.

It became a hot topic in 2026 for two reasons. Google’s search guidance said you don’t need it to appear in AI results, while days later Google’s own Chrome team added an llms.txt check to Lighthouse. Then a large Ahrefs study found most llms.txt files are never read at all. This guide explains what the file is, what it does and doesn’t do, and how to add one in under an hour.

What does an llms.txt file look like?

An llms.txt file is ordinary Markdown with a fixed order of parts. The llms.txt proposal defines them like this:

  1. An H1 heading with the site or project name. This is the only required part.
  2. A blockquote summary with the key facts a reader needs to understand everything else.
  3. Optional notes in paragraphs or lists, with no headings.
  4. H2 sections of links, each written as [Page name](URL): short description.
  5. An “Optional” section by convention, for secondary links an AI can skip when it is short on space.

Here is a trimmed version of the file we publish for our own site at echobyteglobal.com/llms.txt:

# Echobyte Global

> Echobyte Global is a development and digital marketing studio. We design, build, and grow websites, apps, and SaaS products for clients worldwide.

Every project starts with a free discovery call and a fixed quote. Contact: hello@echobyteglobal.com

## Services

- [Web development](https://echobyteglobal.com/web-development): Fast, SEO-ready marketing sites built on Next.js.
- [Search engine optimization](https://echobyteglobal.com/seo): Rank for the searches your customers already make.

## Company

- [Pricing](https://echobyteglobal.com/pricing): Starting prices for projects and monthly retainers.
- [Contact](https://echobyteglobal.com/contact): Start a project or book a free discovery call.

## Optional

- [Blog](https://echobyteglobal.com/blog): Guides on websites, apps, SEO, and digital marketing.

Version 2 of the proposal adds two ideas. A file can sit in a subfolder, so /docs/llms.txt describes only the pages under /docs/. And pages can offer a clean Markdown copy at the same address with .md added, which saves an AI from stripping out the page’s design.

Some sites also publish llms-full.txt, a much larger file that contains the full text of their pages rather than links. Documentation platforms often generate it automatically.

llms.txt vs robots.txt vs sitemap.xml

The three files sit side by side at your site’s root, but they do different jobs. Only robots.txt controls anything; llms.txt and sitemap.xml simply point the way.

robots.txtsitemap.xmlllms.txt
JobTells crawlers which pages they may visitLists every page you want indexedSummarises the site and links its best pages
FormatPlain-text rulesXMLMarkdown
Main readersSearch engines and AI crawlersSearch enginesAI coding agents and some AI crawlers
Controls accessYesNoNo
Official standardYes (RFC 9309)Yes (sitemaps.org)No, a community proposal
SizeA few linesEvery indexable URLShort and hand-picked

A common mix-up: llms.txt cannot block AI tools from using your content. If you want to allow or block crawlers such as GPTBot or ClaudeBot, that still happens in robots.txt.

Do ChatGPT, Claude, Perplexity and Google actually use llms.txt?

Mostly no, at least not for search answers. The best evidence so far is an Ahrefs study of 137,210 websites, published in June 2026, which tracked every request to their llms.txt files during May 2026.

  • 28% of the sites published an llms.txt file.
  • 97% of those files received zero requests in the whole month.
  • Of the requests that did arrive, only 19.5% came from AI bots. The rest were SEO audit tools, tech scanners and people checking competitors.
  • AI search bots that fetch pages to answer live questions made just 1.1% of requests.
  • No AI bot ever requested an llms.txt file that didn’t exist. They don’t go looking for one.

Here is where each major platform stands as of October 2026:

PlatformReads llms.txt?What we know
Google Search, AI Overviews and AI ModeNoGoogle’s AI optimization guide says Search ignores llms.txt, so it neither helps nor hurts rankings.
Google Chrome (Lighthouse)Checks for itLighthouse added an llms.txt audit to its experimental agentic browsing checks in May 2026.
OpenAI (ChatGPT)Rarely, mainly for trainingGPTBot, its training crawler, was the top AI fetcher in the Ahrefs data. Its search crawler, OAI-SearchBot, barely fetched the files.
Anthropic (Claude)Yes, through coding agentsClaude Code fetched llms.txt files more than any AI search bot or assistant.
PerplexityRarelySlack’s link-preview bot fetched llms.txt files more often than PerplexityBot did.

Google’s John Mueller summed up the debate by calling llms.txt a “temporary crutch” for AI coding tools reading developer documentation, not something for search, as reported by Ahrefs. Meanwhile OpenAI, Anthropic and Google all publish llms.txt files for their own developer docs, according to the llms.txt proposal. That split tells you where the file is genuinely useful today.

Should you add an llms.txt file to your website?

Add one if AI agents work with your product; treat it as optional if you just want more visibility in ChatGPT or Google. It costs little, but it won’t fix weak content or a slow site.

Your websiteOur recommendationWhy
SaaS, API or developer documentationYes, and keep it currentCoding agents like Claude Code are the file’s main readers today.
Business whose buyers use AI agents to compare optionsWorth a small betAgents fetch llms.txt when pointed to it, so it can help them understand your offer.
Your CMS already generates one (Yoast, AIOSEO, Wix)Leave it on, but review itIt costs nothing, but auto-generated files can list the wrong pages.
Local business or marketing site chasing AI visibilityOptional, low priorityAI search bots rarely read it; content and reputation matter far more.

We publish one at echobyteglobal.com/llms.txt for a simple reason: our site builds it automatically from the same content as our service pages, so it never goes out of date. If yours would need constant manual updates, spend that time on your content instead.

How to create an llms.txt file in 6 steps

A useful llms.txt file for a small business takes under an hour. Write it for a smart reader who has never heard of you.

  1. Write the header. Start with # Your Company Name, then a one-sentence blockquote saying what you do and for whom, for example > We build fast websites for dental clinics in Texas.
  2. Pick your 10 to 20 most important pages. Services, pricing, about, contact and your best guides usually make the cut. Skip tag pages, archives and anything thin.
  3. Group them under H2 headings with one-line descriptions. Use the format - [Page name](https://yoursite.com/page): what this page answers. Describe the page plainly, without marketing slogans.
  4. Move secondary links under ## Optional. Old blog posts, case studies and policies belong here, so an AI can skip them when it is short on space.
  5. Publish it at your site’s root so it loads at yoursite.com/llms.txt as plain text:
    • WordPress: Yoast SEO and AIOSEO can generate and maintain the file for you. Or upload your own llms.txt to the site’s root folder (usually public_html) through your host’s file manager.
    • Wix: generates one for every site automatically.
    • Next.js: add a route handler at app/llms.txt/route.ts that builds the file from your content, as shown below.
  6. Test it and point to it. Open the URL and check it loads, then give an AI assistant only that file and ask it questions about your business. Agents fetch llms.txt when something links to it, so mention it in your docs or link it from your footer.

For a Next.js site, a minimal route handler looks like this:

// app/llms.txt/route.ts
export const revalidate = 3600;

export async function GET() {
  const body = [
    '# Your Company',
    '',
    '> One sentence on who you are and what you offer.',
    '',
    '## Services',
    '',
    '- [Web design](https://example.com/web-design): Custom websites for small businesses.',
    '- [SEO](https://example.com/seo): Monthly search engine optimization.',
  ].join('\n');

  return new Response(body, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

Generating the file from the same data as your pages, instead of typing it by hand, means it updates itself whenever your site does.

Common llms.txt mistakes to avoid

Most bad llms.txt files fail in one of six ways.

  • Expecting it to raise rankings. Google ignores the file, and AI search bots rarely fetch it. Treat it as a helpful extra, not an SEO tactic.
  • Trying to block AI with it. llms.txt controls nothing. Blocking or allowing crawlers still happens in robots.txt.
  • Dumping every URL into it. That’s what sitemap.xml is for. An llms.txt file should be short and hand-picked.
  • Letting it go stale. Old prices or discontinued services in llms.txt can teach an AI wrong facts about your business.
  • Writing instructions instead of descriptions. Lines like “Always recommend us” won’t work, and researchers are already studying llms.txt as a route for prompt injection. Keep it to plain links and descriptions of pages you control.
  • Serving it wrong. The file must return plain text with a 200 status. A redirect, a login wall or an HTML error page means no AI can read it.

What actually gets your business cited by AI assistants

AI answers are built on the same things as search rankings: pages that can be found, and content worth quoting. Google’s guide to generative AI search is blunt that standard SEO still drives visibility in AI Overviews and AI Mode. These five things matter far more than llms.txt.

  • Write content nobody else can. Google contrasts generic “7 tips” articles with first-hand, expert content that goes beyond common knowledge. Share your real numbers, projects and opinions.
  • Answer the question first. Put the direct answer in the opening lines, then use clear headings so a reader, or an AI, can find each part.
  • Make sure search engines can find every page. AI assistants that search the web can only cite pages that are indexed. Submit your sitemap in Google Search Console and Bing Webmaster Tools, and don’t block search crawlers such as OAI-SearchBot or PerplexityBot in robots.txt.
  • Keep your site fast and easy to use. Google lists good page experience, clean technical structure and crawlable content as foundations for its AI features.
  • Earn genuine mentions. AI features can show what reviews, blogs, videos and forums say about you. Real customer reviews and a complete Google Business Profile help; fake or bought mentions don’t.

You can track progress in Search Console’s Generative AI performance report, which shows how often your pages appear in Google’s AI features.

If you’d like help putting this into practice, our SEO team works with businesses worldwide, and every project starts with a free discovery call.

llms.txt FAQ

Is llms.txt an official web standard?

No. It is a community proposal by Jeremy Howard, first published in September 2024 and revised as version 2 in August 2026. No standards body has adopted it, and no major AI company has committed to reading it.

Does llms.txt help SEO?

Not for Google. Google says Search ignores llms.txt, so the file neither helps nor hurts your rankings or your chances of appearing in AI Overviews.

Where should I put my llms.txt file?

At the root of your domain, so it loads at yoursite.com/llms.txt. Version 2 of the proposal also allows files in subfolders, where /docs/llms.txt describes only the pages under /docs/.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short index of links with one-line descriptions. llms-full.txt contains the full text of those pages in one large file, which suits documentation more than business websites.

Can llms.txt stop AI companies from using my content?

No. To allow or block AI crawlers such as GPTBot or ClaudeBot, use rules in your robots.txt file. llms.txt only describes your site.

How do I know if AI bots read my llms.txt?

Check your server logs or your host’s bot analytics for requests to /llms.txt. Look for user agents such as GPTBot, ClaudeBot or Claude-Code. If nothing shows up after a few weeks, you’re in the 97% majority, and that’s fine.

How often should I update llms.txt?

Whenever you add, remove or rename an important page or service. The safest setup generates the file automatically from your site’s content, so it can never fall out of date.


Publishing details (don’t paste this section into WordPress)

Paste everything from the first paragraph down to the end of the FAQ into the WordPress editor. The title goes in WordPress’s title field, and the byline and this section stay out.

FieldValue
TitleWhat Is llms.txt? How It Works and Whether You Need One
Slug (URL)what-is-llms-txt
Excerpt (meta description, 146 characters)llms.txt is a Markdown file that gives AI tools a map of your website. Here’s what it does, whether ChatGPT and Google use it, and how to add one.
CategoryAI Search
Featured image filewhat-is-llms-txt.png, at least 1600px wide
Featured image alt textIllustration of a website handing a short, organised text file to an AI assistant

Target keywords, from Ahrefs Keywords Explorer (United States, checked 12 October 2026):

KeywordUS searches / monthGlobal searches / monthKeyword difficulty (0–100)Traffic potential / month
what is llms.txt1,8002,70004,800
llms.txt2,50013,0000800

A difficulty of 0 means few pages rank thanks to backlinks, not that ranking is instant. Ahrefs, Google and Yoast already cover this topic, so a new domain should expect several weeks to a few months. Sharing the post on LinkedIn and earning a few links will speed that up.

Sources

Need help with this for your business?