llms.txt: The Complete Guide to Making Your Website AI-Ready (2026)
An honest, complete guide to the llms.txt standard — what it is, who created it, the exact format spec, how it compares to robots.txt and sitemaps, when it actually works (and when the hype outpaces the evidence), 4 real templates by site type, common mistakes, and a free generator to build yours in seconds.

What Is llms.txt?
Quick Answer
llms.txt is a Markdown-formatted file placed at the root of a website — yoursite.com/llms.txt — that helps AI language models understand what the site covers and which pages are most important. Think of it as a curated, AI-readable table of contents: not a complete copy of the site, but a structured guide to its best content, written in plain Markdown that any LLM already knows how to parse.
To understand why llms.txt was created, you need to understand one of the fundamental constraints of large language models: context windows. Even with 200,000-token context windows, feeding an entire website's HTML to an LLM is wasteful and unreliable — HTML is cluttered with navigation chrome, JavaScript, tracking tags, and other noise that makes it harder to extract the actual content. When an AI assistant tries to answer a question using a website, it needs clean, structured information fast.
The web already has two files that help with related problems: robots.txt tells crawlers what they can access, and sitemap.xml tells search engines all the URLs that exist. Neither one tells an AI agent what your site is about, which pages are most informative, or how to prioritize your content within a limited context budget. llms.txt fills that gap.
One sentence summary: llms.txt is to AI assistants what a sitemap is to search engines — but instead of listing all your URLs, it curates your best content and explains what each page covers, in a format AI systems can consume directly without parsing HTML.
Who Created the llms.txt Standard?
Quick Answer
llms.txt was proposed by Jeremy Howard, co-founder of Answer.AI and co-creator of the fast.ai deep learning library, in a blog post published on September 3, 2024. The official specification is hosted at llmstxt.org. It is a community proposal, not a formal W3C or IETF standard.
Howard's original proposal at llmstxt.orgidentified a gap: AI systems needed a way to quickly understand a site's structure without scraping and parsing entire pages. His insight was that most AI models already understand Markdown well, so the format should simply be Markdown — not a new XML schema, not JSON, not a specialized format that requires custom parsers. Just a well-structured Markdown file that any model can read.
Adoption moved quickly among developer-facing platforms. Anthropic, Stripe, Vercel, Cursor, Mintlify, Zapier, and FastHTML all shipped llms.txt files within months of the proposal. A SE Ranking study of 300,000 domains in early 2026 found a 10.13% adoption rate — roughly one in ten sites. That figure is both impressive for an 18-month-old proposal and a useful signal that ~90% of sites have not yet implemented it.
The llms.txt Format Specification (Full)
Quick Answer
A valid llms.txt file requires only one element: an H1 heading with the site name. Everything else is recommended but optional. The full recommended structure is: H1 (name), blockquote (summary), optional prose context, H2 sections (grouped links), each link as - [Title](URL): Description. Keep the file under 50KB and use ## Optional for lower-priority sections.
# Site Name> One-sentence description of what the site covers and who it is for. Optional additional prose context — paragraphs, not bullet lists. ## Section Name- [Page Title](https://yoursite.com/page/): What this page covers.- [Another Page](https://yoursite.com/other/): What it covers. ## Another Section- [Resource Name](https://yoursite.com/resource/): Description. ## Optional- [Secondary Resource](https://yoursite.com/secondary/): Less critical content.Element-by-element breakdown
# H1 — Site name (required). The only mandatory element per the llmstxt.org spec. This is the name AI systems use to identify your brand or project when discussing your content. Use your actual brand name, not a keyword-stuffed title.
> Blockquote — Summary(strongly recommended). Immediately after the H1, write a blockquote that describes what the site covers and who it serves. This is often the only part of the file that ends up in an AI agent's context window, so make it specific. “CSS and frontend developer tools — 43 free generators, converters, and builders for production-ready code” is better than “we help developers build better websites.”
Prose paragraphs(optional). After the blockquote, you can add one or two paragraphs of additional context — useful for explaining your site's positioning, audience, or scope in ways that a single sentence can't capture. Keep this under 150 words.
## H2 sections(recommended). Group your links into 3–6 logical categories that match your site's major content areas. Good section names: Tools, Documentation, Blog, Guides, Products, API, About. Avoid vague names like “Content” or “Resources.”
- [Title](URL): Description (the core unit). Each link should have an absolute URL and a one-sentence description that explains what the page covers well enough that an AI agent could use the description as a standalone answer — it may never fetch the URL. Include concrete facts in descriptions: version numbers, counts, specific features.
## Optional — a special section (see section 6 below for why this matters).
llms.txt vs robots.txt vs sitemap.xml
Quick Answer
robots.txt = access control (what bots may crawl). sitemap.xml = complete URL inventory (what pages exist). llms.txt = curated intelligence (what your site means and which pages matter most for AI). All three serve different purposes and should coexist. llms.txt does not replace the other two.
| File | Purpose | Audience | Format | Required? |
|---|---|---|---|---|
| robots.txt | Access control — permit or block specific bots from specific paths | All web crawlers | Custom directives syntax | De facto standard since 1994 |
| sitemap.xml | URL inventory — tell search engines all the pages that exist | Search engine indexers | XML | Strongly recommended for SEO |
| llms.txt | Curated intelligence — explain what the site covers and which pages matter | AI assistants, LLM agents, RAG pipelines | Markdown | Optional proposal (2024–) |
One important practical point: some site owners have accidentally added their AI bot rules to robots.txt in ways that inadvertently block crawlers they want to allow. Before adding llms.txt, verify that your robots.txt is not blocking GPTBot, ClaudeBot, PerplexityBot, or Google-Extended — these are the main AI crawlers, and a blocked crawler ignores llms.txt entirely. The three files are layers that work together, not alternatives.
llms-full.txt: The Companion File for AI Ingestion
Quick Answer
llms-full.txt is an optional companion file — where llms.txt is a curated navigation index (links to pages), llms-full.txt contains the full text of your important pages in a single Markdown document. Use llms.txt for real-time AI assistants with limited context. Use llms-full.txt for offline ingestion pipelines, RAG systems, and IDE tools like Cursor or GitHub Copilot that index your docs.
The distinction matters because they serve different consumers. An AI assistant answering a live user question can't afford to ingest 500,000 tokens of documentation on every query — it uses the lightweight llms.txt index to navigate to the relevant page and fetch just that. An IDE tool building a project-local knowledge base can and should ingest everything at once — that's what llms-full.txt enables.
To illustrate the scale difference: Anthropic's llms.txt is approximately 8,364 tokens — roughly 6,000 words. Their llms-full.txt is 481,349 tokens — the entire documentation in one file. Stripe and Vercel follow the same pattern. If you maintain a developer tool or documentation site, shipping both files covers all AI consumption patterns simultaneously.
When to ship llms-full.txt: If you run a documentation-heavy site, a developer tool with an API reference, or a knowledge base — and if your content is technical (code, API docs, how-to guides) rather than marketing copy. llms-full.txt is most useful for sites whose content gets embedded in developer workflows via tools like Cursor, Zed, or GitHub Copilot. The LLMs.txt Generator supports both formats — toggle between Standard and Full output with one click.
The ## Optional Section: A Special Semantic Signal
Quick Answer
## Optional is not just a label — it is a spec-defined semantic signal. When an AI agent is operating under tight context-window constraints, it is explicitly permitted by the llmstxt.org specification to skip all content under ## Optional without losing critical information. This is unique among llms.txt section names: no other H2 name carries a built-in behavioral instruction to AI agents.
Most articles about llms.txt treat ## Optionalas just a way to organize secondary content. But the distinction has real consequences. An agent tasked with “summarize this documentation site” that has 40,000 tokens of context budget will process your H1, blockquote, and named sections first. If context runs short, it drops the Optional section. This means anything critical — your pricing page, your API reference, your most-read guide — should be in a named section, not Optional.
Good candidates for ## Optional: post archives, changelog pages, older blog posts, legal pages, press releases, customer case studies, and any page that is informative but not essential for understanding what your product does. Reserve your named sections for the content an AI needs to accurately represent your brand.
Does llms.txt Actually Work? The Honest Evidence
Quick Answer
The evidence is mixed and use-case dependent. An Ahrefs study of 137,000 sites found 97% of llms.txt files are never fetched by any bot. Google does not support it. However, Perplexity uses llms.txt for page prioritization, and teams on documentation-heavy sites see correlated changes in citation behavior. The file works best as infrastructure for AI agents and RAG systems — not as a silver bullet for AI search visibility.
Reality Check
An Ahrefs study of 137,000 sites found that 97% of llms.txt files receive zero fetchesfrom any AI bot. GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot overwhelmingly crawl HTML pages directly rather than looking for llms.txt first. In July 2025, Google's Gary Illyes confirmed Google does not support llms.txt and compared it to the deprecated keywords meta tag.
The counterarguments are real, though. Perplexityhas confirmed it reads llms.txt to prioritize which pages to surface in answers. Teams on developer documentation platforms see measurable correlation between publishing llms.txt and changes in how AI assistants describe their products. For sites with sprawling navigation that's hard to crawl efficiently, providing an explicit curated index helps AI systems reach the right content faster.
The most honest framing: llms.txt's current value is not as an AI search ranking signal (despite what many articles claim), but as infrastructure for AI agent workflows and RAG pipelines — developer tools like Cursor, Zed, and GitHub Copilot, API documentation indexers, and AI assistants that need to operate on your content domain efficiently. If your users are developers who use AI coding tools, shipping an accurate llms.txt improves the AI-assisted experience of your product.
One more point worth making: the file takes less than 30 minutes to create. Even if AI search bots barely read it today, the cost of having it is near zero. The upside in specific use cases (Perplexity, developer tools, RAG) is real. This asymmetry makes it worth doing for most sites — just without the expectation that it will immediately move the needle on ChatGPT citations.
Who Should Add llms.txt (and Who Can Skip It)
Quick Answer
Add llms.txt if: your site is a developer tool, API, documentation site, SaaS product, or technical resource. Skip or defer it if: your site is a brochure site, a small personal blog, or a local business page. The ROI is highest for sites whose content gets used by AI agents in real developer workflows.
Worth Adding
- Developer tool sites
- SaaS product docs
- API documentation
- Open source project sites
- Technical blogs / newsletters
- Design system docs
- E-commerce with large catalogs
Lower Priority
- Small personal blogs
- Local business sites
- Single-page brochure sites
- Sites under 10 pages
- Sites that block AI crawlers
How to Create Your llms.txt: Step-by-Step
Quick Answer
The fastest path: open the LLMs.txt Generator, choose the preset that matches your site type, fill in your site name and description, edit the sections and links to match your actual pages, toggle to Standard or Full format, and download. The file is ready to deploy. The whole process takes under 10 minutes.
If you prefer to write it manually, here is the process:
- H1 — your site name. Write
# Your Site Name. Use your actual brand name, not a keyword title. - Blockquote — one-sentence summary. Write
> Your description here.Make it specific: who you serve, what you offer. This is the most important line in the file. - Optional prose(1–2 paragraphs max). Add context about your site's scope, audience, or differentiation if needed.
- Group your content into 3–6 H2 sections. Match your site's actual information architecture. Use
## Optionalfor secondary content. - Add links with descriptions. Format:
- [Page Title](https://yoursite.com/page/): One-sentence description.Use absolute URLs. Include concrete facts in descriptions. - Curate ruthlessly. Aim for 10–30 links total. Leave out thin content, outdated posts, legal boilerplate, and anything primarily serving search crawlers.
- Deploy at
/llms.txt. Place the file at the root of your domain. Serve astext/plain.
4 Real Templates by Site Type
Quick Answer
The four most common site types that benefit from llms.txt are SaaS products, frontend/tool sites, content blogs, and documentation sites. Each has a different optimal section structure. The examples below are production-quality templates — not generic placeholders. Use the LLMs.txt Generator to load these as starting presets and edit with your own content.
Template 1 — SaaS Product
# Acme Analytics> Real-time analytics and business intelligence for SaaS teams. Track> user behavior, build funnels, and export reports in seconds. Acme Analytics serves over 3,000 SaaS teams who need reliable, fastevent tracking without managing infrastructure. ## Product- [Features Overview](https://acme.io/features/): Full list of analytics features including event tracking, funnels, and retention cohorts.- [Pricing](https://acme.io/pricing/): Starter ($99/mo), Growth ($399/mo), Enterprise (custom).- [Integrations](https://acme.io/integrations/): 90+ one-click integrations including Segment, Mixpanel, Amplitude, and Slack. ## Documentation- [Quickstart Guide](https://acme.io/docs/quickstart/): Install the tracking snippet and send your first event in under 5 minutes.- [Events API](https://acme.io/docs/api/events/): REST API reference for server-side event ingestion.- [Dashboard Setup](https://acme.io/docs/dashboards/): Build and share custom dashboards with any team member. ## Blog- [What Is Product Analytics?](https://acme.io/blog/product-analytics/): Beginner guide to event tracking and funnel analysis. ## Optional- [Case Studies](https://acme.io/customers/): Customer stories and outcome data from Acme Analytics users.Template 2 — CSS / Frontend Developer Tool Site
This is an example of how CSSAWWWARDS itself uses an llms.txt — prioritizing the free tools, key guides, and clear description of the audience:
# CSSAWWWARDS> Free CSS and frontend developer tools — generators, converters, and> builders for production-ready CSS code. 43+ tools, no signup. CSSAWWWARDS provides interactive CSS tools including design tokengenerators, animation builders, layout tools, and accessibilitycheckers for frontend developers and UI designers. ## Tools- [Frontend Toolkit](https://cssawwwards.com/frontend-toolkit/): Complete directory of all 43+ free CSS tools.- [CSS @keyframes Builder](https://cssawwwards.com/tools/css-keyframes-builder/): Visual animation editor with 8 presets and live preview.- [CSS Container Query Builder](https://cssawwwards.com/tools/css-container-query-builder/): Build @container rules visually with a resizable preview slider.- [Color Blindness Simulator](https://cssawwwards.com/tools/color-blindness-simulator/): View any palette through 6 color vision deficiency types.- [Color Format Converter](https://cssawwwards.com/tools/color-format-converter/): Convert HEX, RGB, HSL, OKLCH, and HSV in both directions.- [LLMs.txt Generator](https://cssawwwards.com/tools/llms-txt-generator/): Build and download a valid llms.txt or llms-full.txt file.- [CSS Design Token Generator](https://cssawwwards.com/tools/css-design-token-generator/): Generate a complete design system as CSS custom properties.- [CSS Grid Generator](https://cssawwwards.com/tools/css-grid-generator/): Build CSS grid layouts visually with live preview. ## Guides- [Blog](https://cssawwwards.com/blog/): In-depth CSS tutorials, tool guides, and frontend best practices.- [CSS Container Queries Guide](https://cssawwwards.com/blog/css-container-queries-guide-2026/): Complete guide to component-first responsive design.- [CSS @keyframes Guide](https://cssawwwards.com/blog/css-keyframes-animation-builder-guide-2026/): Master CSS animations from scratch. ## Optional- [About](https://cssawwwards.com/about/): Background on the CSSAWWWARDS project and team.Template 3 — Content Blog
# The Frontend Dispatch> Weekly articles on CSS, JavaScript, and UI engineering for> practicing frontend developers. ## Latest Articles- [CSS Container Queries in 2026](https://frontdispatch.io/css-container-queries-2026/): Complete practical guide with component-first responsive patterns.- [The Case for OKLCH](https://frontdispatch.io/oklch-colors/): Why OKLCH is the best color format for accessible design systems. ## Series- [CSS Foundations](https://frontdispatch.io/series/css-foundations/): Eight-part guide covering layout, animation, color, and typography.- [Design Systems](https://frontdispatch.io/series/design-systems/): Building and maintaining a CSS design system at scale. ## Optional- [Archive](https://frontdispatch.io/archive/): All posts from 2019–2024.Template 4 — Developer Documentation Site
# Superdocs SDK> JavaScript SDK for document parsing, extraction, and AI-powered> search. Used by 800+ developers in production. ## Getting Started- [Installation](https://superdocs.io/docs/install/): npm install and quickstart in under 3 minutes.- [Authentication](https://superdocs.io/docs/auth/): API key setup and OAuth2 flow. ## Core API- [parse()](https://superdocs.io/docs/api/parse/): Extract structured data from PDF, Word, and HTML documents.- [search()](https://superdocs.io/docs/api/search/): Semantic search across all indexed documents.- [Webhooks](https://superdocs.io/docs/webhooks/): Real-time event notifications for async document processing jobs. ## Optional- [Changelog](https://superdocs.io/changelog/): Version history and migration guides.In all four templates, the pattern is the same: an honest one-sentence summary in the blockquote, a small number of well-named sections, and concise link descriptions that are useful as standalone answers even if the AI never fetches the URL. Use the LLMs.txt Generator to load any of these as a starting preset and customize with your real content.
Deploying llms.txt: Next.js, Static Sites, WordPress
Quick Answer
For most frameworks: place the file at /public/llms.txt (Next.js, Astro, Vite) or the root of your web server. The file must be served as Content-Type: text/plain. In Next.js, add a custom headers rule if you want to be explicit about the content type, though most servers default correctly for .txt files.
Next.js
// next.config.ts — serve llms.txt as plain text
import type { NextConfig } from "next";
const nextConfig: NextConfig = {
async headers() {
return [
{
source: "/llms.txt",
headers: [
{ key: "Content-Type", value: "text/plain; charset=utf-8" },
],
},
];
},
};
export default nextConfig;
// Place the file at: /public/llms.txt
// It will be available at: https://yoursite.com/llms.txtStatic sites (Astro, Vite, plain HTML)
Place llms.txt in your /public folder (Astro, Vite) or directly in the web root folder. The build output will include it at the root, making it available at yoursite.com/llms.txt automatically.
WordPress
Upload llms.txt directly to your server's root directory via SFTP or use a plugin like AIOSEO or RankMath which generate the file automatically from your sitemap and post data. Alternatively, use the LLMs.txt Generator to create a hand-curated file and upload it directly — the hand-curated version often produces better results because it gives you full control over which pages you surface to AI systems.
Verify your deployment
After deploying, verify three things: (1) yoursite.com/llms.txt loads in a browser as plain text — no HTML wrapper. (2) The response Content-Type header is text/plain. (3) Your robots.txt does not block User-agent: * from /llms.txt. These three checks confirm your file is valid and accessible to AI crawlers.
Common Mistakes That Break Your llms.txt
Quick Answer
The most common mistakes are: wrong file location (not at the root), wrong content type (served as HTML instead of plain text), relative URLs instead of absolute, using HTML formatting instead of Markdown, not writing a blockquote summary, and including too many links (diluting curation quality). The file at wrong Markdown syntax also fails — test it in a Markdown renderer before deploying.
# ❌ WRONG — uses HTML, not plain Markdown<h1>My Site</h1><p>Description here.</p><ul> <li><a href="/page">Page</a></li></ul> # ❌ WRONG — description is not a blockquote# My SiteMy site is about CSS tools and frontend development. ## Tools- [CSS Grid](/tools/grid/): Grid tool. # ✅ CORRECT — valid llms.txt# My Site> CSS tools and frontend development resources for modern developers. ## Tools- [CSS Grid Generator](https://mysite.com/tools/grid/): Visual CSS grid builder with live preview and copy-paste output.The complete mistake checklist
- Wrong location —
/docs/llms.txtor/static/llms.txtare not valid. Must be/llms.txtat the root. - Served as HTML — if your CMS or framework wraps the file in an HTML page template, AI agents may not parse it correctly. Verify the response is
text/plain. - Relative URLs —
[Page](/blog/post/)is unresolvable without context. Always use absolute URLs:[Page](https://yoursite.com/blog/post/). - No blockquote summary — the one-sentence blockquote is the most important line. Without it, AI systems have only your H1 name to understand what you do.
- Too many links — more than 50 links defeats the purpose of curation. If everything is important, nothing is. Aim for 10–30 total.
- Keyword-stuffed descriptions — description lines are not for SEO. Write them for AI agents that may use them as standalone answers: clear, factual, specific.
- Putting critical content under ## Optional — remember that Optional content may be skipped. Your pricing page, key product page, and top guide should be in named sections.
- Not keeping it updated — a llms.txt pointing to deleted or renamed pages hurts rather than helps. Review it quarterly or when you make major structural changes.
Frequently Asked Questions
What is llms.txt?
llms.txt is a Markdown file placed at yoursite.com/llms.txt that helps AI language models understand your site's structure and which pages are most important. It functions as a curated, AI-readable index — proposed by Jeremy Howard of Answer.AI in September 2024.
Who created the llms.txt standard?
Jeremy Howard, co-founder of Answer.AI and co-creator of fast.ai, proposed llms.txt on September 3, 2024. The official specification is at llmstxt.org. It is a community proposal, not a formal W3C or IETF standard.
What is the difference between llms.txt and robots.txt?
robots.txt controls which paths crawlers can access (permissions). llms.txt tells AI systems what your site covers and which pages are most valuable (curation and context). They serve different purposes and should coexist.
What is llms-full.txt?
llms-full.txt is an optional companion file containing the full text of your important pages in one Markdown document — for AI ingestion pipelines, RAG systems, and IDE indexers. Anthropic's is 481,349 tokens compared to 8,364 for their main llms.txt.
Does llms.txt improve AI search rankings or citations?
Mixed evidence. An Ahrefs study of 137K sites found 97% of files are never fetched by any bot. Google doesn't support it. Perplexity does use it for page prioritization. Best impact is on documentation sites and developer tools, not general content sites.
What does the ## Optional H2 section mean?
## Optional is a spec-defined semantic signal. AI agents with tight context budgets may skip content under this heading. Use it for lower-priority pages like archives, changelogs, and secondary resources. Keep critical pages in named sections.
Where must the llms.txt file be placed?
At the root path of your domain: yoursite.com/llms.txt. Subdirectory paths are not valid per the spec. Serve it as Content-Type: text/plain. The filename must be lowercase: llms.txt, not LLMs.txt.
Is there a free llms.txt generator?
Yes — the CSSAWWWARDS LLMs.txt Generator offers 4 presets (SaaS Product, Portfolio, Blog, Docs Site), interactive section/link editing, Standard and Full format toggle, live preview, and one-click download. Free, no signup.