Short Answer
Use llms.txt as a curated AI context file. It should tell AI agents which pages explain your website best, what each page is for, and which resources are essential versus optional.
A good llms.txt file is short enough to scan, detailed enough to be useful, and updated often enough to reflect your real website.
What Information Should Be Included in llms.txt?
The best content depends on your website type, but most files should include a clear title, a short summary and grouped links to high-value resources. The goal is to help an AI agent answer questions accurately without wandering through low-priority pages.
- A clear H1 title with the site, product or project name.
- A short summary of what the website does and who it helps.
- Important documentation and getting-started pages.
- API reference, SDK docs, changelog and developer resources.
- Product, pricing, comparison and feature pages.
- Support, contact, legal and policy pages when they clarify usage.
- Optional resources that are useful but lower priority.
How to Organize the File
Group links into simple sections that match how users and AI systems would think about your site. Avoid clever labels. Clear names like Documentation, API Reference, Product, Tutorials, Support, Policies and Optional are easier to understand.
Each link should have a descriptive title. When possible, add a short description after the link. A description can explain the page type, the user problem it solves, or the information an AI agent will find there.
Better link format: - [API Authentication](https://example.com/docs/auth): Explains API keys, OAuth flows, scopes and token rotation.
How Different Websites Can Use llms.txt
- SaaS sites: include product overview, setup docs, pricing, support and API references.
- Documentation sites: include getting-started guides, concepts, reference pages and release notes.
- Ecommerce sites: include buying guides, category explanations, shipping, returns and support policies.
- Blogs: include pillar articles, topic hubs and evergreen resources instead of every post.
- Agencies: include services, case studies, process pages, contact details and policies.
What Not to Include
A bloated file is harder to use. If you include every URL, you turn llms.txt into a weaker sitemap. Keep the file focused on resources that explain your site, product, service or subject clearly.
- Login, checkout, cart or account pages.
- Internal search result pages.
- Thin tag archives and duplicate category pages.
- Outdated pages that no longer represent your product or service.
- Private pages, staging URLs, tokens or internal documents.
How Often Should You Update It?
Update your llms.txt file when your main documentation, product pages, API docs, pricing, support resources or policies change. If your website changes often, review it monthly. If your site is stable, review it after major launches or content updates.
A stale file can mislead AI systems by pointing them to outdated docs or missing important new content. Treat it like a small public index that deserves maintenance.
Best Practice Workflow
Start with the llms.txt generator, import your sitemap or URL list, remove weak pages, add descriptions, and download the file. Then check it with the llms.txt Validator before uploading it to your site root.