1. Proposal Status & Origin
The /llms.txt specification is an open community proposal initiated by Jeremy Howard (Answer.AI) in September 2024, subsequently revised to v2 in August 2026.
2. Core File Grammar & Syntax
An llms.txt file is a plain-text Markdown document structured in a strict order. The grammar allows classical deterministic parsing (regular expressions, line scanners) without requiring a full AST compiler:
# Project or Site Title (Required) > Short project summary blockquote (Optional, recommended) Optional introductory prose, notes, or usage constraints (paragraphs, lists; no headings) ## Section Name (Optional, zero or more) - [Resource Name](https://example.com/docs/page.md): Concise description of resource ## Optional (Optional convention) - [Secondary Resource](https://example.com/tos.md): Licensing, policies, or auxiliary tools
An optional UTF-8 BOM is permitted at byte offset 0, though standard UTF-8 without BOM is preferred.
Exactly one top-level H1 heading (# Name) is required. This establishes the authoritative name of the project or site. Multiple H1 headings violate the grammar.
A blockquote starting with > immediately following the H1. Provides a high-level summary that an LLM agent uses to determine whether the site is relevant to the user query.
Zero or more paragraphs or bulleted lists explaining architectural prerequisites, compatibility tables, or guidance on how to interpret linked files. Heading tags (H1–H6) are not permitted in this section.
Zero or more sections delimited by H2 headings (## Section Name). Each section contains a Markdown list of resources using the syntax: - [Link title](URL): Optional description.
3. The ## Optional Convention in v2
One of the most significant refinements in v2 concerns the ## Optional section:
• v1 Behavior (Mechanical Semantics): In the initial proposal, ## Optional had programmatic meaning for external context-packing scripts (such as llms_txt2ctx), instructing tools to mechanically drop those URLs when bundling files into a single LLM prompt.
• v2 Behavior (Editorial Convention): Context-packing scripts have been removed from the core specification. In v2, ## Optional is recognized as a standardized convention for secondary links (terms of service, secondary licenses, optional extension repos) that an agent may safely skip if its context window or token budget is constrained.
• Parser Rule: An llms.txt file is completely valid without an ## Optional section. When present, it must be capitalized as ## Optional rather than lowercase.
4. Path Scoping & Subdomain Architecture
Unlike /robots.txt, which exists strictly at the origin root, llms.txt v2 explicitly supports path-scoped files:
| Placement Path | Covered URL Scope | Resolution Precedence |
|---|---|---|
| /llms.txt | All paths across the entire origin (/*). |
Fallback scope if no deeper path file exists. |
| /docs/llms.txt | Only pages residing within /docs/*. |
Most-specific match wins: Takes precedence for any URL under /docs/. |
| /api/v2/llms.txt | Only pages residing within /api/v2/*. |
Takes precedence over both /api/llms.txt and root. |
user.github.io/my-project/) to publish a fully functional /my-project/llms.txt covering their documentation without administrative access to the root origin.
llms.txt on https://example.com/llms.txt does not automatically govern or describe https://docs.example.com/ unless explicitly linked or referenced.
5. Machine Discovery & Markdown Alternates
v2 establishes two standardized link relations allowing agents to navigate directly from an arbitrary HTML page to its machine-readable representations:
Context File Discovery
Points from an individual web page to the applicable llms.txt file covering that path.
Clean Markdown Alternate
Points from a rendered HTML page directly to its clean, LLM-friendly Markdown equivalent.
HTTP Link Header Support
Both relations may be delivered via HTTP response headers (RFC 8288), allowing server-level or CDN-level deployment without editing HTML source files:
Markdown Alternate URL Conventions in v2
v2 officially blesses two widespread naming conventions for Markdown alternate URLs:
- Appended Extension:
page.html.md(original v1 convention). - Replaced Extension:
page.md(allowed in v2 to align with common static site generators). - Directory Endpoints: For URLs without filenames (e.g.,
/docs/), appendindex.html.mdorindex.md.
6. v1 vs. v2 Specification Changes Matrix
Based on two years of real-world web adoption, the August 2026 revision introduced the following specific changes documented in llmstxt.org/changes.html:
| Feature Dimension | v1 (September 2024) | v2 (August 2026 Revision) |
|---|---|---|
| Discoverability Relations | Unspecified; clients guessed /llms.txt. |
rel="describedby" and rel="alternate" type="text/markdown" via HTML and HTTP Link headers. |
| Markdown URL Syntax | Strictly required appending .md (e.g., page.html.md). |
Allows both appending (.html.md) and replacing extensions (.md). |
| Path Scoping Semantics | Permitted subpaths without defining resolution rules. | Explicitly defined: covers pages under that path; most-specific applicable file takes precedence. |
| Consumption Model | Described bulk expansion tool (llms_txt2ctx) to concatenate files into prompts. |
Agents view/search llms.txt, then selectively fetch linked resources on demand. Tooling removed from spec. |
| ## Optional Section Semantics | Mechanical directive for tools to omit links during context compilation. | Editorial convention for secondary resources; mechanical omission rules eliminated. |
7. What llms.txt Does Not Do
Misconceptions about the role of llms.txt are common. Technical practitioners should maintain clear operational boundaries:
llms.txt does not grant or revoke crawling permissions. Robots exclusion rules (robots.txt) govern bot access; llms.txt merely curates high-value content for allowed agents.
Sitemaps comprehensively list all indexable public URLs for search engine discovery. llms.txt is a selective, curated briefing document designed to fit within agent context limits.
Publishing an llms.txt file does not provide automated search rank boosts or guaranteed visibility in Google or Bing search results.
LLM agents synthesize responses based on internal retrieval mechanisms. Having a valid llms.txt file facilitates cleaner retrieval, but cannot compel an agent to cite your domain.
8. Primary Sources & Community References
- llmstxt.org — Official v2 proposal document authored by Jeremy Howard.
- llmstxt.org Changes Log — Documented historical changes from v1 to v2.
- AnswerDotAI/llms-txt GitHub Repository — Open community repository for issues, revisions, and discussions.
- LLMs.txt Lab Validator Methodology — Detailed breakdown of how our automated validation engine enforces this specification.
- Free llms.txt Validator Tool — Test your own file against the v2 grammar.