1. Executive Summary & Key Findings
While the llms.txt proposal has sparked substantial discussion in developer and AI SEO communities since its introduction in late 2024, widespread speculation has persisted regarding its actual adoption rate. To provide transparent, reproducible empirical evidence, LLMs.txt Lab executed a high-volume, SSRF-defended crawl across the top 10,000 websites from the hardened Tranco Research List.
/llms.txt files is strongly skewed toward the highest-ranking web properties. While overall adoption across the entire 10,000 domain population is measured at 7.75% (12.39% among determinable outcomes, 10.20% among reached hosts), prevalence rises significantly among the top tier (11 / 100 (11.00%) in the Top 100), led primarily by developer infrastructure, cloud computing providers, and AI tooling platforms.
2. Research Population & Tranco Snapshot
To eliminate the ranking volatility common in commercial SEO rank trackers, this research sampled domains from the peer-reviewed Tranco Research List:
74V6Xgoogle.com) through Rank 10,000 (huan.tv)https://{domain}/llms.txt (with automatic safe HTTP fallback)3. Complete Response Classification Breakdown
Every audited domain was assigned to one of seven mutually exclusive classifications defined prior to executing the crawl:
| Outcome Category | Domain Count | % of Sample (10k) | Technical Diagnostic Description |
|---|---|---|---|
| Observed public llms.txt | 775 | 7.75% | HTTP 200, intentional text/markdown resource observed (not HTML, not shell, not error). Decoupled from strict validator syntax compliance. |
| Missing File (HTTP 404 / 410) | 4,563 | 45.63% | Origin server responded with standard 404 Not Found or 410 Gone status. |
| Soft-200 HTML Shells | 414 | 4.14% | HTTP 200 returned, but body is an HTML SPA shell or HTML error page. |
| Blocked / Forbidden (401/403/429) | 1,100 | 11.00% | Access denied by cloud WAF (Cloudflare/Akamai challenge) or HTTP 403. Status unresolved. |
| Redirected Away (Home/Other) | 502 | 5.02% | Request redirected away from llms.txt to homepage or non-context path. |
| Network / Connection Timeout | 2,404 | 24.04% | Host unreachable after two-pass attempts, DNS failure, or SSL handshake error. |
| Ambiguous / Empty Body | 242 | 2.42% | 200 OK with 0-byte payload or plain text lacking standard headings. |
- Full Population (N = 10,000): All sampled domains; gives the conservative baseline of 7.75%.
- Determinable Adoption Cases (N = 6,254): Domains where presence or absence could be verified (confirmed, 404, soft-200, redirect-away); yields 12.39%.
- Reached Hosts (N = 7,596): Hosts that returned any non-network HTTP response; yields 10.20%.
Domains blocked by perimeter WAFs (1,100) or returning ambiguous responses (242) are classified as unknown/unresolved rather than presumed absent.
4. Adoption by Ranking Tier
Partitioning the dataset across traffic tiers reveals an unmistakable adoption pattern:
| Ranking Cohort | Sample Size | Confirmed llms.txt | Cohort Adoption Rate |
|---|---|---|---|
| Top 100 Domains | 100 | 11 | 11.00% |
| Top 1,000 Domains | 1,000 | 91 | 9.10% |
| Top 10,000 Domains (Full Study) | 10,000 | 775 | 7.75% |
5. Content Format & HTTP Delivery Telemetry
Among the 775 confirmed public implementations, telemetry was collected to analyze how organizations structure and serve their context files:
Structural & Content Patterns
- Syntax Compliance (Valid Single H1): 91.35%
- Median Byte Weight: 6,837 bytes (~6.7 KB)
- Median Link Count: 28 links per file
- ## Optional Section Presence: 128 files (16.52%)
HTTP Deployment Signals
- Served over Secure HTTPS: 99.87%
- ETag Header Present (RFC 9110): 515 files
- Last-Modified Header Present: 518 files
- Primary Content-Type: text/plain; charset=utf-8
6. Notable Observed Real-World Deployments
The crawl verified live root llms.txt files across prominent high-ranking domains in the sample:
See our dedicated Real-World llms.txt Examples Directory for in-depth structural breakdowns of Anthropic, OpenAI, FastHTML, Google Gemini API, and Mintlify.
7. Open Data Downloads (CSV & JSON)
In keeping with open research principles, LLMs.txt Lab provides the complete dataset for public download and independent verification under CC BY 4.0:
Review our complete Data Dictionary for definitions of all telemetry attributes.
8. Methodology, Reproducibility & Limitations
Two-Pass Reproducibility Pipeline
The data collection pipeline was executed using our offline research script (research/scripts/crawl-tranco.py) running Python 3.9 and aiohttp. Pass 1 sampled all 10,000 domains with a 35-worker ceiling and 6.0s timeout. Pass 2 automatically executed a low-concurrency retry pass (8 workers, 12.0s timeout) targeting transient network timeouts exclusively.
For complete step-by-step instructions on reproducing this scan on your own workstation, see the LLMs.txt Lab Validator Methodology and research documentation.
Research Limitations
• Root-Level Scope Boundary: This study tested exclusively for the presence of https://{domain}/llms.txt. It did not traverse internal HTML pages to locate path-scoped files (e.g., /docs/llms.txt) or HTML rel="describedby" links. Actual adoption of path-scoped files is therefore expected to be higher than root prevalence alone.
• Point-in-Time Nature: Public web endpoints are updated dynamically. These metrics represent a point-in-time snapshot collected on 2026-09-07 from Tranco list 74V6X.
• WAF & Bot Defense Obstacles: Highly defended domains employing Cloudflare Managed Challenges or perimeter WAFs return HTTP 403 to automated crawlers. These endpoints are recorded as blocked_forbidden rather than presumed absent.
• Agent Usage vs. File Presence: An observed llms.txt confirms that the publisher has provided an AI context file; it does not prove that third-party LLM systems (e.g., ChatGPT, Claude, Gemini) actively ingest the document during inference.
9. Citation & Attribution
If you reference this dataset or study in technical documentation, academic papers, or articles, please cite as follows: