llms.txt Checker & Validator

What Is the llms.txt Checker & Validator?

The llms.txt Checker & Validator is the flagship diagnostic tool for llms.txt files, evaluating both file correctness (Markdown syntax, heading hierarchy, and link structure) and deployment correctness (live endpoint availability, HTTP status, redirect chains, file weight, and server freshness headers) in a single unified report.

Instead of running fragmented checks across separate pages, this tool inspects both your file's internal composition and its public accessibility on the web.

The Two Audit Layers: File Validation vs. Deployment Inspection

A complete audit requires examining two distinct layers of your llms.txt implementation:

Result Interpretation: Requirements vs. Recommendations vs. Diagnostics

To avoid confusion, the validator strictly separates specification requirements from site management recommendations and raw HTTP diagnostics:

• Specification Requirements (Errors): Direct violations of the current llmstxt.org proposal, such as a missing H1 title, multiple H1 titles, or unparseable Markdown links.
• LLMs.txt Lab Recommendations (Warnings & Suggestions): Best practices identified by LLMs.txt Lab for efficient retrieval, including descriptive link annotations, avoiding duplicate URLs, lowercase section warnings, and file-weight guidance.
• HTTP & Deployment Diagnostics (Information): Factual server response characteristics reported under standard RFC 9110 HTTP semantics (e.g. response code, TLS status, caching headers, or "Not provided by server" when headers are absent).

Understanding HTTP Response Codes & Soft-200 Fallbacks

A healthy deployment returns HTTP 200 (OK). Content-Type is reported as an HTTP deployment diagnostic: while the current llms.txt proposal does not define a mandatory MIME type for the resource, serving the intended text or Markdown response with an appropriate text-oriented Content-Type (such as text/plain or text/markdown) helps clients distinguish the resource from an HTML application shell or binary response. Crucial scenarios monitored by the validator include:

• HTTP 404 (Not Found): The file is missing from the public root or path-scoped directory. See our guide on Where to Put an llms.txt File for platform-specific folders.
• Soft-200 HTML Fallbacks: Many CMS platforms and single-page applications return an HTML 404 page with an HTTP 200 status code. The validator analyzes the response body and flags HTML pages so they are not mistaken for valid context documents.
• HTTP 401 / 403 (Forbidden): The server refused access due to firewall rules, basic auth, or bot protection, blocking AI assistants from reading the file.
• HTTP 301 / 308 (Redirects): Redirects can be valid if they terminate at the correct file, but circular redirects or home page redirects are flagged immediately.

Server Freshness: Last-Modified Headers & ETags

Last-Modified and ETag are standard HTTP cache validators (RFC 9110) that help clients determine whether a resource has changed without downloading the entire body. Whether an external crawler or LLM agent utilizes these headers depends entirely on that individual client's HTTP implementation. If your web server omits these headers, the validator reports them as "Not provided by server" without fabricating synthetic timestamps.

File Size, Byte Weight, and LLMs.txt Lab Guidelines

The official specification does not establish a strict numeric file size limit. The core intent is to keep the main file concise as an immediate navigational overview, while pointing to deeper individual pages for detailed context. For practical site management, LLMs.txt Lab provides these reference tiers:

• LLMs.txt Lab Guideline — Compact (≤ 50 KB): Ideal for immediate context loading and fast retrieval.
• LLMs.txt Lab Guideline — Moderate (50 KB – 200 KB): Comprehensive documentation index. Ensure linked resources carry in-depth code samples and details.
• LLMs.txt Lab Guideline — Large (> 200 KB): Substantial context payload. Consider dividing secondary sections into path-scoped files (e.g. /docs/llms.txt).

Validator vs. Link Checker: What is the Difference?

The Validator inspects the syntax, structure, and live endpoint deployment of the llms.txt file itself. The llms.txt Link Checker, by contrast, is an independent tool that performs live HTTP requests against every individual URL referenced inside the file to detect broken links and dead external targets. Keeping these tasks separate ensures rapid validation without waiting for dozens of third-party network requests.

Methodology & Evidence Boundaries

Validation in this tool reflects the technical proposal published at llmstxt.org and standard web protocol definitions. Read our complete Validator Methodology for detailed documentation on parsing rules, SSRF security defenses, and diagnostic classification. Publishing a syntactically valid llms.txt file provides an organized context resource for compatible tooling, but does not guarantee inclusion, search ranking improvements, or AI model citations.