How the LLMs.txt Validator Works – Methodology & Verification Standards

1. Purpose & Scope

The llms.txt Checker & Validator at LLMs.txt Lab is engineered to answer a singular technical question: Does this file accurately conform to the structural rules of the current llms.txt proposal, and is it reliably accessible over the public web for automated AI agents?

To maintain research integrity and trust, we do not treat validation as a black box. This methodology document explains the exact mechanisms executed during validation, including our string parser, HTTP safety perimeter, machine discovery heuristics, and diagnostic categorization.

2. Input Processing & Text Extraction

The validator accepts content through three distinct ingestion workflows:

Direct Raw Input: Users paste raw Markdown into the browser textarea. This bypasses all networking layers and tests syntax in isolation.
File Upload: Users provide a local text file (.txt or .md). The client enforces a strict 2 MB payload boundary prior to submission.
Live URL Inspection: The user provides a domain (e.g., example.com) or direct endpoint URL. The request is proxied through our hardened backend client (SafeHttpClient), which fetches the content and extracts response metadata.

Once ingested, the validator normalizes line terminators (\r\n to \n) and computes structural measurements:

Metric	Computation Method	Technical Purpose
UTF-8 Byte Length	strlen($content)	Measures exact payload transfer weight over HTTP.
Character Count	mb_strlen($content, 'UTF-8')	Counts multi-byte Unicode characters accurately.
Line Count	count(explode("\n", $content))	Maps error diagnostics to exact 1-indexed line numbers.
Code Fence State	strpos($line, '```') === 0	Excludes sample headings and code snippets inside code blocks from rule triggers.

3. Validation Rule Engine

The parsing logic runs in a single deterministic pass across non-code-fence lines. Every check is identified by a stable diagnostic code:

LLMS001 • Specification Requirement Classification: Error

Required Single Top-Level H1 Heading

The document must contain exactly one top-level H1 heading (# Project Name). This is the only strictly mandatory structural element in the proposal. Missing an H1 results in document invalidity.

LLMS002 • Specification Requirement Classification: Error

Multiple H1 Headings Forbidden

Only one H1 is permitted per file. Multiple H1 tags create ambiguous project naming for automated parsers and are flagged as format violations.

MD001 • Markdown Syntax Requirement Classification: Error

ATX Heading Whitespace Syntax

CommonMark requires a whitespace character following the hash markers (# Title, not #Title). Missing whitespace prevents automated parsers from identifying the document title or section boundaries.

MD004 • Markdown Syntax Requirement Classification: Error

Malformed Markdown Link Syntax

File list items must follow standard Markdown link syntax: - [Title](URL): Description. Link rows that cannot be reliably parsed into a title and destination URL are flagged as syntax errors.

LLMS006 • Structural Warning Classification: Warning

Duplicate H2 Section Headings

H2 section headings are optional under the proposal, but when present, duplicate section names (e.g., two ## Documentation headings) can cause parsers that key off heading names to overwrite earlier dictionary entries.

LLMS008 • Convention Warning Classification: Warning

Optional Section Capitalization

The specification proposal designates ## Optional (capitalized) as the conventional section for secondary resources. Lowercase ## optional is flagged with an automated normalization suggestion.

URL004 • Quality Recommendation Classification: Warning

Duplicate URL Removal

Repeating the exact same destination URL across multiple sections consumes agent context tokens without providing new information. The validator flags duplicates and offers automated de-duplication.

4. Classification System: Standards vs. Recommendations

A core trust signal of LLMs.txt Lab is our strict taxonomy. We never conflate our own best-practice recommendations with the authoritative rules of the official proposal:

1. Specification Requirements

Derived strictly from the official proposal at llmstxt.org. Violations result in an Invalid document status. Example: Missing H1, multiple H1s, unparseable Markdown links.

2. LLMs.txt Lab Recommendations

Practical curation heuristics developed by LLMs.txt Lab to optimize token efficiency for agents. Violations emit Warnings/Suggestions but do NOT invalidate the file. Example: duplicate URLs, generic link anchor text, missing summaries.

3. HTTP & Deployment Diagnostics

Factual observations of the public web server response under RFC 9110 HTTP semantics. These are Deployment Diagnostics, not file format errors. Example: HTTP status code, HTTPS, Content-Type, Last-Modified, ETag.

Important File Size Clarification: The official specification proposal defines no numeric file-size limit (neither 100 KB, 250 KB, nor 500 KB). File size guidelines displayed in LLMs.txt Lab (Compact ≤50 KB, Moderate 50–200 KB, Large >200 KB) are purely editorial site-management tiers to help developers manage context budgets. A 300 KB file is 100% syntactically valid if its Markdown structure conforms to the proposal.

5. Safe Remote Fetching & Security Architecture

Fetching arbitrary user-supplied URLs introduces serious security risks, including Server-Side Request Forgery (SSRF), internal network scanning, cloud metadata exfiltration, and denial-of-service loops. LLMs.txt Lab executes all live fetches through a hardened isolation client (SafeHttpClient):

• Private & Reserved IP Filtering Prior to issuing any network connection, the client resolves the hostname's A and AAAA DNS records. Each IP is filtered with FILTER_FLAG_NO_PRIV_RANGE | FILTER_FLAG_NO_RES_RANGE. Connections to IPv4 private subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), loopbacks (127.0.0.1, ::1), and cloud metadata endpoints (169.254.169.254) are unconditionally aborted.

• DNS Pinning to Prevent TOCTOU Rebinding Standard HTTP clients validate an IP, then invoke cURL, which re-resolves the domain—leaving a window for Time-of-Check to Time-of-Use (TOCTOU) DNS rebinding attacks. Our client uses CURLOPT_RESOLVE to pin the connection directly to the pre-verified safe IP.

• Strict Port & Protocol Boundaries Only http:// and https:// schemes are permitted. URLs containing credentials (user:pass@) or arbitrary port specifications other than 80 and 443 are rejected immediately.

• Conservative Timeouts & Streaming Caps Connections time out after 5.0 seconds. A streaming write function tracks byte consumption and terminates transfers that exceed 5 MB, preventing zip-bomb and infinite-stream memory exhaustion.

• Controlled Redirect Recursion Automatic cURL redirect following is disabled (CURLOPT_FOLLOWLOCATION = false). Redirects (HTTP 301, 302, 307, 308) are handled manually up to a maximum depth of 3 hops, re-verifying SSRF safety on each hop.

6. v2 Machine Discovery Resolution

Per the llms.txt v2 specification, an llms.txt file is not required to live exclusively at the site root. When inspecting live domains, the validator executes discovery in this sequence:

Direct Candidate Fetch: If an explicit file path is requested (e.g., https://example.com/docs/llms.txt), the validator attempts direct retrieval.
Root Default: For bare domain queries, the validator requests https://example.com/llms.txt.
HTTP Link Header Discovery: If the endpoint returns a non-text response, the validator inspects HTTP response headers for Link: <URL>; rel="describedby".
HTML <link> Element Discovery: If a root URL serves an HTML page, the response body is parsed for <link rel="describedby" href="...">. If found, the validator follows the reference to fetch the actual context file.

7. HTTP & Deployment Diagnostics

When auditing a live endpoint, the validator records server telemetry governed by standard web protocols (RFC 9110 and RFC 9111):

HTTP Status Code: Confirms public reachability (HTTP 200). Statuses such as 403 (Forbidden), 404 (Not Found), and 5xx (Server Error) are clearly distinguished.
Soft-200 HTML Fallbacks: Single-page applications and content management systems frequently return HTTP 200 for missing pages while serving an HTML error page. The validator inspects the response body for HTML doc-types and flags soft-200 responses to prevent false-positive validation.
Content-Type MIME Reporting: The official llms.txt proposal does not mandate an official MIME type. Serving text or Markdown (such as text/plain or text/markdown) is reported as a deployment observation to confirm the server does not serve binary or HTML shells.
Server Freshness Validators (Last-Modified & ETag): Cache validators allow crawlers to perform conditional requests (If-Modified-Since, If-None-Match). If your web server omits these headers, the validator reports them as "Not provided by server"—we never synthesize fake dates.

8. Technical Boundaries & Limitations

Full transparency requires acknowledging what our validator can and cannot do:

• Specification Proposal Currency: Our validator evaluates documents against the current community proposal documented at llmstxt.org. As the proposal evolves, validation rules are updated accordingly.

• No Guarantee of AI Ingestion or Citations: Publishing a 100% valid llms.txt file provides clean context for compatible systems, but does not guarantee that third-party AI platforms (OpenAI, Anthropic, Google, Perplexity) will crawl, index, or cite your content.

• Anti-Bot & WAF Interception: Some web hosts employ aggressive bot protections (Cloudflare Managed Challenges, Akamai, AWS WAF) that block automated fetchers. When access is denied, our validator accurately reports the HTTP 403 status without guessing the file contents.

• Internal Link Verification: The validator inspects the structure of links inside the file. To verify the live availability of every external URL referenced within the document, use our dedicated llms.txt Link Checker.

9. Authoritative Primary References

All validation logic and terminology are grounded strictly in primary sources:

llmstxt.org — The primary proposal for the /llms.txt file standard.
llmstxt.org Changes — Official change history document detailing v1 to v2 modifications.
RFC 9110: HTTP Semantics — Standard definitions for HTTP methods, status codes, and header fields.
RFC 9111: HTTP Caching — Standards for Last-Modified, ETag, and Cache-Control validators.
Tranco Research List — Transparent, hardened ranking list for web research.

How the LLMs.txt Validator Works: Methodology & Verification Standards