No universal requirement for llms.txt can currently be established. The file may provide a useful curated machine-readable entry point for tools that explicitly support the proposal, but it does not replace robots.txt, accessible public HTML, sitemaps, structured data, or clear source pages. Publishing it does not guarantee indexing, ranking, citation, traffic, or model use.
What llms.txt is
The llms.txt proposal recommends publishing a Markdown file at the root of a website:
https://example.com/llms.txtA maintained file can provide:
- a canonical product or organization name;
- a concise description;
- short explanatory notes;
- groups of links to important public resources;
- optional links to detailed Markdown representations.
The design goal is a small, curated context document rather than a machine-generated copy of every site URL.
What llms.txt is not
It is not:
- part of the Robots Exclusion Protocol;
- authentication or authorization;
- a universal crawler permission file;
- a replacement for
sitemap.xml; - a replacement for visible HTML;
- a guaranteed ranking signal;
- a guaranteed citation mechanism;
- proof that a named provider reads or prioritizes the file.
When a provider or tool documents support, evaluate that implementation on its own terms. Do not infer universal adoption from the existence of the proposal.
When the file can be useful
Curating authoritative entry points
A documentation team can point to the pages that best define the product, API, policies, changelog, limitations, and support model.
Reducing entity ambiguity
A concise canonical description and stable links can make it easier for supported consumers to identify the intended product and source pages. The same information still needs to be accurate on the visible website.
Supporting documented consumers
Some developer tools, documentation systems, and retrieval workflows intentionally read llms.txt. For those consumers, a valid file creates a predictable discovery path.
Improving documentation hygiene
Maintaining a short inventory can reveal stale URLs, contradictory product language, missing canonical pages, and outdated documentation.
What the current evidence does not establish
The proposal itself does not prove that:
- ChatGPT search reads or prioritizes every llms.txt file;
- Claude search requires the file;
- Perplexity requires it for citation;
- Google AI features use it for eligibility;
- absence reduces ranking;
- presence improves referral traffic;
- every linked page will be fetched, indexed, selected, or cited.
Google currently states that no special AI file or markup is required for inclusion in its AI features in Search. That provider statement does not describe every other tool, but it is strong evidence against presenting llms.txt as a universal requirement.
A useful minimal example
# Example Product
> Example Product helps independent consultants create clear project proposals in the browser.
## Product
- [How it works](https://example.com/how-it-works/): Core workflow and supported use cases.
- [Data and privacy](https://example.com/privacy/): Storage, processing, and external-service boundaries.
- [Documentation](https://example.com/docs/): Setup and product guidance.
## Company and support
- [About](https://example.com/about/): Organization identity and stewardship.
- [Changelog](https://example.com/changelog/): Material product changes.
- [Contact](https://example.com/contact/): Support and correction route.The summary and link descriptions should match the visible pages. Do not add capabilities or guarantees that the site cannot support.
Quality requirements
A useful file should:
- use the canonical product and organization names;
- link only to public, successful, canonical URLs;
- describe every link accurately;
- prioritize the pages that define the product and its boundaries;
- exclude private, temporary, duplicate, and low-value pages;
- avoid unsupported claims;
- show enough context to distinguish similar entities;
- be reviewed when important documentation changes;
- remain concise enough to function as a curated entry point.
How it differs from other web files
| Resource | Primary job | What it does not prove |
|---|---|---|
robots.txt | Express crawler policy for matching product tokens and paths | Successful delivery, indexing, or citation |
sitemap.xml | Help supported crawlers discover URLs | Content quality, entity clarity, or selection |
| Structured data | Describe visible entities and page meaning in a machine-readable form | Provider use, ranking, or citation |
llms.txt | Curate context and important links for supporting consumers | Universal adoption or visibility benefit |
| Public HTML | Deliver the actual page content and links | Selection by a private answer system |
These resources can complement one another. None substitutes for clear, accurate source pages.
Do not use llms.txt to repair weak source pages
Fix the main website first when it has:
- empty or thin initial HTML;
- unclear product positioning;
- contradictory organization names;
- broken canonical URLs;
- stale documentation;
- unsupported claims;
- inaccessible navigation;
- blocked important paths;
- weak primary-source evidence.
A machine-readable index cannot make an unclear or inaccessible source authoritative.
Do not copy the sitemap
A sitemap may contain every indexable URL. llms.txt should select the most useful entry points and explain why they matter.
Copying hundreds of URLs into Markdown creates another inventory to maintain without adding useful context.
How OpenForBots evaluates llms.txt
When the file exists, OpenForBots can report:
- the exact URL checked;
- retrieval status and content type;
- whether the response appears readable as text;
- basic Markdown structure;
- the presence of a title and summary;
- syntactic validity of listed links;
- obvious stale, contradictory, or low-value patterns using bounded heuristics;
- the limitations of the check.
OpenForBots treats absence as an optional-resource result, not a core access failure. It does not award guaranteed-visibility points for presence.
Verification checklist
- Open the exact root URL and confirm a successful text response.
- Read the file as a person and confirm its description is accurate.
- Check every listed URL for a successful canonical destination.
- Compare names and claims with the visible pages.
- Remove obsolete, private, duplicate, or low-value links.
- Record the review date and responsible owner.
- Test with a specific tool only when that tool documents support.
- Re-run the check after material documentation changes.
Decision rule
Publish llms.txt when you can maintain a concise and accurate index that helps a documented consumer or your own documentation workflow.
Defer it when the site’s core pages are inaccessible, unclear, stale, or unsupported. Improving the source pages produces more general value than adding another file.
Next step
Run the OpenForBots audit to inspect the public llms.txt resource alongside crawler access, initial HTML, content clarity, structured data, and identity signals.