Is llms.txt Required for AI Visibility?

No universal requirement for llms.txt can currently be established. The v2 proposal can provide a useful curated machine-readable entry point for agents and tools that support it, but publishing a file does not guarantee crawling, indexing, ranking, citation, referral traffic, or model use.

The important October 2026 update is that the proposal is no longer root-only. Version 2, published in August 2026, formally supports llms.txt at the site root or at a subpath, and adds standard ways for pages to advertise the relevant file.

What llms.txt v2 is

The proposal defines a Markdown file named llms.txt. A site can still publish the familiar root file:

https://example.com/llms.txt

But v2 also permits scoped files such as:

https://example.com/docs/llms.txt

A scoped file describes pages below its path. If several files could apply, the proposal says the most specific one should be used. That means /docs/llms.txt can describe the documentation area without pretending to describe the entire website.

A maintained file can provide:

• the canonical product, project, or organization name;
• a concise description;
• short explanatory context;
• grouped links to useful public resources;
• links to LLM-friendly or Markdown representations where available.

The design remains intentionally curated rather than a second sitemap containing every URL.

New in v2: explicit discovery links

Version 2 addresses a practical problem: given a normal page, how should an agent discover its Markdown representation or the llms.txt file that covers it?

The proposal recommends standard link relations:

• rel="alternate" type="text/markdown" can point to a Markdown version of the current page;
• rel="describedby" can point to the llms.txt file that covers the page.

These can be published as HTML <link> elements or HTTP Link: response headers. For example:

<link rel="alternate" type="text/markdown" href="/docs/page.md">
<link rel="describedby" href="/docs/llms.txt">

This is proposal-level discovery metadata. It does not prove that a specific answer engine consumes either relation.

What llms.txt is not

It is not:

• part of the Robots Exclusion Protocol;
• authentication or authorization;
• a universal crawler permission file;
• a replacement for sitemap.xml;
• a replacement for visible, accessible HTML;
• a guaranteed ranking signal;
• a guaranteed citation mechanism;
• proof that a named provider reads or prioritizes the file.

When a provider or product documents support, evaluate that implementation on its own terms. Do not infer universal adoption from the popularity of the proposal.

When the file can be useful

Curating authoritative entry points

A documentation or product team can point agents toward the pages that best define the product, API, policies, changelog, limitations, and support model.

Scoping complex sites

A large site can use a root file for broad orientation and a more specific file for a documentation or project area. This is one of the practical changes formalized in v2.

Reducing entity ambiguity

A concise canonical description and stable links can make the intended product and primary sources easier to identify for supporting consumers. The same claims still need to be accurate on the visible website.

Improving documentation hygiene

Maintaining a small curated index can reveal stale URLs, contradictory descriptions, duplicate destinations, missing canonical pages, and weak source organization even when no external AI system consumes the file.

What current evidence does not establish

The proposal itself does not prove that:

• ChatGPT Search requires llms.txt;
• Claude requires it for search or citation;
• Perplexity requires it for citation;
• Google AI features use it for eligibility;
• absence reduces ranking;
• presence improves referral traffic;
• every linked page will be fetched, indexed, selected, or cited.

Google continues to state that no special AI file or markup is required for inclusion in its AI features in Search. That does not describe every other tool, but it is strong evidence against presenting llms.txt as a universal requirement.

A useful minimal example

# Example Product

> Example Product helps independent consultants create clear project proposals in the browser.

## Product

- [How it works](https://example.com/how-it-works/): Core workflow and supported use cases.
- [Data and privacy](https://example.com/privacy/): Storage and processing boundaries.
- [Documentation](https://example.com/docs/): Setup and product guidance.

## Company and support

- [About](https://example.com/about/): Organization identity and stewardship.
- [Changelog](https://example.com/changelog/): Material product changes.
- [Contact](https://example.com/contact/): Support and correction route.

The summary and link descriptions should match the visible source pages. Do not add capabilities or guarantees that the site itself cannot support.

Root file or scoped file?

Use a root /llms.txt when one concise file can accurately describe the important resources for the whole site.

Consider a scoped file when:

• a documentation area is maintained independently;
• one project lives under a shared host;
• different sections have distinct canonical resources;
• the root site cannot accurately maintain the specialist links for every subsection.

Whichever scope you choose, keep ownership and review responsibility clear.

How OpenForBots evaluates llms.txt today

OpenForBots currently checks the root file at /llms.txt. It can report:

• the exact root URL checked;
• retrieval status, redirects, content type, and bytes read;
• title and introductory context;
• H2 sections and Markdown links;
• malformed link-like lines;
• duplicate destinations;
• insecure HTTP links;
• explicit limitations around provider adoption and visibility claims.

The live check does not discover subpath-scoped v2 files, rel="describedby" declarations, or Markdown rel="alternate" links. A missing root file therefore means only that OpenForBots did not find /llms.txt; it does not prove that the site publishes no v2-scoped file elsewhere. To validate the structure of a scoped file or an undeployed draft, paste its content into the checker’s paste mode, which applies the same structure rules in your browser without fetching a website.

That product boundary is intentional until broader v2 discovery is implemented and tested.

Verification checklist

1. Decide whether the file describes the whole site or a specific path.
2. Open the exact deployed URL and confirm a successful text response.
3. Read the file as a person and confirm the description is accurate.
4. Check every listed URL for a successful canonical destination.
5. Compare names and claims with the visible source pages.
6. Remove obsolete, private, duplicate, or low-value links.
7. If using v2 discovery links, verify the rendered HTML or HTTP Link: header points to the intended file.
8. Record the review date and responsible owner.
9. Re-check after material documentation changes.

Decision rule

Publish llms.txt when you can maintain a concise and accurate index that helps a documented consumer, agent workflow, or your own documentation governance.

Defer it when the site’s core pages are inaccessible, unclear, stale, or unsupported. Improving the source pages usually produces broader value than adding another machine-readable file.

Next step

Check or validate your llms.txt: inspect the deployed root file, or paste a draft or scoped file to validate it before publishing. Use the v2 proposal above if your site needs scoped files or explicit discovery relations.