Short answer: llms.txt is a proposed Markdown file served at the root of your domain that gives language models a short, curated map of your site: what you are, who it is for, and which pages matter. It is a convention, not a standard that any AI vendor has committed to read. Write one because it is cheap and forces useful decisions about your own content, not because it will put your brand into AI answers.
This guide covers what the file is for, what to put in it, how each major site platform serves it (they differ more than you would expect), how to check it, and where it belongs in your priorities.
What llms.txt is, and what it is not
The proposal is simple. A plain Markdown document at /llms.txt starts with the name of the site or product, follows with a short summary, and then lists the important pages in sections, each link with a one-line description. The idea is that a model, or an agent acting for a user, can read one small file instead of crawling the whole site and guessing which pages are authoritative.
It is not an access control and it is not a ranking signal. Whether an AI agent can fetch your pages is decided by robots.txt and, in practice, by what your CDN or firewall returns to that agent. The vendors document their agents in detail: OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity separates PerplexityBot from Perplexity-User. None of the vendor pages we checked for our AI crawler list says that llms.txt is read when an answer is composed. Treat any promise that the file "gets you into ChatGPT" accordingly.
The three root files are often confused, so it helps to set them side by side:
| File | Its job | Controls access? | Documented by AI vendors as read for answers? |
|---|---|---|---|
robots.txt | Tells each named agent which paths it may fetch | Yes, for agents that honour it | Yes: every vendor page we checked describes how its agents treat it |
| Sitemap | Lists canonical URLs and when each changed | No | Not in the vendor pages we checked |
llms.txt | A curated, described map of the pages that matter | No | Not in the vendor pages we checked |
There are still good reasons to write one. It costs little, it makes you decide which of your pages are canonical, and one major commerce platform now serves the file by default, so many sites already have one whether they wrote it or not.
What to put in llms.txt
A structure that holds up on a real site looks like this:
# Example Co
> Example Co sells modular standing desks to home and small-office buyers
> in the UK and EU. This file lists the pages that describe the products,
> prices, delivery and returns.
## Products
- [Standing desks](https://example.com/desks/): frame options, sizes, load limits
- [Accessories](https://example.com/accessories/): monitor arms, cable trays
## Buying
- [Pricing](https://example.com/pricing/): current prices and what each bundle includes
- [Delivery and returns](https://example.com/returns/): regions served, return window
## Company
- [About](https://example.com/about/): who runs the company and where it is based
Six habits make the difference between a useful file and a decorative one:
- Treat the summary as the most important part. Say what the business is, who it serves and where its scope ends, in the words a stranger would use. Marketing copy that would fit any competitor tells a model nothing.
- Group links by the decision a reader is making, not by your navigation menu. Products, then buying information such as prices and policies, then documentation or guides, then company pages.
- Write link text and a description for every link. A bare URL carries no meaning; the description is the part that explains what the page answers.
- Define your own terms if your product uses names or metrics a stranger would not know. That is exactly the ambiguity a model would otherwise resolve by guessing.
- List only pages that are crawlable and readable without JavaScript. A file that points at blocked or client-rendered pages helps nobody. Our guide to JavaScript rendering and AI crawlers explains how to check.
- Generate it from your routes or CMS rather than writing it by hand. A hand-written file drifts as pages are renamed or deleted, and a file that points at pages that no longer exist is worse than none.
For more worked examples of the format, see llms.txt example and format.
How each platform serves llms.txt
Where the file lives and how you change it depends almost entirely on your platform. These are the documented behaviours:
| Platform | How /llms.txt is served | What to watch for |
|---|---|---|
| Shopify | Served by default with no theme file present: Shopify serves /agents.md and mirrors it to /llms.txt and /llms-full.txt. The lookup order for /llms.txt is llms.txt.liquid, then agents.md.liquid, then Shopify's managed default. | Adding agents.md.liquid changes all three files; adding llms.txt.liquid changes only /llms.txt. |
| WordPress (self-hosted) | Upload llms.txt to the document root. The recommended server configuration serves existing files before routing to index.php, so the static file wins. | Does not apply to WordPress.com hosted plans, where filesystem access is restricted. Confirm the file is not swallowed by the rewrite. |
| Webflow | The help centre documents a dedicated llms.txt upload that serves the file from the domain root on a custom domain. | Other arbitrary root-level files are still unsupported. Check on the custom domain, never on the webflow.io staging subdomain. |
| HubSpot | No documented way to serve an arbitrary static file at the root; File Manager assigns URLs under /hubfs/. | A page whose slug is llms.txt resolves at the root but is served as text/html. A reverse proxy in front of HubSpot is the reliable option. |
The Shopify row deserves a second look if you run a store. Because Shopify serves a managed default, your store is probably already publishing an llms.txt you have never read. Open it before deciding whether to override it, and if you do override it, prefer agents.md.liquid so that all three files say the same thing. The sources are Shopify's llms.txt template documentation, Webflow's llms.txt upload article, WordPress's nginx guide and HubSpot's supported file types. Platform-specific detail is collected on our Shopify platform page.
How to check your llms.txt
Two commands cover most of the check:
curl -sI https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -n 5
- It answers at the root itself, not after a redirect. Programs that fetch control files commonly do not follow a redirect, so a file that only exists behind one is effectively missing.
- The content type is plain text, not text/html. This is the failure the HubSpot slug workaround produces, and it is invisible in a browser.
- The body starts with your title line, not with an HTML error page, a cookie banner or a login wall.
- Every linked page is reachable by the agents you care about. Test with the real user agent rather than reading robots.txt: a permissive robots.txt behind a CDN rule that returns 403 to those agents is a block in practice. Our AI crawler access guide walks through that test.
- You checked the production domain. On Webflow in particular, the staging subdomain serves different control files from the custom domain.
Re-run the check after any redesign, migration or platform change, because those are the moments when root files quietly disappear.
Where it belongs, and common mistakes
If you have limited time for machine readability, spend it in this order: confirm the search and user agents can actually fetch your content pages; make sure the key answers exist in the raw HTML; write passages that stand on their own; make your brand unambiguous with consistent descriptions and Organization markup; and then write llms.txt. The order is about dependencies, not about a measured effect: each earlier step is a precondition for the later ones. An llms.txt that points at pages an agent cannot fetch, or that render empty without JavaScript, changes nothing.
Read the stack from the bottom. If a lower layer is missing, fix it before spending time on anything above it; if all the lower layers hold, llms.txt is a sensible, cheap addition rather than a gamble.
Common mistakes
- Using it as robots.txt. The convention has no access-control meaning. Rules about which agents may fetch what belong in robots.txt and in your CDN configuration.
- Listing every URL. That is a sitemap. The value of llms.txt is the curation and the descriptions.
- Letting it go stale. If it is not generated, put a review date next to it and an owner on it.
- Measuring success by whether the file exists. The outcome you care about is in the answers. Measure that directly, with a fixed set of buyer questions, rather than inferring it from the presence of a file.
Sources
- Shopify, llms.txt.liquid and agents.md.liquid template documentation, checked 2026-09-09.
- WordPress, nginx server configuration.
- Webflow Help Center, Upload an llms.txt file to your site (as reported by the help centre).
- HubSpot Knowledge Base, Supported file types (as reported by the help centre).
- Crawler documentation: OpenAI, Anthropic, Perplexity.
An earlier version of this guide said writing the file takes about an hour and that its priority order reflected how much each step changes an answer. Neither was measured, so both are gone. The Webflow and HubSpot rows are taken from their help centres rather than tested on a live site.