SEO title: llms.txt Explained: Uses, Limits and WordPress Advice
Updated: 10 October 2026
llms.txt is an optional, proposed information index for AI tools. It does not block AI training or guarantee credit when your content is used.
For a small business owner, the practical question is whether a particular tool or workflow would benefit from the file. Adding one simply because you have heard that it is essential for AI SEO is not a sound reason to spend time on it.
What is llms.txt
Read Jeremy Howard’s llms.txt proposal
When might it be useful
Software documentation is a prominent use case. The proposal also supports other sites with information an agent needs to navigate. It remains a proposal, rather than a requirement followed by every AI service.
Consider a fictional business with a large library of equipment instructions. Its support team uses an assistant that can read a curated index. The team could test whether that index helps the assistant find the correct installation guide for a particular model.
The useful outcome would be fewer wrong documents selected and less time spent searching. Merely publishing a file would not establish either result. The business would need to compare the assistant’s behaviour with and without the index.
What it does not control
llms.txt is not an access-control or training-permissions standard. It cannot guarantee that an AI service will read the file, follow its guidance or cite your business.
Adding wording such as “no training” or “always link to this website” does not turn an information file into an enforceable instruction. A business can publish its preferred terms, but should not treat those words as a technical block.
For this reason, we recommend keeping information publishing and crawler settings as separate tasks. First decide what public material helps users. Then decide which documented controls match your preferences for crawling, search and training.
Does llms.txt help Google rankings or AI Overviews
Google states that Google Search does not use llms.txt files. Creating one does not help or harm your rankings or visibility in Google Search, including its generative AI features. A file being discovered or indexed does not mean Google gives it special treatment.
Google’s guidance instead emphasises useful original content, crawlable pages and the existing technical requirements for Search. There is no special llms.txt requirement for appearing in AI Overviews or AI Mode.
Google’s guidance on generative AI features in Search
Our recommendation for a typical small business is to prioritise clear service information, accurate business details and a website customers can use easily. If your booking page is confusing or your product information is outdated, address those problems before experimenting with an optional file.
How llms.txt differs from other website files
| File or control | Main role |
| llms.txt | A curated information index for tools that use the proposal. |
| XML sitemap | Lists website URLs to help search engines discover pages. |
| robots.txt | Communicates crawling rules to automated clients that respect them. |
| Provider controls | Manage specific uses through the mechanisms documented by that service. |
These tools solve different problems. A working sitemap does not demonstrate that a crawler can access every page, and a crawler being allowed to visit does not guarantee that the page will be selected for an answer.
Using robots.txt and provider controls
robots.txt provides crawling instructions. Responsible crawlers can respect them, but the file is not a security barrier. Google also explains that a blocked URL can still appear in Search without its content being crawled. Keep confidential information behind proper access controls, such as authentication.
Google’s introduction to robots.txt
OpenAI separates search from training
OpenAI documents different agents for different purposes:
- OAI-SearchBot supports ChatGPT search results.
- GPTBot crawls content that may be used to train OpenAI’s generative AI foundation models.
- ChatGPT-User handles some visits initiated by users. It is not the automatic search crawler, and robots.txt rules may not apply to these user-initiated requests.
OpenAI allows search and training preferences to be set independently. For example, a website can allow OAI-SearchBot while disallowing GPTBot. Allowing search access does not guarantee inclusion or citations.
OpenAI’s crawler documentation
Google controls have their own scope
Google-Extended controls certain uses of crawled content for Gemini model training and grounding in specified Gemini and Vertex AI products. Grounding means using retrieved information to inform an answer. Google says it does not affect inclusion or rankings in Google Search. It should not be treated as a universal switch for all Google AI features.
Google’s documentation for Google-Extended
Before changing a setting, write down the intended result: for example, retaining search discovery while opting out of a particular provider’s training crawl. Match that result to the provider’s current documentation. Broadly blocking anything labelled “AI” can create effects you did not intend.
A short decision checklist for WordPress owners
Work through these questions before adding a plugin or creating a file:
- What am I trying to achieve? If the answer is blocking training, work on documented provider controls. If it is helping a specific assistant find information, an llms.txt experiment may be relevant.
- Does the intended tool actually use the file? Ask the tool supplier or check its documentation. Do not assume support because another website publishes one.
- Are the important public pages accurate and accessible? Check the pages themselves before building an index of them.
- Can I keep the file current? Choose a person to review it when services, policies or key URLs change.
- What would show that it helped? Define a small test, such as whether an assistant consistently retrieves the correct support guide.
If you have no supported use case and no useful test, leaving llms.txt out is a reasonable decision. Spend the time on the website improvements your customers need.
If you decide to add the file
Treat this as a small experiment. You do not need to commission a large “AI optimisation” project to establish whether an information index is useful in your workflow.
- Follow the proposal’s format: a site heading, with an optional summary and grouped links. Select public resources relevant to your test.
- Check what your existing WordPress tools provide. If a plugin generates a file, read the output rather than assuming it is accurate. Avoid adding a second generator for the same URL.
- Publish using a suitable plugin or your hosting file tools. Agree which method owns the file so a generated version does not unexpectedly replace a manually maintained one.
- Open the file URL while signed out. Confirm it returns the intended text rather than an error, login screen or security challenge. Test every included link.
- Keep private content out of the file. Review linked exports or Markdown versions too, if your implementation produces them.
- Run the test you defined, record the result and review the file when its linked information changes.
For an equipment supplier, a focused trial might cover one product range and five common support questions. Record whether the assistant chooses the correct instructions, whether its answers are accurate and how much maintenance the index requires. Expand only if the results justify the work.
Our recommendation
For most small business WordPress websites, llms.txt is an optional experiment with a specific use case. Clear content, reliable public information and correctly configured crawler controls deserve attention first.
Use the file when you can identify a tool that benefits from it and a practical result to measure. If your concern is training or access, use the controls documented for that purpose and check what each setting actually affects.
Sources and further reading
Jeremy Howard — The llms.txt proposal
Google — Optimising for generative AI features
Google — Introduction to robots.txt
Google — Common crawlers and Google-Extended
Provider documentation and supported features can change. The sources above were reviewed on 11 October 2026.


















