In under two years since it appeared, the llms.txt file has become the most important instrument for communicating a website’s content towards AI systems, even though how much the crawlers of the big AI companies really use it remains a matter of debate. Maintaining both formats, the concise navigational llms.txt and the complete llms-full.txt, is strategically worthwhile, because each serves a different purpose and a different type of AI agent.
For an e-shop with a large product range there is a proven model of segmenting content by category, used successfully by Cloudflare among others. Tying authorial identity through schema.org sameAs to professional profiles then represents one of the strongest technically measurable trust signals AI systems are able to process [1].
The two-layer architecture of llms.txt and llms-full.txt
The llms.txt specification, proposed in September 2024 by Jeremy Howard of Answer.AI, formally defines only the main llms.txt file, as a curated navigational index in Markdown format. The llms-full.txt file emerged in parallel out of collaboration between Mintlify and Anthropic, who needed a way to hand entire documentation sets to AI models without parsing HTML. Today it is a de facto standard part of the ecosystem, although formally it is not part of the original specification [2].
The two files serve two different scenarios. llms.txt works as a “treasure map”, a brief overview with curated links and descriptions that real-time AI assistants such as Claude or ChatGPT browsing the web, or Perplexity searching, can process quickly and use to decide where to navigate next on the site. llms-full.txt, by contrast, is “the whole treasure at once”, the complete content in a single file, ideal for ingestion pipelines, RAG systems and coding assistants such as Cursor or GitHub Copilot, which need the full context [3].
The ideal size in tokens
The main llms.txt should stay under 10,000 tokens (roughly 7,500 to 9,000 words, about 30 pages of typescript; you can check the exact token count at, for instance, https://platform.openai.com/tokenizer), and some sources recommend as little as 2,000 tokens for optimal processing. For llms-full.txt the situation is looser: Anthropic runs one of nearly 500,000 tokens, Vercel roughly 500,000 tokens and Cloudflare in aggregate as much as 3.7 million tokens. A file up to 100 KB (approximately 33,000 tokens) should be handled without difficulty by every current model [4].
Related: Token economy
It is essential to distinguish between the nominal capacity of the context window and the space you can actually use. There is a well documented “lost in the middle” effect, where models attach greater weight to information at the beginning and the end of a text. That is why the most important information, the main product categories, the e-shop’s unique value proposition, the key pages, belongs at the start of the file [5].
Segmenting llms-full.txt for an e-shop with thousands of products
For an e-shop with around a thousand products, a monolithic llms-full.txt containing everything is impractical and probably counterproductive. Splitting it into several thematic files is not only possible but is established practice, implemented by some of the most sophisticated technology companies.
The best existing example is Cloudflare, which runs one global llms.txt as a central signpost and alongside it dozens of product-specific llms-full.txt files on sub-paths, for example /workers/llms-full.txt, /ai-gateway/llms-full.txt or /agents/llms-full.txt. An AI agent can therefore download only the documentation relevant to its query without having to process a corpus of millions of tokens. The specification explicitly allows this approach when it states that llms.txt may be placed not only in the root directory but optionally in subfolders as well [6].
A recommended architecture for an e-shop
For an e-shop with a thousand products the optimal architecture might look like this: in the root directory sits the main llms.txt containing a brief description of the shop, its focus and its distinctive features, plus links to the category files and to key pages such as buying guides, FAQs, terms and conditions and the about-us page. Each product category then has its own llms-full.txt on the corresponding sub-path.
There are also specialised open-source projects for e-commerce. The site-llms project extends the llms.txt concept with a site-llms.xml file, essentially an XML sitemap pointing to individual llms.txt files for individual products, designed specifically for shops with thousands to millions of items. The alternative CommerceTXT protocol offers a format optimised directly for e-commerce with a declared token reduction of roughly 95 % compared with JSON [7].
It is important to remember that for an e-shop it is neither necessary nor often desirable to include every single product in llms-full.txt as plain text. A more effective approach is to combine category descriptions with links to structured product feeds in JSON format, which AI models process more efficiently than plain text listings. Dell Technologies, one of the first large e-commerce players with an llms.txt, is taking exactly that route [8].
Linking to sitemap.xml from llms.txt
The llms.txt specification addresses the question of the relationship with sitemap.xml explicitly, and treats the two formats as complementary rather than interchangeable. Sitemap.xml is not a substitute for llms.txt, because a sitemap typically does not contain LLM-readable versions of pages, does not include external URLs useful for understanding context, and in aggregate covers documents far too extensive for an AI model’s context window [1].
An analysis of real implementations shows that none of the big companies, Cloudflare, Stripe, Anthropic or Vercel, links to sitemap.xml in its llms.txt. Instead they all use curated lists of links organised into H2 sections with descriptions. Even so there are practical arguments for including a link to the sitemap as one of the links in llms.txt, particularly for e-shops [9].
The recommended approach for an e-shop is a pragmatic compromise: the main body of llms.txt stays curated and structured according to the specification, while at the end of the file, ideally in an ## Optional section (skippable when the context has to be shortened), you can include a link to the product sitemap as a supplementary resource. The priority, however, should be structured feeds in JSON format, which AI models process an order of magnitude more efficiently than crawling hundreds of HTML pages via a sitemap.
A low-risk investment
As part of optimising for generative language models, llms.txt and structured data in general are an easily achievable investment with low risk and potentially high return, even though the ecosystem is still evolving fast. The two-layer architecture of llms.txt plus llms-full.txt covers the needs of both real-time AI assistants and ingestion pipelines. For an extensive product range, segmentation by category with a central signpost is the proven model that eliminates the problem of exceeding context windows. A link to the sitemap has its place in llms.txt as a supplementary resource, not as a replacement for curated selection.
At the same time it has to be stressed that llms.txt and llms-full.txt are important facilitators, but the foundation remains high-quality expert content [12].
Practical free tools: LLMs generator + a WordPress plugin + an ebook on GEO
References and sources
[1] HOWARD, Jeremy. The /llms.txt file [online]. Answer.AI, 2024-09-03 [cit. 2026-02-24]. Available at: https://llmstxt.org/
[2] MINTLIFY. The value of llms.txt: Hype or real? [online]. Mintlify Blog, 2025 [cit. 2026-02-24]. Available at: https://www.mintlify.com/blog/the-value-of-llms-txt-hype-or-real
[3] DAYDREAM JOURNAL. What is LLMs.txt + LLMs-Full.txt? [online]. 2025 [cit. 2026-02-24]. Available at: https://journal.withdaydream.com/p/what-is-llms-txt-llms-full-txt
[4] HITLSEO. llms.txt vs llms-full.txt: The Complete 2025 Guide to AI-Friendly Documentation [online]. hitlseo.ai, 2025 [cit. 2026-02-24]. Available at: https://hitlseo.ai/blog/llms.txt-vs-llms-full.txt-the-complete-2025-guide-to-ai-friendly-documentation/
[5] DATA STUDIOS. How Large Language Models Handle Extended Context Windows [online]. datastudios.org, 2025 [cit. 2026-02-24]. Available at: https://www.datastudios.org/post/ai-how-large-language-models-handle-extended-context-windows-chatgpt-claude-gemini
[6] CLOUDFLARE. AI tooling · Cloudflare Style Guide [online]. Cloudflare Developers, 2025 [cit. 2026-02-24]. Available at: https://developers.cloudflare.com/style-guide/ai-tooling/
[7] COMMERCETXT. CommerceTXT: Transaction-ready context for AI agents [online]. GitHub, 2025 [cit. 2026-02-24]. Available at: https://github.com/commercetxt/commercetxt
[8] BIGCOMMERCE. LLMs.txt for Ecommerce (Get Products Discovered in AI Search) [online]. BigCommerce Blog, 2025 [cit. 2026-02-24]. Available at: https://www.bigcommerce.com/blog/ecommerce-llms-txt/
[9] 365i. Create a Great llms.txt File: What to Include and What to Exclude [online]. 365i.co.uk, 2025-08-27 [cit. 2026-02-24]. Available at: https://www.365i.co.uk/news/2025/08/27/create-a-great-llm-txt-file-what-to-include-what-to-exclude/
[10] AGENXUS. Entity Graphs for Generative Engine Optimization: From Organization to Person Schema [online]. agenxus.com, 2025 [cit. 2026-02-24]. Available at: https://agenxus.com/blog/building-entity-graph-organization-person-schema
[11] YUNG, Aubrey. Person Schema Markup: A Complete Guide with Example [online]. aubreyyung.com, 2025 [cit. 2026-02-24]. Available at: https://aubreyyung.com/person-schema-markup/
[12] AGGARWAL, Pranjal et al. GEO: Generative Engine Optimization [online]. Princeton University / arXiv, 2023 [cit. 2026-02-24]. Available at: https://arxiv.org/pdf/2311.09735[13] SEROUNDTABLE. ChatGPT & Perplexity Treat Structured Data As Text On A Page [online]. seroundtable.com, 2026 [cit. 2026-02-24]. Available at: https://www.seroundtable.com/chatgpt-perplexity-structured-data-text-40862.html
