Adopting llms.txt for my blog

The scenario

From time to time, my blog gets fully crawled. It used to be mostly Google, and a few times Bing. Lately, I still have one or two full-site crawls per month, and I've noticed an increase in requests with referrers such as chatgpt.com, duck.ai, gemini.google.com, perplexity.ai and similar [1]. I do not particularly mind machine-readable access to my content, whether it is used for retrieval, indexing, or potentially training, same as it was in the past helping Google reap more advertising money. I write what I write openly and for any reader, human or digital.

Why does that context matter? Well, I do like to play with new technologies and formats, and I recently came across llms.txt. I also like to tweak my site, an old web-development habit, so why not provide a cleaner, machine-readable version for LLMs? And even more so when I already write the blog posts as Markdown.

What llms.txt provides

The llms.txt proposal defines a file named precisely that way, containing a Markdown description and a list of posts. Like a sitemap.xml, but with a text introduction. It also suggests providing an easily recognizable pattern for LLMs to access the Markdown versions of your pages, for example https://blog.kartones.net/post/a-simple-ai-evals-runner/index.html.md as a companion to https://blog.kartones.net/post/a-simple-ai-evals-runner/ (which really resolves to https://blog.kartones.net/post/a-simple-ai-evals-runner/index.html). I encourage you to directly check how it looks at https://blog.kartones.net/llms.txt. Whether major AI services will consistently use this proposal remains unclear, but it is inexpensive to support, and fun to build.

I recently finished a first sweep of adding tags to every post, and I've been adding AI-generated summaries to every blog post for the last few years. I also back-filled all past entries, so not having a search page for LLMs [2] is not that much of a problem. A single file listing every post with description and tags was quite big, though, so I decided to expose both separately. They are referenced inside llms.txt, but for a quicker look the URLs are https://blog.kartones.net/tags.html.md and https://blog.kartones.net/archives_with_descriptions.md.

My implementation

On the technical side, the static version of the blog is generated via Pelican, which never ceases to amaze me with its simplicity and extensibility. Creating a plugin and a few templates to achieve the LLM-friendly output was very straightforward. I've published the plugin on GitHub.

You can go check it out, it is just ~130 lines of code plus two templates, so I'll instead briefly mention what it does, and how to configure it.

The plugin generates Markdown counterparts for Pelican articles and pages. It can exclude selected individual items and listing pages, such as author and category indexes. It also renders the new llms.txt and the archive with descriptions files, from Pelican templates.

The pelicanconf.py configuration changes, apart from adding llms_export to the list of plugins, are very small:

LLMS_EXPORT_CONTENT_EXCLUSIONS = ["404", "not-found"]
LLMS_EXPORT_LISTING_EXCLUSIONS = ["authors", "categories"]
LLMS_EXPORT_MAX_INDEX_PAGES = 9

TEMPLATE_PAGES = {
    "llms.txt": "llms.txt",
    "archives_with_descriptions.md": "archives_with_descriptions.md",
}

You also need to place the templates inside your Pelican theme folder, and probably tweak them a little.

While the main idea is for LLMs, I believe in open content, so I also added to every article and page a link to the Markdown version in an unobtrusive way, as a <link> inside the header:

Articles:

<link rel="alternate" type="text/markdown" href="{{ SITEURL }}/{{ article.url }}index.html.md">

Pages:

<link rel="alternate" type="text/markdown" href="{{ SITEURL }}/{{ page.url }}index.html.md">

And finally, I made my web server render Markdown files instead of sending them as content to download. For Nginx, this is what you need to add:

location ~* \.md$ {
    default_type text/markdown;
}

Summary

My resulting plugin is a simple and effective addition to the site. Having it does not guarantee that LLM-based services will use the new content format, but it gives them a cleaner option if they do. More importantly, it provides a minimalistic representation of the blog that is potentially useful beyond AI tools or crawlers; Anyone can render it using their preferred tools, process it programmatically, or reuse it in other formats.

Footnotes

[1] : I only store the visitor's browser user agent and referrer URL in my minimalist stats system.

[2] : This blog is a static site, in which I try to avoid adding "backend services" unless critical. There's also no search page for humans at the moment, although that might change in the future.