Title: Adopting llms.txt for my blog
Slug: adopting-llms-txt-for-my-blog
Date: 2026-07-20 20:00:00
Author: Kartones
Lang: en
Tags: Development, Python, AI & ML, Patterns & Practices
og_image: 
Description: How I added llms.txt and Markdown exports to my Pelican blog, giving LLMs and other tools a cleaner, machine-readable way to access its content.


### The scenario

From time to time, my blog gets fully crawled. It used to be mostly Google, and a few times Bing. Lately, I still have one or two full-site crawls per month, and I've noticed an increase in requests with referrers such as `chatgpt.com`, `duck.ai`, `gemini.google.com`, `perplexity.ai` and similar [1]. I do not particularly mind machine-readable access to my content, whether it is used for retrieval, indexing, or potentially training, same as it was in the past helping Google reap more advertising money. I write what I write openly and for any reader, human or digital.

Why does that context matter? Well, I do like to play with new technologies and formats, and I recently came across [llms.txt](https://llmstxt.org/). I also like to tweak my site, an old web-development habit, so why not provide a cleaner, machine-readable version for LLMs? And even more so when I already write the blog posts as Markdown.

### What `llms.txt` provides

The `llms.txt` proposal defines a file named precisely that way, containing a Markdown description and a list of posts. Like a `sitemap.xml`, but with a text introduction. It also suggests providing an easily recognizable pattern for LLMs to access the Markdown versions of your pages, for example `https://blog.kartones.net/post/a-simple-ai-evals-runner/index.html.md` as a companion to `https://blog.kartones.net/post/a-simple-ai-evals-runner/` (which really resolves to `https://blog.kartones.net/post/a-simple-ai-evals-runner/index.html`). I encourage you to directly check how it looks at [https://blog.kartones.net/llms.txt](https://blog.kartones.net/llms.txt). Whether major AI services will consistently use this proposal remains unclear, but it is inexpensive to support, and fun to build.

I recently finished a first sweep of adding tags to every post, and I've been adding AI-generated summaries to every blog post for the last few years. I also back-filled all past entries, so not having a search page for LLMs [2] is not that much of a problem. A single file listing every post with description *and* tags was quite big, though, so I decided to expose both separately. They are referenced inside `llms.txt`, but for a quicker look the URLs are [https://blog.kartones.net/tags.html.md](https://blog.kartones.net/tags.html.md) and [https://blog.kartones.net/archives_with_descriptions.md](https://blog.kartones.net/archives_with_descriptions.md).

### My implementation

On the technical side, the static version of the blog is generated via [Pelican](https://getpelican.com/), which never ceases to amaze me with its simplicity *and* extensibility. Creating a plugin and a few templates to achieve the LLM-friendly output was very straightforward. I've published the plugin [on GitHub](https://github.com/Kartones/python/tree/master/pelican/plugins/llms_export/).

You can go check it out, it is just ~130 lines of code plus two templates, so I'll instead briefly mention what it does, and how to configure it.

The plugin generates Markdown counterparts for Pelican articles and pages. It can exclude selected individual items and listing pages, such as author and category indexes. It also renders the new `llms.txt` and the archive with descriptions files, from Pelican templates.

The `pelicanconf.py` configuration changes, apart from adding `llms_export` to the list of plugins, are very small:

```python
LLMS_EXPORT_CONTENT_EXCLUSIONS = ["404", "not-found"]
LLMS_EXPORT_LISTING_EXCLUSIONS = ["authors", "categories"]
LLMS_EXPORT_MAX_INDEX_PAGES = 9

TEMPLATE_PAGES = {
    "llms.txt": "llms.txt",
    "archives_with_descriptions.md": "archives_with_descriptions.md",
}
```

You also need to place [the templates](https://github.com/Kartones/python/tree/master/pelican/plugins/llms_export/_templates_) inside your Pelican theme folder, and probably tweak them a little.

While the main idea is for LLMs, I believe in open content, so I also added to every article and page a link to the Markdown version in an unobtrusive way, as a `<link>` inside the header:

Articles:

```html
<link rel="alternate" type="text/markdown" href="{{ SITEURL }}/{{ article.url }}index.html.md">
```

Pages:

```html
<link rel="alternate" type="text/markdown" href="{{ SITEURL }}/{{ page.url }}index.html.md">
```


And finally, I made my web server render Markdown files instead of sending them as content to download. For Nginx, this is what you need to add:

```
location ~* \.md$ {
    default_type text/markdown;
}
```

### Summary 

My resulting plugin is a simple and effective addition to the site. Having it does not guarantee that LLM-based services will use the new content format, but it gives them a cleaner option if they do. More importantly, it provides a minimalistic representation of the blog that is potentially useful beyond AI tools or crawlers; Anyone can render it using their preferred tools, process it programmatically, or reuse it in other formats.


### Footnotes


[1] : I only store the visitor's browser user agent and referrer URL in my minimalist stats system.

[2] : This blog is a static site, in which I try to avoid adding "backend services" unless critical. There's also no search page for humans at the moment, although that might change in the future.