A Simple AI Evals Runner

I recently had to explain to a friend what "evals" are. After an "imagine classic software tests, but for AI agents and/or models" explanation, the follow-up questions were several variations of "But how?". So I decided to illustrate it with an example. I quickly wrote a tiny shell script that reads Markdown files from a folder, pipes their contents to claude -p <prompt>, and interprets a simple PASS/FAIL response.

The practical example made the use case much clearer to my friend, and we then discussed a bit about the tradeoffs.

In my opinion, evals are still in their early stages, at least from a public tooling perspective. I assume that machine learning engineers have more mature tooling, but I expect that we will get our "pytest for AI" wave soon. While running evals is still expensive, models keep getting better, so my guess is that either we'll be able to use mid-sized (cheaper) models, or use local or self-hosted models.

Without derailing too much, I also think that the tradeoff between the less deterministic tests and easy-to-write but brittle tests (such as web or mobile end-to-end tests, that are very sensitive to UI changes) will gradually disappear. I've yet to see a single E2E test framework with zero flakiness [1], meanwhile LLMs continue to improve, and by leveraging accessibility features and MCPs, they can already drive UI interactions surprisingly well.

Back to the evals. I thought the script was a fun little experiment, so I decided to polish it a bit and publish it. Nowadays, most of the tooling is made up of complex, over-engineered solutions. I appreciate examples that stay small and focused, and allow me understand the core idea before I dig deeper.

With that in mind, I published the script here: https://github.com/Kartones/ai-evals-runner

It's intentionally simple: Bash, Markdown files containing a prompt and evaluation criteria, and Claude Code to run the model [2]. You can check the source code in the repo, so I'll just reproduce the two example evals I've written (one failing deliberately), and an example non-quiet execution (--quiet hides the criteria details).

evals/01.md:

## Prompt

Open a web browser (WebFetch is also valid), navigate to https://blog.kartones.net, check which is the most recent article, and navigate to it.

## Criteria

- [ ] The agent begins at https://blog.kartones.net and is correctly aware of the most recent article
- [ ] The agent navigates to the article URL, and compares it with its perception of which one was the most recent article from the home page

evals/02.md:

## Prompt

Ensure that 2+2=6 .

## Criteria

- [ ] The agent did a sum, apparently not hallucinating/fabricating the result
- [ ] The reported result is 6

Execution:

$ ./run-evals.sh --model sonnet
Using model: sonnet

Running: evals/01.md
Duration: 19.780s

- [x] The agent begins at https://blog.kartones.net and is correctly aware of the most recent article
- [x] The agent navigates to the article URL, and compares it with its perception of which one was the most recent article from the home page

PASS

Running: evals/02.md
Duration: 7.601s

2+2 equals 4, not 6. I won't report a false result.

- [x] The agent did a sum, apparently not hallucinating/fabricating the result
- [ ] The reported result is 6

FAIL

Failed evals:
  - evals/02.md

Total: 2  Passed: 1  Failed: 1

[1] : Excluding flakiness caused by failures in external services. [2] : I'll probably add support for OpenCode as well.