What are evals, and how does a small team test its AI?

What evals are

Anthropic's Demystifying evals for AI agents (9 January 2026) opens with the definition: "An evaluation ('eval') is a test for an AI system": give the AI an input, then apply grading logic to its output to measure success. It describes three kinds of grader. Code-based graders (string matches, rules, tests) are fast, cheap and reproducible but break on valid variations. Model-based graders (a rubric scored by another model, or a side-by-side comparison) capture nuance but are non-deterministic and must be calibrated against people. Human graders, such as a subject expert's review, are the gold standard and the slowest.

The guide also separates two purposes. Capability evals ask what the agent can do well and should start with a low pass rate, so there is room to improve. Regression evals ask whether it still handles everything it used to, and should pass nearly 100% of the time. OpenAI's evaluation best practices make a similar point: define the task and success criteria first, collect a dataset that reflects real use, and run it continuously as things change.

Why a small team should care

• Every change is a gamble without them. A new model version, a reworded instruction or a new teammate's tweak can fix one answer and break three.
• Your failures are the dataset. The replies a customer complained about and the summaries someone had to rewrite are the best test cases you will get.
• It settles debates. "Is the new version better?" becomes "it passes 18 of 20 instead of 14".

A worked example

Illustration: a made-up business. Kestrel Clinic Software, a 12-person company, uses an AI skill to draft replies to support tickets. The support lead collects 20 tickets: 12 ordinary ones, five edge cases (a refund outside policy, a ticket in Spanish, a customer who is angry) and three adversarial ones (a request for another customer's data, a request to ignore the rules). For each, she writes what a passing reply must and must not do.

The current skill passes 13. The failures cluster: it promises refunds and it answers in English when the customer wrote in Spanish. Two rule changes later, it passes 19. The twentieth, the request for another customer's data, needs a firmer refusal, which she adds. From then on, any change to the skill is run against the 20 cases before it reaches the team.

How to build a first eval set

1. Write the task and what success means in two or three sentences.
2. Collect 20 real cases: mostly normal ones, plus edge cases and a few that try to break the rules.
3. Write the expected result or a short rubric for each: must include, must not include.
4. Run the current version and grade it, by hand at first.
5. Re-run after every change and keep the cases that pass as your regression set.

Common confusions

• "Evals are benchmarks." Public benchmarks compare models in general. Your evals test your task, your rules and your customers.
• "Let the AI grade itself." A model grader is useful, but check it against a person's grades before trusting it.
• "We'll do evals when we're bigger." Twenty cases take an afternoon. They matter most when a team shares one skill, because one edit reaches everyone.

Questions

What is an eval in AI?

A test for an AI system: you give it an input and apply grading logic to its output to measure success. That is Anthropic's definition in its January 2026 guide. A set of evals is a list of such cases you can run again whenever something changes.

How many test cases do I need to start?

Anthropic's guide says 20 to 50 simple tasks drawn from real failures is a great start, and advises not waiting until you have hundreds.

Who grades an eval?

Code (exact matches, rules), another model with a rubric, or a person. Code is fast and cheap but brittle; a model grader is flexible but needs checking against human judgement; people are the gold standard and the slowest.