LLM evaluation for non-engineers: how to check AI answers before you use them

Why AI answers need checking

Language models write fluent text whether or not it is true. In January 2024, Stanford RegLab and HAI researchers reported that the models they tested (GPT-3.5, Llama 2 and PaLM 2) hallucinated on specific legal queries at rates of 69% to 88%, and at least 75% of the time when asked about a court's core ruling. Models have improved since, but the pattern holds: specific facts, citations and niche topics are where errors hide.

For repeated tasks, the engineering version of this check is an eval. Anthropic defines an eval as "a test for an AI system": an input, and grading logic applied to the output. OpenAI's evaluation best practices describe the same loop: define the objective, collect a dataset, define metrics, run and compare, and keep evaluating as things change. NIST's AI Risk Management Framework names measurement as one of its four functions.

Why a small team should care

• The person sending is responsible. "The AI wrote it" doesn't help with a client.
• Errors look like good work. A wrong figure in a confident paragraph is easy to miss.
• Shared prompts multiply mistakes. One flawed team skill produces the same error for everyone, which is why shared skills deserve a test set.

A worked example

Illustration: a made-up business. Lumen Design, a six-person studio, asks an AI for a short market summary for a pitch deck. The draft cites "a 2025 Gartner report" with a market size and quotes a competitor's founder.

The designer runs a review before using it. She lists five claims: the market size, the report, the quote, two competitor names. The report can't be found under that title; the quote appears nowhere online; one competitor closed last year. The market size is replaced with a figure from a source she can open, the quote is cut, and the competitor list is fixed. The final paragraph is shorter and every claim in it has a link.

How to review an AI answer

1. Re-read your request and list what the answer had to include.
2. Mark every specific claim: names, numbers, dates, quotes, laws, prices.
3. Verify each claim in a source you open yourself; delete or fix what you can't confirm.
4. Check tone and promises: nothing your company can't deliver, nothing a customer could read as a commitment.
5. If the task repeats, save the request and the corrected answer as a test case for an eval set.

Common confusions

• "It cited a source, so it's right." Open the source. Citations can be invented or misread.
• "Benchmarks show the model is accurate." Benchmarks measure general tasks, not your client's numbers.
• "Reviewing every answer doesn't scale." Review by risk: always for anything a customer sees, lightly for internal notes.

Questions

How do I know if an AI answer is correct?

Check each factual claim against a source you trust, open every link or citation it gives, and compare the answer with what you asked for. Treat names, numbers, dates, quotes and legal or medical statements as unverified until checked.

How often do AI models make things up?

It depends on the task. A Stanford RegLab and HAI study published in January 2024 found hallucination rates of 69% to 88% on specific legal queries for the models it tested (GPT-3.5, Llama 2 and PaLM 2). Newer models do better on many tasks, but specific facts still need checking.

What is the difference between reviewing an answer and running evals?

Reviewing checks one output before you use it. Evals test the same task on 20 or more saved cases, so you can tell whether a prompt, skill or model change made answers better or worse overall.