What AI evaluation means
AI evaluation is the systematic measurement of how well an AI system performs against defined quality criteria and use cases.
Define quality dimensions first
Before selecting metrics, define what “good” means for the product.
- Correctness
- Relevance
- Completeness
- Groundedness
- Safety
- Tone
- Latency
- Cost
Build an evaluation dataset
An evaluation set should represent real usage: common questions, critical workflows, edge cases, ambiguous prompts, difficult retrieval cases, unsafe requests, and known historical failures.
Choose evaluation methods
| Method | Strength | Limitation |
|---|---|---|
| Reference answer | Objective for known facts | Not every valid answer is identical |
| Rule-based metric | Fast and repeatable | Limited for nuanced quality |
| Model grader | Scalable qualitative scoring | Requires calibration and oversight |
| Human reviewer | Strong business judgment | Costly and potentially inconsistent |
Use rubrics
A rubric translates vague judgment into repeatable criteria. Example relevance rubric:
| Score | Interpretation |
|---|---|
| 5 | Directly answers the question with no material distraction |
| 3 | Partially answers but misses important context |
| 1 | Mostly irrelevant or fails to address the request |
Establish thresholds
Release criteria should define acceptable aggregate and critical-case performance. Averages alone can hide severe failures on high-risk scenarios.
Use evaluation for regression
Track quality over time
Store versioned results so teams can see whether changes improve one dimension while degrading another—for example, higher groundedness but worse latency or completeness.