Definition
An Evaluation Harness is an automated software testing suite that continuously benchmarks generative AI models and agents against standardized datasets, rubric criteria, and golden reference examples.
Purpose
Quantifies generative quality, detects performance regressions across prompt adjustments or model upgrades, and ensures production reliability.
Key Concepts
How It Works
The harness runs hundreds of deterministic scenarios against the AI system, scoring outputs using automated assertions, reference comparisons, and model-based evaluators.
Related Terms
Sources
- OpenAI Evals Framework
- Thoughtworks, Testing and Evaluating AI Agents