interactive · 12 min

An Eval Harness

Objective: run an agent over a labelled set of cases and compute a pass rate — the smallest useful evaluation.

MockLLM is the same Foundations seam; here it’s the system under test. The harness is just: run every case, compare to the label, aggregate. Everything sophisticated (judges, trajectory checks) is a richer comparator wrapped around this same loop.

The same loop measures retrieval recall@k: the case label is whether the expected document appears in the top-k retrieved, and the comparator checks membership instead of equality.

Best practice: start every agent with a tiny golden set and a pass rate in CI; grow the set from every production failure.

Next: Evaluation Pitfalls