quiz · 4 min Evaluation & Testing Check Why are task-success metrics alone not sufficient for agentic systems?They require an LLM judge and are therefore too expensiveThey only work when the model's temperature and seed are pinnedThey cannot be compared with string equality and must use regexAn agent can produce the right answer by accident or via a flawed path that will fail laterWhat is eval overfitting, and what is the prescribed fix?Tuning the agent's instructions against the same cases you measure on; fix by holding out cases the prompt author never seesRunning too many eval cases so the suite is slow; fix by trimming the golden set to only edge casesUsing an LLM judge whose temperature is not pinned; fix by routing evaluation through a deterministic mockLetting the golden set grow stale relative to production traffic; fix by adding production failures to the setIn the eval harness built in lesson 31, what is the core loop structure?Compare agent outputs pairwise and keep the better result each roundAsk a judge model to rate each case on a rubric, then average the scoresRun every case, compare each result to its label, aggregate into a pass rateRetrieve the top-k documents and check whether the expected answer appearsAccording to lesson 32, what is the correct way to handle non-deterministic agent outputs in evaluation?Accept some flakiness because real agents must be tested against real model outputsRoute evaluation through a deterministic mock, or pin temperature and seed when the real model must be usedRun the golden set only once per release to avoid seeing varianceSwitch to an LLM judge, which averages out variance across multiple verdictsWhat is eval-set drift, and why does a stable golden-set pass rate not protect against it?The golden set no longer reflects the distribution of real production inputs because production traffic shifts after launch; pass rate on the original set stays high while production error rate risesThe eval set grows so large that CI takes too long to run it; pass rate appears stable but test coverage thins outThe agent prompt is updated but the golden set labels are not; the comparator returns stale matchesLLM-judge verdicts shift when the underlying judge model is upgraded; stable scores hide model driftSubmit Mark complete ← Evaluation Pitfalls Input & Output Guardrails →