Evaluating AI Features with ai-eval
- 1Part 1·4 min read·BeginnerUp next
An AI Eval Pipeline That Can Say NOT-EVALUABLE
A quality number for an AI feature means little when nobody can say who checked the judge, how many cases it rests on or how sure it is. Five questions to ask of any setup, and the route ai-eval takes from traces to a scorecard that can say NOT-EVALUABLE.
- 2Part 2·4 min read·Intermediate
Read 100 Real Traces Before You Write One Test
Most LLM test suites measure failures their author imagined. LLM error analysis in ai-eval: read real traces, name failure modes that need three trace references each, and build a dataset with at least 30% real inputs.
- 3Part 3·4 min read·Intermediate
Who Checks the AI Judge? Calibrate It Before It Gates
A judge that agrees with your reviewers 60% of the time produces numbers that look like a good one. LLM judge validation in ai-eval: held-out labels, rates computed by calibrate.py, and a lint that fails any gate on an unvalidated judge.
- 4Part 4·4 min read·Advanced
Read the Scorecard: Intervals, Gates and a Frozen Baseline
A scorecard that says 87% can hide that 60 cases cannot support it. An LLM evaluation scorecard in ai-eval: every rate with its 95% interval, gates on the lower bound, and a regression gate on the largest drop in the total or any failure mode.