4 posts
A scorecard that says 87% can hide that 60 cases cannot support it. An LLM evaluation scorecard in ai-eval: every rate with its 95% interval, gates on the lower bound, and a regression gate on the largest drop in the total or any failure mode.
A judge that agrees with your reviewers 60% of the time produces numbers that look like a good one. LLM judge validation in ai-eval: held-out labels, rates computed by calibrate.py, and a lint that fails any gate on an unvalidated judge.
Most LLM test suites measure failures their author imagined. LLM error analysis in ai-eval: read real traces, name failure modes that need three trace references each, and build a dataset with at least 30% real inputs.
A quality number for an AI feature means little when nobody can say who checked the judge, how many cases it rests on or how sure it is. Five questions to ask of any setup, and the route ai-eval takes from traces to a scorecard that can say NOT-EVALUABLE.
