A team ships an LLM-powered customer support agent. Their evaluation dashboard shows 4.2 out of 5 across eight dimensions — helpfulness, accuracy, tone, completeness, relevance, safety, conciseness, and fluency. Leadership is satisfied. Two weeks later, churn spikes. The agent was confidently fabricating return policies that didn’t exist. None of the eight dimensions caught it.

This is the evaluation trap that nearly every AI team falls into. Not the absence of evals — the presence of bad ones.

The fix is not more metrics. It is a fundamentally different methodology.

The Verifier’s Rule

Before diving into practice, the theory. Jason Wei’s “verifier’s rule” states that the ease of training AI to solve a task is proportional to how verifiable the task is. Every task that is possible to solve and easy to verify will eventually be solved by AI.

Five properties make a task optimizable:

  1. Objective truth — everyone agrees what good solutions are
  2. Fast to verify — any given solution can be verified in seconds
  3. Scalable to verify — many solutions can be verified simultaneously
  4. Low noise — verification correlates tightly with solution quality
  5. Continuous reward — easy to rank solutions by quality

The implication for eval builders: you can improve verifiability by front-loading work. Competition math becomes trivial to check with an answer key. Coding problems become verifiable with test cases. LLM outputs become verifiable when you define pass/fail criteria with enough specificity that a new employee could apply them consistently.

This is exactly what good evals do — they front-load the effort of defining “good” so that ongoing verification becomes cheap and scalable.

The Core Problem: Criteria Drift

The seminal finding in evaluation research comes from Shankar et al. at UC Berkeley (UIST 2024):

“To grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria. We dub this phenomenon criteria drift.”

This means it is impossible to write the perfect rubric before you start grading. The criteria evolve as you see real data. Teams that spend weeks defining evaluation frameworks before looking at a single LLM output are building on sand.

The practical consequence: stop trying to write the perfect rubric upfront. Start grading, and let the criteria emerge from the data.

The Critique Shadowing Methodology

The most battle-tested methodology for building aligned LLM judges is Critique Shadowing, developed by Hamel Husain across 30+ AI implementations. It has six steps.

Step 1: Find the Principal Domain Expert

In every organization, one or two individuals have the judgment that matters most for your AI product. Not the engineering lead. Not the VP. The person who actually knows what good looks like.

  • A psychologist for a mental health AI assistant
  • A lawyer for a legal document analyzer
  • A customer service director for a support chatbot
  • A quality manager for a GxP compliance tool

Why this matters: they set the standard, capture unspoken expectations, ensure consistency, and develop ownership. In a smaller company, this might be the founder. If you’re an independent developer, you should be the domain expert — but be honest about your expertise.

Many developers attempt to act as the domain expert themselves. This is a recipe for disaster.

Step 2: Create a Dataset Across Meaningful Dimensions

Structure your test data across dimensions that matter for your use case:

  • Features — specific functionalities (order tracking, document search, refund processing)
  • Scenarios — situations the AI must handle (no matches found, ambiguous request, system error, invalid data)
  • Personas — user profiles (new user, expert, non-native speaker, busy professional)

Generate enough data to cover all dimension combinations. Use real user interactions where available, supplement with synthetic data. For synthetic data, use LLMs to generate only the user inputs — feed those into your actual AI system to get real responses.

How much data? Start with 200+ samples, aim for 50-100 fail cases. A dataset with hundreds of labels but only five failures is not useful. Smaller, less capable models naturally produce “organic” failures that are far more valuable than synthetic defects generated by strong models.

Step 3: Binary Pass/Fail with Critiques

This is where most teams go wrong. They create elaborate scoring systems — 1-5 scales across eight dimensions — and end up with dashboards full of numbers nobody acts on.

The domain expert’s job is to answer one question: “Did the AI achieve the desired outcome?”

Pass or fail. Nothing else.

Alongside the binary decision, the expert writes a detailed critique explaining their reasoning. These critiques:

  • Capture nuances that a binary label cannot
  • Provide specific, actionable improvement guidance
  • Serve as few-shot examples for the LLM judge
  • Should be detailed enough that a new employee could understand them

A common objection: “The business said these 8 dimensions are important.” If someone says you need to measure 8 things on a 1-5 scale, they don’t know what they’re looking for. They’re guessing. Let the domain expert drive with pass/fail and critiques, and you’ll discover what actually matters.

From Eugene Yan’s experience across dozens of product teams: stakeholders sometimes ask for granular scores so they can “adjust thresholds later.” In practice, zero of them actually do. They eventually ask for a pass/fail rate. Start where you’ll end up.

Step 4: Build the LLM Judge from Expert Examples

You cannot write a good judge prompt until you’ve seen the data. This is the operational consequence of criteria drift.

Start with the expert’s critiques as few-shot examples in the judge prompt. A real-world prompt structure:

You are evaluating [application] responses.

Context: [what the application does, domain knowledge needed]

Criterion: [one precise criterion, including what to ignore]

Example (fail): [input] → [output] → Reasoning: [why it fails] → Verdict: fail
Example (pass): [input] → [output] → Reasoning: [why it passes] → Verdict: pass

Evaluate the reply below. Write your reasoning first, then output
exactly one of: pass, fail, unknown.

Five critical elements:

  1. Context — what the application does and domain knowledge needed
  2. One precise criterion — including what to ignore
  3. Labeled examples — 2-4 cases with expert reasoning
  4. Reasoning first, verdict last — measurably improves judge accuracy
  5. An explicit way out — “unknown” when information is missing

Step 5: Iterate Until Convergence

Send the domain expert a spreadsheet with the LLM’s critiques alongside their own. Use this to iteratively improve the prompt. Track agreement rates over time.

In the Honeycomb case study, it took three iterations to achieve >90% agreement between the LLM judge and the domain expert. The expert, Phillip Carter, reported:

“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.”

The process of building the judge standardized the evaluation criteria. This is the hidden value — the judge is a “hack” to force careful data examination.

Step 6: Error Analysis

Once you have an aligned judge, apply it to production data. Calculate error rates across your dimensions (feature × scenario × persona). Classify traces by root cause. Build a distribution of errors.

A typical root cause distribution might reveal: 40% missing user education, 30% authentication issues, 20% poor context handling, 10% inadequate error messages. Now you know where to focus.

Start with 30 examples for error analysis. Keep going until you stop seeing new failure modes. After the first pass, focus on errors rather than random sampling — search for more examples that trigger the same pattern.

The God Evaluator Anti-Pattern

The single most common mistake in evaluation design is the “God Evaluator” — one LLM judge that rates accuracy, tone, completeness, relevance, safety, and conciseness in a single prompt on a 1-10 scale.

This never works. The resulting score doesn’t tell you what to fix. The judge is impossible to calibrate because you can’t isolate which dimension is misaligned. And the scores don’t map to actionable decisions.

Build individual evaluators per criterion and combine them via simple heuristics (the output passes only if all dimensions pass). This gives you granular metrics — you can see exactly which dimension is dragging down performance. Some dimensions are guardrails (must-pass for shipping); others are north-stars (aspirational improvement targets). Treating them identically obscures both.

Prefer Deterministic Where Possible

Code Evaluator LLM-as-Judge
Cost cheap expensive
Speed milliseconds seconds to minutes
Consistency always same verdict varies between runs
Scope structure, state, comparisons meaning, relevance, tone

If the thing you’re evaluating is visible in your system state (a row was written, a ticket was closed, a refund was processed) or comparable against an expected output — use code. It’s fast, cheap, deterministic, and easy to debug.

Save LLM-as-judge for things only language can assess: whether a response is empathetic, whether a summary captures the right nuance, whether an explanation is clear to a non-expert.

The critical rule from Anthropic’s agent evaluation guide: grade the outcome in the environment, not the claim in the transcript. A support agent might end a conversation with “Your refund of $200 has been processed, you’re all set!” while no refund exists in the database. Check the refunds table.

Agent Evaluations Are Harder

Agents operate over many turns, modify state, and adapt based on intermediate results. Mistakes propagate. Creative solutions can “fail” static evals while actually being superior.

Anthropic’s Opus 4.5 initially scored 42% on CORE-Bench. After fixing grading bugs, ambiguous task specs, and stochastic tasks, the score jumped to 95%. The eval was broken, not the model. METR discovered tasks that penalized models for following instructions — the grading required exceeding a stated threshold, not meeting it.

For agents, you need three types of graders working together:

Code-based graders for deterministic checks — unit tests, state verification, tool call validation. Fast, cheap, reproducible.

Model-based graders for nuanced assessment — rubric scoring, natural language assertions, pairwise comparison. Flexible but requires calibration.

Human graders for gold-standard calibration — SME review, inter-annotator agreement. Expensive but irreplaceable for setting the standard.

And you need to handle non-determinism. Two metrics help:

  • pass@k — the likelihood of at least one success in k attempts (for tools where one success matters)
  • pass^k — the probability that all k trials succeed (for agents where consistency is essential)

At k=1, they’re identical. By k=10, they tell opposite stories. Choose based on your product requirements.

Position Bias and Perturbation

LLM judges have systematic biases that must be tested:

  • Position bias — preference for outputs in certain positions
  • Verbosity bias — preference for longer responses
  • Self-enhancement — models rate their own outputs higher
  • Number bias — GPT-3.5 prefers the number 7; GPT models favor 42

Mitigations:

  • Run pairwise evaluations twice with order swapped
  • Use binary/categorical scores instead of scales
  • Conduct perturbation studies — swap positions, invert scoring systems
  • If the judge still prefers the same output after perturbation, it may be effective

Prompt sensitivity is extreme. Formatting variations can cause performance differences of up to 76 accuracy points. Use string concatenation instead of triple-quote multi-line strings. Be meticulous about character casing, spacing, and string parsing.

The Evaluation Stack

No single method catches everything. The effective approach is a Swiss Cheese Model — multiple layers where failures that slip through one are caught by another:

Automated evals run on every commit — fast iteration, regression testing, no user impact. But they can miss real usage patterns.

Production monitoring tracks live metrics — ground truth on actual behavior. But it’s reactive; problems reach users before you know about them.

A/B testing measures real user outcomes at scale. But it’s slow — days or weeks for significance.

User feedback surfaces unexpected problems. But it’s sparse, self-selected, and rarely explains why.

Manual transcript review builds intuition for failure modes. But it doesn’t scale.

The most effective teams combine all of these. Automated evals for CI/CD, production monitoring for ground truth, periodic human review for calibration.

Capability and Regression Evals

Two types of evals serve different purposes:

Capability evals ask “what can this agent do well?” They start at a low pass rate — this is the hill to climb. As the agent improves, pass rates climb.

Regression evals ask “does the agent still handle all the tasks it used to?” They should be near 100%. A decline signals something broke.

As capability evals reach high pass rates, they “graduate” to the regression suite. Tasks that once measured “can we do this at all?” now measure “can we still do this reliably?”

Watch for eval saturation — an eval at 100% tracks regressions but provides no signal for improvement. SWE-bench Verified started at 30% this year; frontier models now exceed 80%. As evals saturate, only the hardest tasks remain, and large capability improvements appear as small score increases.

The Statistical Reality

How many samples do you need? It depends on your confidence requirements.

If your product requires a defect rate below 5%, and you run on 200 samples observing 3% defects, your 95% confidence interval is roughly 3% ± 2.4%. The upper bound of 5.4% exceeds your requirement — you can’t confidently claim compliance.

Double to 400 samples and the interval shrinks to 3% ± 1.7%. Now the upper bound of 4.7% is below 5%. Standard error decreases proportionally to the square root of sample size — to halve the margin of error, quadruple the sample size.

The Real Value

A team invested four weeks building their evaluation harness — defining criteria, collecting annotations, aligning evaluators, building the experiment pipeline. Stakeholders worried it was a distraction.

In the next two weeks, they ran dozens of experiments across models, retrieval configurations, and prompt templates. In the next few months, hundreds more. This would have been impossible bottlenecked on human annotations after each change.

The benefit of evaluation is not just measuring quality. It is tightening the feedback loop so you can iterate faster.

But here is the deeper truth, stated by the practitioner who has done this more than anyone: the real business value comes from looking at your data. Building an LLM judge is a “hack” to trick people into carefully examining their data. The judge is a byproduct. The data literacy is the prize.

Research notes: [[Writing Good Evals - Comprehensive Report]]