A QA manager recently showed me an evaluation report for a CSV AI agent. It had a single composite score: 8.2 out of 10. She asked me if it was good enough for her next FDA inspection. I asked her what the difference between a 7.2 and an 8.2 was. She stared at the number for a long time.

Nobody can answer that question — not the QA manager, not the model vendor, and certainly not the FDA inspector who will ask it during a pre-approval inspection. A 1-10 score for regulatory compliance is not an evaluation. It is a liability dressed up as a metric.

The problem is not that evaluating AI agents in computer system validation is hard. The problem is that most teams build evaluation systems that optimize for the wrong thing: subjective quality ratings instead of deterministic compliance checks, vibes instead of verifiable outcomes, and transcript claims instead of environment state.

Good evals for CSV are boring. A row in your trace matrix is either linked or it isn’t. A regulatory citation either exists in the controlled reference database or it doesn’t. A test step either has measurable acceptance criteria or it uses the word “correctly.” That boringness is what makes you audit-ready.

The Core Paradox

Traditional CSV assumes deterministic systems. Same input, same output, every time. That assumption is the legal foundation of every validation protocol ever written.

AI agents break all of it. They reason probabilistically. They chain tool calls in emergent sequences. They retrieve context that changes between runs. They produce different outputs from identical prompts.

The validation strategy for an AI agent must therefore shift from validating code execution to validating the boundary, operational envelope, and outcomes of the agent. And the evaluation system that proves those outcomes must itself be defensible to an FDA inspector.

This is where most teams go wrong. They build evaluation as an afterthought — a quick “rate this output 1-10” LLM judge tacked onto the end of a pipeline. In regulated environments, that is not evaluation. That is an opinion without audit trail.

The Asymmetry of Verification

Here is the foundational insight that should drive every evaluation decision: it is hard to generate a correct validation protocol, but cheap to check one.

Generating a risk assessment for a GAMP Category 4 LIMS requires understanding the system, the regulatory framework, the intended use, the data flows, and the patient safety implications. Checking whether that risk assessment correctly classifies the system as Category 4 is a single comparison: output.category == expected.category.

This asymmetry is your best friend. Every time you can reduce an evaluation to a deterministic check against a known expected output, you win three times: the check is instant, it is consistent across runs, and it is trivially auditable.

The rule is simple: if the thing you are evaluating is visible in your system or comparable against an expected output you saved ahead of time, use code. Use LLM judges only when the outcome exists purely in language and no expected output exists.

What You Want to Check Deterministic (Prefer) LLM Judge (Avoid if Possible)
GAMP Category Expected: Category 4. Code: output.category == expected.category Judge reads 20-page FS and guesses if Category 4 makes sense
Traceability Expected: Every URS has ≥1 test. Code: parse matrix, check for orphans Judge reads matrix and says “looks complete”
21 CFR Part 11 citation Expected list from your SOP. Code: citation in allow-list? Judge checks if citation sounds plausible
Action vs. Claim Agent says “VP created.” Code: check Veeva API that file exists Judge reads transcript “I created the VP” and marks pass

Breaking the God Evaluator

Do not create one judge that rates overall quality. “Rate this validation package 1-10 for compliance, completeness, and tone” is a God Evaluator. It yields an uninterpretable score, it hides what needs to be fixed, and it is indefensible in an inspection.

Break it into narrow, binary, categorical checks. Each evaluator answers exactly one question with exactly one decision.

Here is a starter set for a CSV agent — every output binary or categorical, no scales:

1. GAMP Category Correctness — Code evaluator. Input: system description. Output: pass/fail vs. expected category. This is a lookup, not a judgment.

2. Traceability Integrity — Code evaluator. Input: agent’s trace matrix JSON. Output: fail if orphan requirements or orphan tests exist. This is a graph traversal check.

3. No Hallucinated Regulation — Code + Judge hybrid. Input: agent output + allow-list of regs from your SOPs. Code checks exact match first. A judge handles only paraphrases that don’t match the allow-list exactly.

4. GxP Impact Assessment — Categorical. Output: correct / missed_critical / over_classified. Needs expected output. A missed patient safety risk is a critical failure; over-classifying a document management system as high-risk is a waste of resources.

5. Risk Assessment Completeness — LLM Judge. Binary: does it assess both Severity and Probability per GAMP 5 2nd Edition? Not “is it good” — does it have the required components.

6. Action vs. Words — Code evaluator. If the agent says “OQ executed,” verify the test evidence row was written to the QMS. Grade the outcome in the environment, not the claim in the transcript. This is the single most important evaluator in your entire system.

7. ALCOA+ Grounding — LLM Judge. Binary: is every data integrity claim grounded in provided SOP/context, not invented? An agent that hallucinates ALCOA+ requirements is as dangerous as one that misses them.

8. No Premature Compliance Promise — LLM Judge. Binary: does the agent claim “system validated / audit-ready / compliant” before IQ/OQ evidence exists? This is the number one false positive in CSV agents. The temptation to declare victory before the evidence exists is the most common and most dangerous pattern.

The Five-Layer Architecture

Organizing evaluators into layers prevents the common failure of treating all checks as equivalent. A JSON schema failure and a hallucinated regulation are not the same severity — they should not live in the same flat list.

         BUSINESS IMPACT                                              
  ┌─────────────────────────────┐                                     
  │  Layer 5: Human QA Review   │  Approve / Revise / Reject          
  └─────────────┬───────────────┘                                     
                │                                                     
  ┌─────────────▼───────────────┐                                     
  │  Layer 4: Compliance        │  GAMP 5, Part 11, ALCOA+            
  └─────────────┬───────────────┘                                     
                │                                                     
  ┌─────────────▼───────────────┐                                     
  │  Layer 3: Task Success      │  Artifact checks (RTM, URS, scripts)
  └─────────────┬───────────────┘                                     
                │                                                     
  ┌─────────────▼───────────────┐                                     
  │  Layer 2: Reasoning         │  LLM judge for subjective checks    
  └─────────────┬───────────────┘                                     
                │                                                     
  ┌─────────────▼───────────────┐                                     
  │  Layer 1: Basic Output      │  100% deterministic (JSON, schema)  
  └─────────────────────────────┘

Layer 1 is pure code. If the output does not parse as valid JSON with the required fields, nothing else matters — the agent failed before it even got to compliance. Layer 5 requires a qualified human. The layers are not optional alternatives; they are a stack. A failure at Layer 1 blocks everything above it.

Writing LLM-as-a-Judge Prompts That Survive Inspection

When you do need an LLM judge — for risk assessment reasoning, deviation rationale, or regulatory interpretation — the prompt itself becomes a controlled document. It needs version control, change control, and calibration evidence.

Structure every judge prompt as onboarding material for a junior Validation Engineer. Five parts, no exceptions:

Part 1: Context. What the agent does, what regulatory framework applies, what SOPs govern it. Not generic instructions — specific to the task and domain.

Part 2: Precise Criterion + What to Ignore. Not “is the risk assessment good?” but “does the risk rating cite a specific hazard with probability and severity estimates?” Explicitly state what to ignore: formatting, length, non-critical wording differences.

Part 3: Labeled Examples. Two to four real cases you have manually labeled, mixing pass and fail, with a short reason for each. Pull these from your error analysis — not from hypotheticals.

Part 4: Reasoning First, Verdict Last. Force the model to output its step-by-step reasoning before the binary pass/fail. This creates the audit trail for why a failure was flagged. An inspector can read the reasoning and evaluate whether the judge’s logic is sound.

Part 5: Explicit Way Out. The judge must output UNKNOWN when the input lacks sufficient context to make a judgment. Never let the model guess. Guesswork is unacceptable in regulated environments.

Example: No Hallucinated Regulation

Context: You evaluate an AI agent that drafts Computer System Validation documents for GxP systems under FDA 21 CFR Part 11 and EU Annex 11. It must only cite requirements from the provided context documents.

Criterion: Does the response invent a regulatory citation or requirement not present in CONTEXT? A citation is invented if it mentions a specific clause (e.g., “21 CFR 11.300”) that does not exist in context, or claims a regulation requires something not stated. Ignore formatting differences.

Examples: [PASS] Response: “Per SOP-042, audit trail is required.” — SOP-042 in context does require audit trail. [FAIL] Response: “Per 21 CFR 11.70, blockchain is mandatory.” — 21 CFR 11.70 does not exist / not in context.

Instructions: First explain your reasoning step by step. Then output verdict as JSON: {“verdict”: “PASS” | “FAIL” | “UNKNOWN”}. Use UNKNOWN if context is missing.

Calibrating the Judge

Label 20-30 real cases with human SME approval. Run the judge on half. Check each class separately. Beware the 90% accuracy trap: if your failure rate is 10%, a judge that always says PASS is 90% accurate and completely useless. It will never catch a real failure.

The calibration artifacts are themselves validation evidence:

  • Judge prompt = controlled document, versioned, change-controlled
  • Labeled cases = validation evidence referenced in the change record
  • Per-class agreement rates = stored as the validation report
  • Any judge prompt change = a re-validation event, not a silent redeploy

Building the Eval Dataset

Your eval dataset is your OQ/PQ test suite. Under GAMP 5, it must be version-controlled, traceable, and designed to cover the risk space — not just the happy path.

Start With One Release Question

Not “test CSV agent.” But: “Can we ship v2 that drafts a Validation Plan for a COTS system?” One goal = one dataset. If you need different inputs or release decisions, split datasets.

The Minimally Complete First Version

Fifteen to thirty rows. That is enough to run a meaningful experiment and catch the worst regressions. Expand deliberately from there.

Routine Ambiguous / High-Risk
Infrastructure (Cat 1) OS patching Cloud VM qualification on AWS
COTS configurable (Cat 4) LIMS with config Veeva Vault with custom workflow
Custom (Cat 5) Lab instrument interface GxP AI model for batch release
Data Integrity Issue Missing audit trail Excel with macros, no access control

Do not mirror production frequency. Over-represent high-risk and known failures. Your dataset is not a statistical sample of reality — it is a stress test designed to find problems before an auditor does.

The Item Schema

Every dataset item must be a structured, evaluable object — not a free-text prompt:

{
  "input": {
    "task": "generate_iq_protocol",
    "system_description": "SaaS LIMS, configured workflow, stability data",
    "gamp_category": 4,
    "context_docs": ["SOP-042", "GAMP5 Cat definitions"]
  },
  "expectedOutput": {
    "required_docs": ["VP", "RA", "URS", "IQ", "OQ"],
    "must_not_claim": "System validated",
    "traceability": {"URS-042": ["OQ-05", "OQ-06"]}
  },
  "metadata": {
    "source": "audit_finding_2024_07",
    "scenario_type": "COTS_configurable",
    "difficulty": "ambiguous",
    "role": "known_regression"
  }
}

Keep input to only what the agent sees in production. Keep expectedOutput to only what the evaluator needs — behavior, not wording.

Expand Deliberately

Three expansion patterns, each with a distinct purpose:

Production-mirroring: Add anonymized, QA-approved real traces weekly. Keeps the dataset aligned with real-world usage.

Bad-trace expansion: Every serious production failure becomes one new dataset row. This is a permanent regression test to ensure that specific failure never returns. In regulated environments, this is not optional — it is continuous validation.

Purpose-specific splits: When your dataset tries to be regression + adversarial + red-team simultaneously, split it. You want three datasets: end-to-end plan generation, step-level risk assessment, and adversarial/compliance traps. Mixing them makes your pass rates meaningless.

Offline vs. Online — Both, With Different Purposes

Offline (Pre-deployment OQ): Run the agent through the full dataset before releasing any prompt, model, or tool configuration change. Compare v2 against v1. A regression drops the build immediately. This is your CI gate.

Online (Post-deployment PQ): Continuously evaluate production traces with deterministic checks and LLM judges. Watch trends — is the rate of hallucinated citations going up this week? Is the traceability completeness degrading? Trigger formal re-validation if accuracy drifts below predefined quality boundaries.

For CSV agents specifically, offline experiments are the primary validation activity — you gate releases against curated datasets. Online scoring is secondary and observational — trending judge verdicts against human reviewer overrides. That disagreement rate is itself a useful metric: if the Evidence Reviewer’s “sufficient” verdict disagrees with what the human QA reviewer actually approves 30% of the time, your judge needs recalibration.

The Evaluator Is a GxP Record

This is the piece most teams miss. Once your judge verdicts feed into a human review gate, the verdict plus reasoning trace needs the same ALCOA+ treatment as any other quality record:

  • Attributable to a specific model/prompt version
  • Timestamped at the moment of evaluation
  • Immutable once logged
  • Legible and reconstructable for audit

The evaluations themselves should be stored in a 21 CFR Part 11-compliant system with immutable audit trails. If an LLM judge gave a “pass” to a deviation rationale that later caused a 483 observation, you need to reconstruct why — what the judge saw, what reasoning it produced, what prompt version was active.

This is not over-engineering. This is the minimum standard for a computerized system that influences quality decisions in a regulated environment.

The Closed-Loop System

Present your evaluation framework to auditors as a closed-loop:

Gold Standard Dataset → AI Agent → Evaluator (Code + LLM)
    → Discrepancy Report → Human Review → Dataset Update

This shows the regulator you are not blindly trusting the AI. You are actively, scientifically, and quantitatively proving its reliability over time. The dataset grows with real failures. The judges get recalibrated with real disagreements. The system gets more reliable with every iteration — and every iteration is documented.

What to AVOID

1-10 quality scales. A regulator cannot distinguish a 6 from a 7. You cannot validate a 7. Replace every scale with binary or categorical outputs.

Single “God Evaluators.” One judge that rates compliance, completeness, and tone will tell you nothing about what to fix. Split into narrow, actionable checks.

Grading transcript claims. If the agent says “Deviation #402 closed,” do not read the transcript and mark it pass. Query the QMS database. Grade the outcome in the environment.

Uncalibrated judges. A judge prompt without labeled calibration cases is an unvalidated computerized system making quality decisions. It is indefensible in an inspection.

Mixing regression and adversarial in one dataset. Your pass rate stops meaning anything when half the rows are designed to fail. Keep them separate.

Skipping human review for high-risk outputs. Evals inform human judgment; they do not replace it. Every high-risk agent output needs a qualified person’s sign-off before it becomes a regulated record.

The Bottom Line

Evaluating an AI agent for CSV is not about finding the best quality score. It is about building a system where every failure is measurable, every metric is actionable, and every evaluation is auditable.

Start with error analysis on your last 20 traces. Pick one failure mode. Can a code evaluator with an allow-list settle it? If yes, build it and stop. If not, label 20 cases, write a 15-line judge with context and precise criterion, and calibrate it.

Create a 20-row dataset covering three scenario types and two difficulty levels. Run it. Fix the schema and the judge, not just the agent prompt.

Good evals are boring by design. A requirement is either traced or it isn’t. A citation either exists or it doesn’t. A test step either has measurable criteria or it uses the word “correctly.”

That boringness is what makes you audit-ready.

Research notes: [[Evaluating AI Agents in CSV Life Sciences]]