An AI agent answers correctly in validation, ships to production, and fails on a request nobody thought to include in the test set. The team patches the prompt. The patch fixes that request but quietly breaks another workflow. A dashboard still shows a healthy average score.

That is not a rare edge case. It is the normal consequence of treating a probabilistic system like ordinary application code.

The better model is a closed engineering loop:

Trace → Monitor → Analyze → Build datasets → Experiment → Evaluate → Release → Trace again

Each stage changes the evidence. Tracing records what happened. Monitoring decides where to look. Error analysis turns incidents into a failure taxonomy. Datasets preserve those failures as repeatable cases. Experiments isolate a proposed change. Evaluation decides whether the change is safe enough to release.

The loop matters in any LLM application. In life sciences, where an agent may retrieve a superseded SOP, omit a safety qualification, or call the wrong system tool, it is the difference between an interesting demo and a defensible production system.

Why unit tests are necessary but insufficient

Traditional software tests work best when a defined input should produce a defined output. LLM systems add several sources of variation at once: user wording, retrieved context, model behavior, tool selection, conversation history, and the order in which an agent acts.

You should still test deterministic properties. JSON must parse. Required fields must exist. A tool argument must match its schema. A prohibited identifier must never appear. But a passing schema says nothing about whether the answer used the correct SOP revision or whether the proposed CAPA addresses the root cause.

This is why LLM quality work cannot begin with a generic “helpfulness” score. The useful criteria have to come from observed behavior in the application. Otherwise a team can build a polished evaluation system that measures the wrong things.

1. Trace one complete unit of work

A trace is the structured record of one system invocation. It should show the input, the steps taken, the data each step received, the output it produced, and operational metadata such as model, token use, cost, and latency.

For an agent that answers a question about a controlled procedure, one trace might contain:

Trace: answer SOP question
├── generation: interpret the request
├── retriever: search approved procedures
├── span: filter by site, status, and effective date
├── generation: draft a grounded answer
├── tool: validate citations
└── event: route to human review

The hierarchy is not cosmetic. If the final answer is wrong, the trace should let an engineer distinguish among a bad query, irrelevant retrieval, a version-filter failure, unsupported generation, and a broken citation check.

Trace boundaries also matter. A useful rule is one trace per coherent invocation and one session for a sequence of related invocations. For a deviation assistant, a trace may be one investigative turn while the session contains the full case. For an automated document review agent, a trace may be one review run while the session groups all runs for the document revision.

Cut traces too small and the evidence loses its story. Cut them too large and failures disappear inside an unreadable tree.

Start with one important workflow from end to end. Do not instrument every path at once. Review a handful of traces and ask a blunt question: can another engineer reconstruct what the agent saw and why it acted?

Tracing creates records. Monitoring makes those records usable.

Two activities belong here, and they should not be confused:

  • Aggregate tracking answers whether quality, latency, cost, or override rate is moving over time.
  • Signal detection identifies the individual traces that deserve investigation now.

An average score can fall after a model change, but the average does not reveal the cause. The underlying trace does. Conversely, one terrible trace can be alarming without indicating a system-wide regression. Teams need both views.

The strongest signals are often specific to the workflow. A thumbs-down is explicit but sparse and biased toward unhappy users. Implicit behavior produces more data: a user retries the question, corrects an extracted field, rejects a draft, requests a human handoff, or abandons a session. In a document extraction flow, the most useful quality signal may be a human changing the lot number before approval. In a quality assistant, it may be a reviewer replacing the cited procedure because the agent selected an obsolete revision.

Those actions are not self-explanatory. A retry can mean a wrong answer, an unclear question, or simple curiosity. Monitoring should surface the trace for review, not declare the root cause by itself.

3. Let failures define the taxonomy

The most valuable step in the loop is also the least automated: read the traces.

Error analysis borrows a method from qualitative research. Pull a representative sample, note the first meaningful failure in plain language, then cluster similar notes. Do not begin with a vendor’s generic categories. Let the application reveal what is broken.

A life sciences RAG agent might produce a taxonomy like this:

Failure category What the trace shows Likely action
Superseded document retrieved Retrieval returned an obsolete SOP despite an effective-version filter Fix retrieval logic; add deterministic version check
Correct source, wrong section Document was relevant but the cited section did not support the claim Add retrieval dataset cases and citation-grounding evaluator
Missing clarification Agent classified a deviation before asking for required facts Revise workflow; add multi-turn cases
Unsupported compliance claim Final answer added a requirement absent from approved context Add judge rubric, human review, and regression cases
Wrong tool chosen Agent queried a training system instead of the QMS Restrict routing and validate tool selection
Review bypassed High-risk output did not enter the approval queue Fix code path; create a release-blocking test

Once every sampled trace is labeled, the team can calculate a failure rate per category. A vague concern such as “the bot feels worse” becomes a measurable statement: 14% of sampled failures involved stale-document retrieval, while 3% involved unsupported generation.

The taxonomy should not become permanent doctrine. Re-run the analysis after a prompt rewrite, model swap, retrieval change, or major feature. As the system improves, the distribution changes. Old categories shrink and new ones appear.

4. Turn incidents into datasets

A dataset is not a warehouse of prompts. It is a collection of cases that the system must handle, assembled for a defined testing purpose.

Each item needs an input. It may also include an expected output and metadata. The expected output does not always have to be a gold response. Depending on the task, it can be:

  • an exact label or extracted value;
  • a reference answer;
  • a list of requirements, such as “must cite the effective SOP and state the escalation path”;
  • nothing at all, when a reference-free evaluator checks format, language, safety, or prohibited content.

Metadata is where operational context lives: site, product, document revision, risk class, failure category, source trace, and whether the case is synthetic or observed in production.

Do not force one dataset to do every job. A small component dataset can test retrieval quickly and cheaply. An end-to-end dataset catches failures caused by interactions among retrieval, prompts, tools, and routing. Both are useful.

The source mix should be deliberate:

  1. Production cases reveal how people actually use the system.
  2. Handwritten cases encode requirements and known edge conditions before they occur.
  3. Synthetic cases expand coverage only after the team knows which dimensions need expansion.

The sizes can grow with the question. Roughly 10 difficult cases may be enough to explore one defect. Twenty to thirty real examples are enough for an initial side-by-side experiment. A CI suite for a mature workflow may contain hundreds of cases. A guardrail dataset should keep growing whenever production reveals a new attack or bypass.

The crucial habit is simple: every recurring production failure earns a durable place in a regression dataset. Otherwise the organization pays to learn the same lesson twice.

5. Experiment against a fixed baseline

An experiment needs four things: a baseline, a dataset, the variable being changed, and outputs that can be compared.

The baseline is the current system. The dataset stays fixed across the comparison. The variable may be a model, prompt, retrieval strategy, context policy, tool set, or agent architecture.

Change one variable when possible. If a prompt and retrieval filter change together, an improvement cannot be assigned confidently to either. Some architectural changes cannot be separated cleanly, but that should be an acknowledged limitation rather than an accidental one.

Read the candidate and baseline traces side by side before reaching for an aggregate score. A few examples often expose the actual tradeoff: the candidate follows instructions better but asks unnecessary questions; retrieval improves while latency doubles; a cheaper model passes common cases but fails the rare high-risk ones.

Quality, cost, and speed rarely improve together. A useful experiment makes the trade explicit.

The distinction between a specification problem and a generalization problem is especially important. If the prompt never said to use only effective documents, clarify the specification and fix it. If the instruction is clear but the system follows it inconsistently, measure the behavior across cases and test a change. Not every bug deserves an evaluator.

6. Evaluate with the method that fits the claim

Evaluation is the release decision, not a single metric.

A mature stack combines three methods:

Deterministic code checks

Use code when the property has an unambiguous answer:

  • valid JSON and schema conformance;
  • required citation fields;
  • allowed tool names and argument types;
  • effective document status;
  • prohibited identifiers or phrases;
  • numerical ranges and identifier formats.

These checks are fast, cheap, and reproducible. They should block releases when a binary requirement fails.

LLM judges

Use a model judge for semantic criteria: relevance, faithfulness to source, completeness, appropriate escalation, or whether a summary preserved the important parts of a record.

Judges need calibration against human labels. A plausible rubric and a strong model do not prove that the judge agrees with the people accountable for the process. The judge can also share blind spots with the application, particularly when both use the same model family.

A calibrated judge backed by deterministic checks is useful. An uncalibrated judge is another opinion with an API.

Human review

Manual review comes first, not last. Reading outputs teaches the team what good and bad mean in context, produces labels for calibration, and catches novel failure modes that automated evaluators do not yet know about.

For higher-risk workflows, route critical failures and low-confidence results to qualified reviewers, then sample apparently successful traffic as well. If humans only review flagged traces, they will never learn how often the flagging system misses.

Each quality claim should map to its own evaluator. “Quality” is too vague. “All procedural claims are supported by an effective approved source” can be tested.

The release gate should preserve risk, not average it away

Suppose a candidate prompt improves the average groundedness score from 0.86 to 0.91 but introduces one case that gives unsupported batch-disposition advice. The average improved. The release should still fail.

A release gate needs at least three layers:

  1. Critical invariants: zero tolerance for privacy leakage, prohibited actions, stale controlled documents, bypassed approvals, and other safety-critical failures.
  2. Category thresholds: no unacceptable regression within an important failure class, user segment, site, or risk tier.
  3. Portfolio tradeoffs: overall quality, latency, and cost improve enough to justify the change.

This is why failure taxonomies and dataset metadata matter. Without them, aggregate scores can hide a small but dangerous regression inside many easy wins.

What makes the loop defensible in GxP work

The engineering loop is not itself a validated system, and an observability platform is not automatically a compliant record repository. The surrounding controls still have to be designed, qualified, and documented.

For regulated deployments, preserve at least:

  • the released prompt, model, retrieval, and tool configuration;
  • the dataset version and provenance of each case;
  • experiment runs linked to complete traces;
  • evaluator code and judge rubric versions;
  • judge-to-human calibration results;
  • acceptance thresholds and the release decision;
  • reviewer identity, date, comments, and approval;
  • the connection between production incidents and added regression cases.

Sensitive trace content also creates obligations. Prompts, retrieved passages, and tool results may contain personal or confidential data. Apply access control, redaction, retention, and data residency policies to observability data with the same care used for the application itself.

The strongest evidence package tells a continuous story: this production trace exposed a failure; error analysis placed it in this category; the case entered this dataset; this change was tested against the fixed baseline; these evaluators and reviewers accepted the result; this configuration was released; production monitoring confirmed the expected behavior.

That is much stronger than a slide showing a 92% score.

A practical 30-day rollout

Do not attempt to automate the full loop on day one.

Week 1: Instrument one consequential workflow. Capture meaningful inputs, outputs, retrievals, tool calls, model configuration, cost, and latency. Review traces for readability.

Week 2: Sample real traces and run open-ended error analysis. Name the first taxonomy and connect a few explicit or implicit user signals to the underlying traces.

Week 3: Build a small dataset from 20 to 30 observed and requirement-based cases. Add deterministic checks for hard constraints. Keep expected outputs only where they help.

Week 4: Compare one proposed change with the current baseline. Read outputs side by side. Add a narrowly defined judge only after human review has produced a usable rubric. Document the release decision and monitor the new version.

This first loop will be manual and imperfect. That is fine. Automation is valuable after the team understands the work. Automating earlier mostly creates distance from the evidence.

The bottom line

Reliable AI engineering is not prompt tuning followed by a dashboard. It is a learning system built around evidence.

Trace coherent executions. Monitor trends without losing the underlying cases. Read failures before naming metrics. Preserve recurring failures as datasets. Compare changes against a fixed baseline. Match each quality claim to deterministic checks, calibrated model judges, or human review. Block critical regressions even when the average rises.

Then ship and start again.

Sources