Your LLM application scores 94% on your evaluation benchmark. You deploy it. Three weeks later, a customer reports that the model confidently fabricates citations in a specific edge case your benchmark never tested.

This is the single-method evaluation trap. One evaluator, no matter how well-designed, has a blind spot. The fix is not a better evaluator. It is a system of evaluators that covers each other’s gaps.

The Five Methods

Langfuse offers five evaluation methods. Most teams pick one. The teams that actually catch production failures compose all five into a layered architecture:

┌──────────────────────────────────────────────────────────┐
│              EVALUATION MATURITY LADDER                   │
│                                                          │
│  Level 0: "We log traces"                                │
│  Level 1: "We run one LLM-as-a-Judge evaluator"          │
│  Level 2: "We combine automated + manual scoring"        │
│  Level 3: "We have a full evaluation pipeline"           │
│  Level 4: "Evaluation drives deployment decisions"       │
│                                                          │
│  Most teams stall at Level 1.                            │
└──────────────────────────────────────────────────────────┘

Here is what each method does, when to use it, and how they compose.

Method 1: LLM-as-a-Judge — The Scalable Semantic Layer

This is the workhorse for production monitoring. You define a rubric, pick a judge model, and let it score every output (or a sample of them) automatically.

The judge model receives the input, the output, and your rubric, then returns a structured score plus reasoning. A typical evaluation prompt looks like:

Rate the helpfulness of this response on a scale of 1-5.

Criteria:
- 1: Completely unhelpful, irrelevant, or harmful
- 3: Partially helpful but missing key information
- 5: Fully addresses the question with accurate, actionable information

User question: {{input}}
Response: {{output}}

Score types matter more than you think. Use numeric scores for continuous dimensions like helpfulness or faithfulness. Use categorical scores when you need discrete labels — correct, partially_correct, incorrect — that map to your quality taxonomy. Use boolean scores for binary policy decisions: does this response contain PHI? Does it violate scope? Is the user disagreeing with the assistant?

The observation-level shift is real. Trace-level evaluators are deprecated. Observation-level evaluators give you operation-level precision — you can evaluate just the final LLM call, just the retrieval step, or just the tool invocation, without scoring entire workflows. This cuts cost and improves signal. The trade-off: if you need overall request/response context, you must target a logical root observation that carries that summary.

The accuracy numbers are honest. Strong judge models (GPT-4o class, Claude Sonnet, Gemini Pro) achieve 80-90% agreement with human evaluators. That is comparable to inter-annotator agreement between two trained humans. The key requirement: the judge model must support structured output so scores parse reliably.

Cost is manageable. A typical evaluation runs $0.01-0.10 per assessment. At 5% sampling on 10,000 daily requests, that is $5-50/day for continuous quality monitoring. Most teams find this is the cheapest quality signal in their stack.

When LLM-as-a-Judge Alone Fails

Three failure modes:

  1. Rubric drift — your rubric does not cover a new failure mode. The judge scores outputs as “correct” because the rubric does not define the edge case.
  2. Judge bias — the judge model has systematic preferences (verbosity bias, position bias) that correlate with your application’s failure modes.
  3. Deterministic checks missed — JSON validity, schema compliance, exact keyword presence — semantic judges are overkill and unreliable for these.

This is where the other four methods come in.

Method 2: Code Evaluators — The Deterministic Floor

Code evaluators run custom Python or TypeScript logic inside Langfuse. No LLM, no ambiguity, no cost per evaluation. Use them for objective, binary checks:

  • Is the output valid JSON?
  • Does it contain the required fields?
  • Does the response include a tool call to the expected function?
  • Is the output length within bounds?
  • Does it match a regex pattern?

The function contract is clean. You write an evaluate function that receives an EvaluationContext and returns an EvaluationResult with one or more scores:

def evaluate(ctx: EvaluationContext) -> EvaluationResult:
    output = ctx.observation.output
    is_valid_json = False
    try:
        json.loads(output)
        is_valid_json = True
    except (json.JSONDecodeError, TypeError):
        pass

    return EvaluationResult(
        scores=[Score(
            name="JSON Valid",
            value=is_valid_json,
            data_type="BOOLEAN",
            comment="Output is valid JSON." if is_valid_json else "Output is not valid JSON.",
        )]
    ]

The constraints are strict and deliberate. Standard library only — no third-party packages. No network access. Two-second runtime limit. This is intentional: code evaluators run on every matching observation at ingest time. If they were slow or had dependencies, they would block your ingestion pipeline.

The two-second limit is generous for what these evaluators should do. If your code evaluator is doing anything that takes two seconds, you are using it wrong. Regex matching, JSON parsing, field presence checks, string containment — these are microsecond operations. If you need heavier logic, run it in your own pipeline and ingest the result via the Scores API.

How They Compose with LLM-as-a-Judge

The pattern is defense in depth:

Observation arrives
    → Code Evaluator: "Is the output structurally valid?"
        (FAIL → score immediately, skip LLM judge)
    → LLM-as-a-Judge: "Is the output semantically correct?"
        (scores on rubric dimensions)

The code evaluator acts as a gatekeeper. If the output fails a basic structural check, there is no reason to spend $0.05 asking a judge model whether it is “helpful.” This is the cheapest form of evaluation triage.

Method 3: Annotation Queues — The Ground Truth Factory

Automated evaluators need calibration. You cannot trust an LLM judge’s scores if you have never checked them against human ground truth. Annotation Queues are the workflow engine for creating that ground truth.

The workflow is batch-oriented and keyboard-driven. Create a queue, attach Score Configs (the dimensions you want annotated), add traces or observations, assign domain experts, and let them score. The keyboard shortcuts (→/← to navigate, 1–9 to select scores, Cmd+Enter to advance) make it fast enough to annotate hundreds of items per session.

The real value is calibration data. Once 100-200 items are human-annotated, you can compare your LLM judge’s scores against human scores. If your hallucination judge agrees with human reviewers 72% of the time, you know exactly how much to trust it. If it agrees 94% of the time, you have a production-grade evaluator. You cannot get this calibration without the annotation workflow.

Corrected outputs are underrated. Annotation Queues let you attach what the model should have produced. This is gold for regression testing — you can create a dataset of corrected outputs and run experiments against it every time you change your prompt or model.

Method 4: Manual Scores via UI — The Ad-Hoc Escape Hatch

Not every evaluation fits a batch workflow. Sometimes you are debugging a specific trace and want to mark it. Sometimes a stakeholder is reviewing a conversation and wants to flag it.

Scores via UI are the lightweight complement to Annotation Queues. Open a trace, click Annotate, select your Score Configs, and record your judgment. No queue, no batch, no workflow overhead.

The use case is narrow but important. It is the only method that requires zero setup beyond creating a Score Config. For teams just starting evaluation, it is the fastest path to getting human scores into the system.

Experiment integration is the hidden feature. When you run experiments via UI or SDK, you can annotate results directly from the compare view. The full experiment context — inputs, outputs, automated scores — is preserved while you review. Summary metrics update as you add scores. This turns experiment review from a spreadsheet exercise into an integrated workflow.

Method 5: Scores via API/SDK — The Integration Backbone

Every external pipeline, custom script, browser feedback widget, and CI job feeds into Langfuse through the Scores API. This is the method that makes the other four composable.

Three attachment levels. Scores attach to traces (overall quality), observations (specific operations), or sessions (entire conversations). This gives you the granularity to score at whatever level makes sense for your use case.

The four data types cover every scoring paradigm:

Type Input Use Case
Numeric float Continuous dimensions (0.0-1.0)
Categorical string Discrete labels (“correct”, “partial”, “wrong”)
Boolean float (0/1) Binary decisions (true/false)
Text string (1-500 chars) Freeform reviewer notes

Browser SDK for frontend feedback. The @langfuse/browser package captures user feedback — thumbs up/down, star ratings, “this was wrong” flags — with only a public key. No backend proxy needed. Each score sends immediately via the ingestion API.

Idempotency prevents duplicate scores. A score is identified by three fields: id, name, and toDate(timestamp). All three must match for an overwrite. Use a stable id (like trace_id-score_name) as an idempotency key. This is critical for pipelines that re-run evaluations — without it, you accumulate duplicates.

Score Config enforcement standardizes your data. Reference a configId when creating scores to validate against predefined ranges (numeric), allowed categories (categorical), or data type constraints. This is the difference between a free-text mess and a queryable quality database.

The Composed Architecture

The five methods are not alternatives. They are layers:

┌────────────────────────────────────────────────────────────┐
│                  PRODUCTION OBSERVATION                     │
│                                                            │
│  1. Code Evaluator runs first (structural gatekeeper)      │
│     → FAIL: attach boolean score, done                     │
│     → PASS: continue                                       │
│                                                            │
│  2. LLM-as-a-Judge scores on semantic rubrics              │
│     → Attach numeric/categorical scores                    │
│                                                            │
│  3. Sampling selects observations for human review         │
│     → Annotation Queue scores calibrate the judge          │
│                                                            │
│  4. External pipelines ingest additional scores via API    │
│     → User feedback, CI checks, custom metrics             │
│                                                            │
│  5. All scores converge in a single quality dashboard      │
│     → Trends, correlations, drift detection                │
└────────────────────────────────────────────────────────────┘

The Experiment Loop

Development follows the same stack but with experiments instead of live traffic:

  1. Create a dataset with test inputs and expected outputs
  2. Run experiments via UI or SDK — execute your application for each dataset item
  3. Code evaluators check structural validity (automated, instant)
  4. LLM-as-a-Judge scores semantic quality (automated, minutes)
  5. Human annotation on the compare view provides ground truth (manual, targeted)
  6. Compare experiment runs — did the prompt change improve or degrade scores?

This loop is where you catch regressions before they reach production. The key insight: do not skip step 5. Automated scores without human calibration are numbers without meaning.

The Game-Changer: Observation-Level Precision

The single biggest architectural improvement in Langfuse’s evaluation system is the shift from trace-level to observation-level evaluators.

Traces are coarse. A trace might contain five LLM calls, three retrievals, and two tool invocations. Scoring the entire trace for “helpfulness” conflates the quality of the final response with the quality of intermediate steps.

Observations are precise. You can target exactly the final LLM generation for helpfulness, exactly the retrieval step for relevance, and exactly the tool call for argument validity. Different evaluators run on different operations within the same trace.

Combined filtering makes this surgical. Stack observation filters (type, name, metadata) with trace filters (userId, sessionId, tags, version). Example: “evaluate helpfulness on all LLM generations in conversations tagged customer-support for users with plan=enterprise.” That is a single evaluator configuration.

The migration deadline is real. Trace-level evaluators are deprecated and tied to the Langfuse v4 removal timeline. If you are still on trace-level, the upgrade path is documented and the observation-level API is stable.

The Backfill Capability

You can run observation-level LLM-as-a-Judge on historical data. This means:

  1. You ship an application with no evaluation
  2. Three months later, you design an evaluator for a failure mode you discovered
  3. You backfill scores on all matching historical observations

The workflow: open the Traces table, filter to your target timeframe, select rows, click Actions → Evaluate. The evaluator runs on the selected observations and attaches scores retroactively.

This is uniquely powerful for regulated environments where you need to demonstrate quality over the full lifecycle of a deployment, not just from the moment you set up monitoring.

Debugging the Evaluation System Itself

Every LLM-as-a-Judge execution creates a full trace in the langfuse-llm-as-a-judge environment. Every code evaluator execution creates a trace in the langfuse-code-eval environment. Both are hidden from the default tracing view — you filter explicitly to see them.

This gives you complete visibility into the evaluation process itself: what prompt was sent to the judge, what the judge model returned, token usage, latency, and errors. If your evaluator is producing unexpected scores, the execution trace tells you exactly why.

Common execution statuses:

Status Meaning Action
Completed Evaluation finished successfully None needed
Error Evaluation failed Check execution trace for details
Delayed LLM provider rate limits hit Automatic retry with backoff
Pending Queued, waiting to run Wait or check queue depth

What We Would Not Skip

If you are building evaluation for a production LLM application, here is the minimum viable stack:

  1. At least one LLM-as-a-Judge evaluator on your final LLM output — this is your scalable quality signal
  2. At least one code evaluator for structural validity — JSON, schema, required fields
  3. An Annotation Queue with 200+ human-annotated samples — this is your calibration dataset
  4. Scores API integration for user feedback — thumbs up/down from the UI is the cheapest quality signal you will ever get

If you skip the annotation queue, you are trusting your automated scores without verification. If you skip the code evaluator, you are spending LLM-judge dollars on outputs that fail basic structural checks. If you skip the API integration, you are flying blind on user satisfaction.

The five methods are not a menu. They are a stack. Each layer makes the other layers more trustworthy.

Research Notes

[[langfuse-evaluation-methods-comprehensive-report]]