4. Building an LLM Judge
Human annotation is the most direct way to obtain trustworthy evaluation labels, but it does not scale to every model release, every prompt change, or every day of production traffic. Large language models used as evaluators, commonly called LLM-as-a-judge, offer a way to extend human judgment to that scale. The evidence for this approach is encouraging but uneven: on MT-Bench, a strong LLM judge agreed with expert human preferences at roughly the same rate that the experts agreed with each other[1], whereas a later study spanning 20 evaluation tasks found that agreement with humans varied widely by task, dataset, and the property being judged[2]. Whether a judge can be trusted is therefore a property of the specific judge, task, and data, and it has to be established rather than assumed.
This chapter addresses the construction of the judge, which corresponds to the second component of the three-way alignment model (Section 1.4): alignment within the judge. It proceeds in the order in which decisions arise in practice: whether an LLM is the right evaluator at all (Section 4.1), which judging format to use (Section 4.2), how to write the prompt (Section 4.3), what the judge should output (Section 4.4), and how to run a development loop that produces an honest estimate of the judge’s quality (Section 4.5). The organizing principle throughout is that a judge prompt is a rubric that a machine executes. The properties that make a human rubric reliable, namely precise definitions, explicit boundary rules, worked examples, and a narrowly defined option for insufficient information, matter equally for the judge. Any vagueness in the rubric remains vague for the model, with the difference that the model will not ask for clarification.
The examples use Mise, a fictional cooking assistant that answers questions about recipes, substitutions, techniques, and food safety (introduced in Section 1.5). Three Mise evaluation tasks appear here:
- Answer-quality task: does the response fully answer what the user asked? Labels: Fully / Partially / Does not answer.
- Food-safety task: does the response contain unsafe food-handling advice? Labels: Safe / Unsafe. Unsafe responses are rare (about 3% of traffic) but severe.
- Pairwise preference task: which of two responses would the user rather receive? Labels: A / B / Tie.
Helpful background: the decision brief in Section 1.2, the executable rubric in Section 2.2, and the agreement statistics in Section 3.2. The chapter assumes that a rubric and a set of human reference labels already exist.
Is an LLM judge needed?
Consider a Mise response that says “Bake at 180°C (350°F).” An LLM judge can be asked whether the conversion is correct, and it will usually answer correctly. A three-line function that extracts the two numbers and compares them within a stated tolerance, however, answers consistently every time, at no cost, in microseconds. The same observation applies to many properties in a realistic evaluation: format, schema compliance, length limits, and checkable facts do not require a model that reads and interprets language. LLM-as-a-judge has become a default response to the question “how do we evaluate this?”, yet it is slow, costly, somewhat stochastic, and capable of being confidently wrong. A sound first step is therefore to list every property to be measured and to identify the simplest sufficient evaluator for each.
| Evaluator | Well suited to | Poorly suited to |
|---|---|---|
| Deterministic rule | Format, schema, length limits, prohibited strings, required fields | Any property that depends on meaning |
| Executable verification | Arithmetic, unit conversions, executable code, lookups against a trusted table | Cases in which no reference or oracle exists |
| Small trained classifier | Stable, high-volume questions with abundant labeled data | New criteria; requires retraining whenever the rubric changes |
| LLM judge | Semantic, open-ended judgments that follow a written rubric | Cost, latency, systematic biases (Section 5.2), plausible but incorrect verdicts |
| Human (expert) | Novel, ambiguous, high-stakes, or subjective judgments; construction of the reference itself | Cost, speed; requires its own quality control (Chapter 3) |
The answers to this exercise point toward a hybrid architecture, a configuration that many mature evaluation systems adopt:
- Inexpensive checks first. Schema, format, and verifiable facts are checked on every item in milliseconds. A response that fails a hard check frequently does not need to reach a judge at all.
- LLM judges where semantic interpretation is required. A question such as “Does the response answer what the user asked?” is the kind of judgment for which an LLM judge is appropriate.
- Humans where independent judgment is required. Humans build and refresh the reference set, adjudicate difficult cases, and audit the judge itself. This role does not disappear; Chapter 6 describes how to use it efficiently.
A frequently observed design is a single, long judge prompt that checks format, arithmetic, safety, helpfulness, and tone, and returns one score from 1 to 10. Such a judge is expensive, it makes arithmetic errors at a small but nonzero rate where a short function would make none, and when the score declines, the team cannot determine which of the five properties caused the decline.
Once the properties that genuinely require an LLM have been identified, the next design decision concerns the form in which the judgment is requested.
Choose the judging format
Suppose the Mise team needs two numbers: the proportion of responses that fully answer the user’s question, and whether model version 4 is preferred over version 3. The first is a rate over individual responses; the second is a comparison between two systems on the same prompts. Although both concern response quality, they call for differently shaped judgments. The MT-Bench study, for example, compared three such shapes (pairwise comparison, single-answer grading, and reference-guided grading) and found that each carried distinct strengths and failure modes[1]. The format determines consistency, exposure to bias, cost, and the uses to which the output can be put. The dimensions also overlap: a reference-guided judge can be pointwise or pairwise, and either can be decomposed into separate criteria.
The following table summarizes when each format is appropriate.
| Format | Appropriate when | Principal risk |
|---|---|---|
| Pointwise | A rate (“% fully answered”) or a launch threshold is required | Scale drift, leniency |
| Pairwise | Two model versions are compared on the same prompts (the pairwise preference task) | Position bias, ties, no absolute signal |
| Listwise | Many candidates must be ranked economically | Order and set-size effects |
| Reference-guided | A verified answer exists (for example, food-safety facts) | Incomplete references |
| Criterion-decomposed | The team needs to know why quality changed | Additional calls; a combination rule is required |
How fine should the scale be?
A 1–10 score appears to capture more information than a three-level label. In practice, neither humans nor models tend to apply fine scales consistently: the difference between a 6 and a 7 is rarely defined, so it is interpreted differently from one item to the next. Two patterns tend to be more robust:
- Few, defined levels. Binary or three-level scales, each level with a written definition and examples, matching the human rubric exactly.
- Decompose, then combine. Several binary criteria are judged separately and the score is derived from them. A result such as “4 of 5 criteria met” is reproducible in a way that an unanchored “7/10” is not.
When a continuous score is genuinely needed (for ranking, or to set a threshold later), G-Eval proposes computing a probability-weighted average over the score tokens instead of taking the single most likely score[4]. This yields finer-grained scores with fewer ties, but it requires token probabilities (or an estimate of them from repeated sampling, as G-Eval itself did with GPT-4) and still requires validation against human labels.
This handbook is deliberately model-agnostic. General-purpose frontier models are a common starting point; open evaluator models trained specifically for rubric-based grading, such as Prometheus 2[5], are an alternative when local execution or strict version control is required. In either case, trust is earned through the validation described in Chapter 5 and Chapter 6, not through the reputation of the model.
With the format chosen, the remaining question is how to express the rubric itself to the judge, which is the subject of the next section.
The judge prompt as an executable rubric
A Mise team that asks a judge “How good is this response, from 1 to 10?” will receive numbers, but the criteria behind those numbers remain inside the model, where they cannot be inspected, versioned, or aligned with the human rubric. In practice, many judge prompts consist of little more than such a sentence. The remedy is to write the prompt as an executable version of the human rubric developed in Section 2.2: the same question, the same label definitions, and the same boundary rules. The two prompts below, both written for the answer-quality task, illustrate the difference.
Practical notes
Several practices, each learned from a recurring failure, improve the reliability of rubric-style prompts.
The label definitions and decision rules should be copied verbatim from the human guideline. If the two documents diverge, part of any measured judge–human disagreement reflects the editing gap between them rather than a genuine difference in judgment. Keeping both in one versioned source file, from which the guideline and the prompt are generated, prevents this divergence.
If two of three worked examples carry the label FULLY, or the FULLY example always appears last, the prompt shifts the judge’s prior toward that label. A safer design includes one example per label, keeps the examples of similar length, and rotates their order across prompt versions to confirm that results do not depend on it.
Decision rule 2 (“politeness is not an answer”) exists because Response B in Section 4.2 misleads vague judges. Rules derived from observed errors are more effective than generic instructions such as “be strict.”
A worked example copied from the locked test set inflates accuracy on exactly that item and on its near-duplicates. Few-shot examples should be drawn only from the development split (Section 4.5).
Expect the criteria to change
Writing the prompt usually changes the rubric. Shankar et al. observed that people refine their evaluation criteria while grading outputs: inspecting real responses reveals cases that the original criteria did not anticipate, a phenomenon the authors term criteria drift[6]. This is expected and beneficial, provided it is handled as a rubric change: the rubric version is incremented, the human guideline is updated alongside the prompt, and affected human labels are re-checked. Unrecorded edits to the judge prompt alone are the mechanism by which the judge and the human reference gradually come to measure different constructs.
A well-specified prompt determines what the judge is asked to do. The next section concerns what the judge returns, and how that output can be made checkable.
Outputs: labels, scores, and evidence
Suppose the judge labels a Mise response FULLY, and nothing else is recorded. A reviewer who later doubts the label has no way to tell whether the judge read the response correctly, overlooked a second question, or credited the response with content it never contained. The last failure is more common than might be expected, as the exercise below demonstrates. Two design choices address it: requesting structured output (JSON that conforms to a schema and is validated in code), and requiring a quoted evidence span. A label alone records what the judge decided; a label with evidence allows a reviewer to check whether the decision is supported. The evidence can also be checked automatically: if the judge claims the response says something, that string should appear in the response.
A verified quotation is necessary but not sufficient, since a quotation can exist and still fail to support the label. Nevertheless, this inexpensive check catches one specific and recurring failure, evidence that the judge has invented, and its failure rate is itself a useful health metric for the judge.
Reasoning before the verdict
Requesting a short reasoning field before the label can improve judgments that require several steps (for example, listing the user’s questions and then checking each one), and it makes errors easier to inspect, although the size of the benefit varies across tasks and models. The costs are additional tokens, additional latency, and the risk of treating fluent reasoning as proof. A persuasive explanation attached to an incorrect label remains an incorrect label. Reasoning should therefore be kept short and structured, and labels should be validated against human references regardless of the apparent quality of the explanations.
Four quantities that are easily confused
A judge can emit several numbers, and each answers a different question. Conflating them is a frequent source of error in downstream reporting.
| Quantity | Question it answers | Example |
|---|---|---|
| Quality score | How good is the evaluated response? | “Answer quality: 4 / 5” |
| Confidence | How likely is the judge’s verdict to be correct? | “P(label is correct) = 0.8” |
| Disagreement | How much do repeated runs or multiple judges vary? | “3 of 5 runs returned PARTIALLY” |
| Evidence sufficiency | Is there enough information to judge at all? | “Response truncated: CANNOT_ASSESS” |
A score of 4/5 does not mean “80% confident.” Likewise, a confidence value reported by the model is not a probability until it has been checked against human labels; this check, called calibration, is treated in Section 6.3 (score mappings) and Section 6.4 (confidence). To retain the option of a usable confidence signal later, the raw inputs should be logged from the start: token probabilities where available, and verdicts from a small number of repeated runs.
With the prompt and output format in place, the judge is ready to be developed iteratively against human labels. The final section describes how to do so without overstating its quality.
A development loop that produces honest estimates
Judge development is iterative: the team runs the judge, reads the items on which it disagrees with human labels, revises the prompt, and repeats. Consider a team that performs thirty such revisions on the same 200 items and observes accuracy rising from 74% to 93%. The improvement may be real, or the prompt may have been fitted to those particular 200 items rather than to the rubric they represent. The two explanations cannot be distinguished unless some labeled data has been withheld from the revision process.
Split the data before development begins
The labeled data should be divided into three parts, each with a distinct role.
| Split | Used for | Rule |
|---|---|---|
| Development | Reading errors, revising the prompt, selecting few-shot examples | May be inspected freely |
| Calibration | Fitting thresholds, score mappings, and confidence (Chapter 6) | Not used to edit the prompt |
| Locked test | Reporting final performance | Run rarely; individual errors are not read before the final report |
The split should be made by the appropriate group. If several Mise items originate from the same conversation, the same user, or the same templated question (“Can I substitute X for Y in Z?”), they belong in the same split. Otherwise, near-duplicates leak across splits and the test set no longer functions as a test.
The practical rule implied by the figure is that when development accuracy continues to rise while test accuracy does not, prompt editing should stop. The remaining errors are more likely to be addressed by a rubric change, a different format (such as criterion decomposition), a stronger model, or routing of those cases to human reviewers. The same reasoning applies with greater force to automated prompt optimizers, which evaluate far more revisions than a person would and therefore overfit a small development set more quickly.
Systematic error analysis
Each revision should respond to a pattern of errors rather than to the most recently read error. A simple procedure, adapted from qualitative coding, is as follows:
- Read the disagreements. Retrieve every development item on which the judge and the human reference disagree, and read the full item together with the judge’s reasoning.
- Annotate each one. Write a short free-text note per item (for example, “treated background facts as an answer” or “missed the second question”).
- Group the notes into categories and count them. A small number of categories typically accounts for most errors.
- Address the largest category with one targeted change, then re-run the full development set and examine the entire confusion matrix, not only the category that was addressed.
- Check whether the human label was wrong. Some disagreements are errors in the reference. These should be sent to adjudication rather than taught to the judge (Section 6.2).
Why examine the entire confusion matrix?
Suppose the judge over-predicts FULLY, and the team adds the rule “when in doubt, choose PARTIALLY.” Errors in which PARTIALLY responses are labeled FULLY decrease, but genuinely complete answers are now also labeled PARTIALLY. Overall accuracy may change very little while the errors simply reverse direction. Only per-class precision and recall, examined after every change, reveal this trade-off.
Version every component
When a judged metric changes in a subsequent month, the team needs to determine whether the evaluated system changed, the judge changed, or the standard changed. This question can be answered only if every run records its inputs, as in the following manifest:
{
"run_id": "mise-answer-quality-2026-10-02-a",
"dataset": { "name": "mise-answer-quality-test", "version": "v3",
"n": 600, "split_seed": 17 },
"rubric": { "name": "answer-quality", "version": "2.1" },
"judge_model": { "provider_model_id": "<exact model snapshot>" },
"judge_prompt": { "version": "answer-quality-judge-14", "sha256": "9f2c..." },
"decoding": { "temperature": 0, "max_tokens": 400, "n_samples": 1 },
"reference": { "labels": "mise-answer-quality-human", "version": "v3-adjudicated" },
"timestamp": "2026-10-02T14:05:00Z"
}The exact model snapshot should be recorded, not a moving alias. Hosted models are updated over time, and an alias that silently points to a new version is a common cause of a metric that changes when nothing else appears to have changed.
Cost and latency as design constraints
A judge that is two percentage points more accurate but five times more expensive is not necessarily preferable if its cost limits evaluation to a tenth of the traffic. Cost per judged item should be estimated early (prompt length grows quickly with few-shot examples and long conversations) and treated as part of the choice of operating point. Section 6.4 shows how routing only uncertain items to human reviewers can retain most of the accuracy at a fraction of the cost; the same logic applies to routing them to a more expensive judge.
A judge built in this way has a clear specification and an honest accuracy estimate on held-out data. Accuracy on a test set, however, does not establish that the judge is stable, unbiased, or robust to irrelevant changes in its input. Chapter 5 describes how to test these properties directly, completing the assessment of alignment within the judge.
Summary & quiz
- Use the simplest sufficient evaluator for each property: rules and code first, LLM judges for semantic judgments, and humans for independent judgment.
- Choose the format according to the decision: pointwise for rates, pairwise for comparisons (judged in both orders), and criterion-decomposed for diagnosis.
- Prefer a small number of defined levels, or decomposed binary criteria, over unanchored fine-grained scales.
- Write the judge prompt as an executable rubric that shares its text with the human guideline, includes a narrow option for insufficient information, and handles untrusted content explicitly.
- Require structured output with verifiable evidence, and keep quality, confidence, disagreement, and evidence sufficiency separate.
- Split the data into development, calibration, and locked test sets by group; stop tuning when only development accuracy improves; and version every run.
Mise responses sometimes include a recipe card that must be valid JSON with a “servings” field. How should this property be evaluated?
A pairwise judge prefers model v4 over v3 in 61% of pairs. What should be checked before the result is reported?
The judge outputs “score: 4/5”. A colleague proposes to treat this as 80% confidence that the response is good. What is the problem with this interpretation?
After 25 prompt revisions, development accuracy rose from 74% to 93%; locked-test accuracy rose from 72% to 80% and has not changed over the last 12 revisions. What is the most appropriate next step?
While reading outputs, the team concludes that answers addressing a different storage situation than the one the user described should not count. What should be done?
References
- Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks.
- Bavaresco, A., Bernardi, R., Bertolazzi, L., et al. (2025). LLMs instead of human judges? A large scale empirical study across 20 NLP evaluation tasks. ACL 2025.
- Wang, P., Li, L., Chen, L., et al. (2024). Large language models are not fair evaluators. ACL 2024.
- Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment. EMNLP 2023.
- Kim, S., Suk, J., Longpre, S., et al. (2024). Prometheus 2: An open source language model specialized in evaluating other language models. EMNLP 2024.
- Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. UIST 2024.