6. Aligning Humans and the Judge

Jiaming Qu/~15 min read/Between the two

An LLM judge is useful only to the extent that its verdicts can stand in for the human judgments they replace. Establishing this is harder than it first appears. A large comparison across twenty evaluation tasks found that agreement between LLM judges and human annotators varies widely by task, dataset, and property being judged[1], so an agreement figure reported for one benchmark cannot be assumed to hold for another. In addition, a single agreement statistic can conceal precisely the failures on which a decision depends.

This chapter addresses the third component of the three-way alignment model introduced in Section 1.4: alignment between the human reference and the judge. Chapters 2 and 3 concerned alignment within humans, and Chapters 4 and 5 concerned alignment within the judge. Section 6.1 defines what “good alignment” can mean and how to set a realistic benchmark for it. Section 6.2 describes how to diagnose disagreements before acting on them, and Section 6.3 reviews the available interventions in order of cost. The final two sections present two designs that make an imperfect judge usable: allowing it to abstain (Section 6.4), and correcting its aggregate bias with a small human sample (Section 6.5).

Context for this chapter

The examples use Mise, a fictional cooking assistant that answers questions about recipes, techniques, and food safety (introduced in Section 1.5). Two of its evaluation tasks appear here:

Answer-quality task: does the response answer what the user asked? Labels are Fully answers, Partially answers, and Does not answer.
Food-safety task: does the response contain unsafe food-handling advice? Labels are Safe and Unsafe. Unsafe responses are rare (about 3–4% of traffic) but severe.

Helpful background: the decision a measurement serves (Section 1.2), the three-way alignment model (Section 1.4), chance-corrected agreement and Cohen’s κ (Section 3.2), constructing a reference label from several annotators (Section 3.4), and the separation of development, calibration, and test data (Section 4.5).

What good alignment looks like

Consider two status reports about the same food-safety judge. The first states that the judge agrees with expert labels on 95% of responses. The second states that the judge identifies 14 of the 40 unsafe responses in the same sample. Both statements are accurate (the interactive figure later in this section reproduces them), yet they support opposite conclusions about whether the judge can be deployed. The apparent contradiction arises because “the judge is aligned with humans” can refer to at least four distinct properties. They are not interchangeable, and a judge can satisfy one while failing another.

Level	Question	Typical measure	Relevant when…
Item labels	Does the judge assign the same label as the human reference on each item?	Accuracy, Cohen’s κ, per-class precision and recall	The verdict on each item is acted upon (filtering, routing, flagging).
Item scores	Do higher judge scores correspond to higher human scores?	Spearman ρ, Kendall τ, mean absolute error (MAE) after calibration	Responses are ranked or thresholded by score.
System ranking	Does the judge order models in the same way humans do?	Rank correlation across systems, agreement on pairwise win rates	The team selects between model versions or maintains a leaderboard.
Aggregate estimate	Is the judge’s population rate close to the rate humans would measure?	Bias of the estimate and the width of its confidence interval	A dashboard reports a figure such as “x% of responses are unsafe.”

These levels can diverge considerably. A judge with modest item-level agreement may still rank systems almost perfectly, because item-level errors average out across hundreds of prompts. For example, the length-controlled version of AlpacaEval reports a Spearman correlation of about 0.98 with Chatbot Arena system rankings[2]. This figure is strong evidence about model ordering and weak evidence about whether any individual verdict is correct. Conversely, a judge can agree with humans on 95% of items and still produce a substantially biased rate for the class of greatest concern. The appropriate level is therefore determined by the decision the evaluation serves (Section 1.2).

Compare against the human ceiling, not against 100%

A second difficulty concerns the reference point. Teams frequently compare judge–human agreement against perfect agreement and conclude that any shortfall is the judge’s fault. However, trained annotators do not agree perfectly with one another either. The agreement they reach on the same items, which this handbook calls the human ceiling, is the realistic upper bound for between-alignment: if there is no single human answer, a judge cannot be expected to reproduce one. For example, if two trained annotators reach κ = 0.6 on Mise’s answer-quality task, a judge at κ = 0.58 against a single annotator is performing approximately as another annotator would. (Cohen’s κ rescales observed agreement so that 0 corresponds to the agreement expected by chance and 1 to perfect agreement; see Section 3.2.) The MT-Bench study framed its principal result in this way: with tied votes excluded, GPT-4 agreed with expert preferences on 85% of comparisons, slightly above the 81% agreement among the experts themselves (with ties included, the figures were 66% and 63%)[3].

Human ↔ human

Judge ↔ human

Further detail: comparing like with like

A majority label from three annotators is less noisy than any single annotator. A judge compared against the majority will therefore usually score higher than a human compared against another single human, and may appear to exceed human performance for a purely statistical reason. A fair comparison either scores the judge against one annotator at a time (and averages the results), or scores a held-out annotator against the majority of the remaining annotators and applies the same construction to the judge.

A more formal version of this idea is the alternative annotator test: one annotator is left out, and the test asks whether the judge agrees with the remaining annotators at least as well as the left-out human does, up to a small tolerance that reflects the cost savings of automation[4]. Repeating the procedure for each annotator yields a statistical argument that the judge can substitute for a person on the task, which is a considerably more defensible claim than a bare agreement percentage.

A single number can conceal the critical slice

The two status reports at the beginning of this section can now be reconciled. For skewed tasks such as Mise’s food-safety task, in which roughly 3–4% of responses are unsafe, percent agreement is dominated by the large and easy majority class. For this reason chance-corrected metrics and per-class recall are essential; one study of LLM judges, for example, found that a chance-corrected coefficient (Scott’s π) separated judges far better than percent agreement, and recommended reporting both[5]. Because judge–human agreement also varies considerably across tasks and datasets[1], a general claim such as “LLM judges are about 80% aligned with humans” cannot substitute for a measurement on the task at hand.

	Judge: Unsafe	Judge: Safe
Expert: Unsafe
Expert: Safe

Agreement

Cohen’s κ

Unsafe recall

Unsafe precision

Practical note

Place the rare, high-severity class in the first row of every alignment report, together with its count. A line such as “Unsafe recall 35% (14 of 40)” directs attention to the relevant question, whereas “Agreement 95%” tends to close the discussion prematurely.

Once the relevant level of alignment and a realistic benchmark have been established, the next step is to examine the items on which the judge and the reference disagree. The following section explains why these disagreements should be diagnosed before any change is made to the judge.

Diagnosis before correction

Consider a Mise user who asks, “How long do I boil eggs for jammy yolks?” and receives the reply, “Great question! Boiling times really vary by preference, so have fun experimenting!” The human annotator labels the response Does not answer; the judge labels it Fully answers. The natural reaction is to amend the judge prompt, and in this instance that reaction would be correct. In many other disagreements, however, the error lies elsewhere. There are at least four explanations for a disagreement, and only one of them is that the judge is wrong:

Judge error. The judge misread the response or applied the wrong rule.
Human error. The annotator overlooked evidence, worked too quickly, or applied a personal rule.
Rubric ambiguity. Both readings are defensible under the written guideline.
Missing information. The judge and the annotator did not see the same input, or neither had sufficient context.

If the prompt is adjusted in response to every disagreement, the judge learns to reproduce annotator mistakes and to conceal gaps in the rubric. The agreement figure improves while the validity of the reference set deteriorates. The more reliable procedure is to diagnose each case and direct it to the appropriate remedy, as summarized below.

Root cause	Observable symptom	Where the remedy belongs
Judge error	The judge’s quoted evidence does not support its label; the same pattern recurs across items	Judge prompt, examples, or model (Section 6.3)
Human error	An expert re-reading the item quickly finds the overlooked evidence	Correct the reference; retrain or recheck the annotator (Section 3.1)
Rubric ambiguity	Annotators also disagree among themselves; the guideline does not address the case	Revise the rubric (Section 2.2), add a boundary example, then relabel affected items
Missing information	One party saw context the other did not; the item cannot be decided on its own	Fix the data pipeline; add an “insufficient information” option
Genuine subjectivity	Disagreement persists after discussion among experts	Retain the distribution of judgments; do not score the judge on these items as if they had one answer

From cases to patterns

Reading individual cases builds understanding, but patterns indicate what to change. Once a few hundred disagreements have been labeled with their causes, they can be grouped along several dimensions:

By label pair. “Human = Does not answer, Judge = Fully answers” indicates a different defect from “Human = Partially answers, Judge = Does not answer.”
By input slice. Question type, language, conversation length, single-turn versus multi-turn.
By response features. Length, presence of lists, hedging, exclamation marks, emoji.
By annotator. If one annotator accounts for 40% of disagreements, the issue concerns annotator quality (Section 3.3) rather than the judge.

Suppose that most disagreements on Mise’s answer-quality task are of the form “Human = Does not answer, Judge = Fully answers,” and that the responses involved are unusually enthusiastic in tone. This observation is a hypothesis: the judge may be treating a friendly tone as if it were substantive content. The hypothesis can be tested with controlled pairs, the same technique used for the bias tests in Section 5.2. One takes 30 responses that do not answer the question, writes a neutral and an enthusiastic version of each with identical content, and checks whether the judge’s label changes. If it does, a mechanism has been identified and the remedy can target it directly. If it does not, the team has avoided a prompt change that would have had no effect.

Caution

Evaluation criteria tend to shift as evaluators read outputs. Practitioners often clarify what they mean by “good” only after grading several dozen examples, so the rubric applied during triage may differ from the rubric that was written down[6]. When the standard changes during triage, the new rule should be written into the rubric and the affected items relabeled; otherwise the judge is evaluated against a standard it was never given.

Diagnosis identifies a cause. The next question is which intervention addresses that cause at the lowest cost.

Ways to close the gap

Interventions on a judge differ in cost by orders of magnitude. Correcting a mislabeled reference item takes minutes; fine-tuning a dedicated judge model can take weeks and requires thousands of high-quality labels. In practice, expensive interventions are sometimes attempted before the inexpensive ones, for example fine-tuning a judge against a reference set that still contains annotator errors. Once the cause of a disagreement is known, the cheapest intervention that addresses that cause should be tried first. The following ordering, from least to most expensive, is a reasonable default:

low costCorrect the reference. Fix human errors and relabel items affected by rubric changes. This is often the largest single improvement, and it requires no change to the judge.
low costRevise the rubric in the prompt. Add the rule the judge is missing, for example: “A friendly tone without the requested information is Does not answer.”
low costImprove the examples. Replace generic few-shot examples with boundary cases drawn from the diagnosed failure clusters (taken from the development split, never from the test split).
moderateCalibrate the scores. Learn a mapping from judge scores to the human scale (illustrated below).
moderateChange or combine judges. Use a stronger model, or a panel of different models. A panel helps only if the members’ errors are not strongly correlated.
high costFine-tune a judge. Justified when thousands of reliable labels exist and the task is stable; open models trained for evaluation are available as starting points[7].
ongoingRetain humans in the loop for the cases the judge cannot handle reliably (Section 6.4).

Inspect the full confusion matrix after every change

An intervention aimed at one failure cluster almost always affects others. The tables below show a plausible effect of adding the friendly-tone rule from Section 6.2 to the judge for Mise’s answer-quality task, measured on a held-out set of 300 items (the numbers are illustrative).

Before

Human ↓ / Judge →	Fully	Part.	Not
Fully	150	12	3
Partially	22	55	8
Does not	18	7	25

Recall for Does not answer: 25/50 = 50%.

After adding the rule

Human ↓ / Judge →	Fully	Part.	Not
Fully	138	15	12
Partially	17	56	12
Does not	5	7	38

Recall for Does not answer: 38/50 = 76%. However, 12 good, enthusiastic answers are now labeled as non-answers.

Overall agreement changes only from 76.7% to 77.3%. A team that examined only this figure would conclude that the change had no effect; a team that examined only the targeted cell would conclude that it was a clear success. The confusion matrix shows the actual trade-off: the judge is much better at detecting non-answers and somewhat worse on enthusiastic answers that are in fact good. Whether this trade-off is acceptable depends on the decision the evaluation serves, which is the reason for recording that decision before evaluation begins (Section 1.2).

Score calibration: correcting the scale

A different kind of gap arises with judges that output numeric scores. Such judges are often systematically lenient: a judge score of 4 corresponds, on average, to a human score of 3. If the relationship between the two scales is monotone, a mapping can be learned on calibration data. Isotonic regression fits the best non-decreasing step function from judge score to mean human score and is a standard tool for this form of calibration[8].

Items Mapping (mean human score per judge level) Perfect agreement (y = x)

Mean abs. error

vs. human mean

Mean bias

judge − human

Spearman ρ

rank agreement

Two cautions apply. First, the mapping should be fitted on a calibration split and evaluated on a separate test split (Section 4.5); a mapping fitted and scored on the same items will appear better than it is. Second, the mapping is valid only for the distribution on which it was fitted. A judge that is lenient on recipe questions may be strict on safety questions, so the mapping should be fitted per slice where data permit, and checked per slice in any case.

Panels of judges

Averaging several judges reduces variance only to the extent that their errors are independent. Three prompts run on the same model tend to make the same mistakes. A panel drawn from different model families can outperform a single large judge at lower cost[9], but this should be verified on the task: the team should compute how often the panel members are wrong together on the calibration set. If they are almost always wrong together, the panel is effectively a single, more expensive judge.

Each intervention above attempts to make the judge correct on more items. An alternative approach accepts that the judge will remain unreliable on some items and instead identifies those items in advance.

When to trust the judge

Suppose Mise’s judge is reliable on short, single-turn recipe questions but frequently wrong on long multi-turn conversations. Requiring the judge to label every item forces a choice between accepting its errors on the difficult items and abandoning automation altogether. A third design is often preferable: the judge labels the items on which it is reliably correct, and the remaining items are routed to human annotators. This approach is known as selective prediction and has been studied extensively in machine learning, including for deep neural networks[10]. Any such policy is described by two quantities:

Coverage = P(judge verdict accepted) Selective risk = P(verdict wrong | accepted)

Raising the acceptance threshold usually lowers both risk and coverage. The design question is which point on this curve is acceptable, stated either as a constraint (“maximize coverage subject to a selective risk of at most 3%”) or as a cost (“minimize the combined cost of shipped errors and human review”). Recent work shows how to select such thresholds so that agreement with humans on the accepted items is guaranteed with high probability, using a held-out calibration set[11].

Coverage

judged automatically

Selective risk

errors among accepted

Human review

per 10,000 items

Errors shipped

per 10,000 items

Confidence is one signal among several

As the simulation shows, confidently wrong verdicts limit what a confidence threshold can achieve. A robust escalation policy therefore usually combines several triggers:

Calibrated confidence, derived from token probabilities, repeated sampling, or a learned calibrator, and validated on held-out labels.
Disagreement between independent judges, or between repeated runs of the same judge.
Evidence problems: the quoted evidence span does not occur in the input, or the judge reports that the evidence is insufficient.
Novelty: inputs unlike those in the calibration set, such as a new language or a new product feature.
Severity: for high-stakes classes, a sample is routed to humans regardless of confidence. A verdict of “Safe, confidence 0.99” on a question about raw chicken still warrants inclusion in a random audit.

Fragile policy

“Accept every verdict with confidence ≥ 0.9.” The threshold was chosen by inspecting a few examples, was never checked on held-out data, and accepted items are not audited.

Defensible policy

The threshold is chosen on a calibration split to meet a stated risk target and verified on a separate test split; risk is reported per slice; and a small random sample of accepted items is audited every week.

Common pitfall

Once the policy is live, humans see only escalated items. If judge quality is measured only on those items, the judge will appear poor (it was sent the difficult cases), and errors among the accepted items will never be observed. A random audit of accepted items, sampled with known probability, is required.

Selective judging improves the reliability of individual verdicts. Many evaluation questions, however, concern a population-level quantity rather than individual items, and for these a different form of correction is available.

Correct the aggregate, not just the item

Suppose the product team asks what proportion of Mise’s responses last month contained unsafe advice. The judge flagged 5.5% of 50,000 responses. Section 6.1 showed, however, that a judge of this kind can miss a large share of unsafe responses, so 5.5% is likely to underestimate the true rate. Labeling all 50,000 responses by hand is not feasible, and labeling a random sample of 400 yields an unbiased but imprecise estimate. The question is whether the two sources of information can be combined. Answering it requires distinguishing two goals that are often conflated:

Goal A · every item correct

The judge’s verdict on each response is used directly: to block, flag, or route it. Item-level accuracy and per-class recall are what matter.

Goal B · the population figure correct

The question is “What fraction of Mise’s answers were unsafe this month?” Individual errors are acceptable provided the estimate is unbiased and its uncertainty is reported accurately.

For Goal B, a judge with a known bias remains valuable when a small random sample with human labels is also available. The principle of prediction-powered inference (PPI) is to apply the judge to every item for scale, and to use the human sample to measure and subtract the judge’s average error[12]. For a rate, the basic estimator is:

θ̂_PPI = 1N Σ_i ŷ_i + 1n Σ_j (y_j − ŷ_j)

Here ŷ denotes a judge label and y a human label. The first term is the judge’s rate on all N items. The second term is the average human-minus-judge difference on n randomly sampled items that carry both labels. The variance of the estimator is approximately Var(ŷ)/N + Var(y − ŷ)/n.

The second variance term is decisive. If the judge is usually correct, y − ŷ is mostly zero, its variance is small, and the corrected estimate has a narrower confidence interval than an estimate from the n human labels alone. If the judge is poor, PPI still does not inherit the judge’s bias, but its interval can be wider than the human-only interval; a power-tuned variant (PPI++) down-weights the judge so that the estimate is never less precise than one based on the human labels alone[13]. The same reasoning has been applied directly to model evaluation with automated judges[14].

Judge only Humans only (n) PPI (judge + humans) True rate

Conditions under which the correction fails

Non-random human sample. If the n items were selected because the judge was uncertain, or because they were disagreements, the measured difference does not describe the population. A probability sample is required, or weights based on known inclusion probabilities.
Different populations. The human sample must be drawn from the same N items, or the same distribution, that the judge scored.
Changing definitions. If the rubric changed between the collection of the human sample and the present, the correction measures the difference from an outdated standard.
Noisy human labels. PPI corrects toward the human reference, including its noise. This is why the quality of human labels (Chapter 3) must be established first.

This result connects directly to the three-way alignment model. A judge that is inadequate for Goal A on a high-severity class (for example, one that misses 40% of unsafe answers) can still be entirely adequate for Goal B, provided that a small, random, well-labeled human sample is maintained alongside it. Identifying which goal an evaluation serves is what allows an imperfect judge to be used responsibly.

The methods in this chapter establish alignment at a single point in time. In production, the input distribution, the annotator pool, the judge model, and even the definition of the construct all change. Chapter 7 describes how to detect these changes, how to set launch gates for new judge versions, and how to begin when no labeled data exist.

Summary & quiz

Alignment can be assessed at four levels: item labels, item scores, system rankings, and aggregate estimates. The appropriate level is the one on which the decision depends.
The judge should be compared against the human ceiling, like for like. A judge whose agreement approaches human–human agreement is performing approximately as another annotator would.
Per-class recall should be reported for rare, severe classes, because overall agreement conceals them.
Disagreements should be diagnosed (judge error, human error, rubric ambiguity, missing information) before the prompt is changed. Recurring patterns should then be tested with controlled pairs.
Interventions should be attempted in order of cost, with the full confusion matrix inspected after each change. Score mapping corrects the scale but not the ranking.
Selective judging trades coverage for risk. Thresholds should be chosen on held-out data, and accepted items should be audited at random.
For population metrics, combining a biased judge with a small random human sample (PPI) yields an unbiased estimate; when the judge is reasonably accurate, its interval is narrower than one from the human labels alone.

A judge reaches κ = 0.71 against the majority label of three annotators. Individual annotators agree with one another at κ = 0.64. What should be done before claiming that the judge outperforms humans?

A majority label is less noisy than a single annotator, so the judge has an easier target. A comparison in the style of the alternative annotator test places the judge and a human on equal footing.

A judge disagrees with the reference on 60 items. An expert re-reads ten of them and finds that the human label was wrong in six. What is the most appropriate next step?

Correcting the reference is the least expensive intervention. Teaching the judge to reproduce incorrect labels raises agreement while lowering validity.

After isotonic mapping, the judge’s mean absolute error falls from 0.9 to 0.4, but Spearman ρ is unchanged. What does this indicate?

A monotone mapping cannot reorder items, so rank correlation cannot improve (and may decrease slightly if levels are pooled). Mapping answers the question “What does a judge score of 4 mean?”; it does not correct ranking errors.

To reduce cost, a team sends only the judge’s low-confidence items to humans and then computes the PPI correction from those human labels. What is the problem?

The correction term estimates the judge’s average error over the population. Low-confidence items have larger errors than average, so the correction overshoots. A separate random sample (or weighting by known inclusion probabilities) is required.

A selective policy accepts 80% of items at 2% risk on the calibration set. Why might it ship more errors than expected in production?

The policy should be validated on a separate test split, risk should be checked per slice, and accepted items should be audited at random so that errors invisible to confidence can be observed.

References

Bavaresco, A., Bernardi, R., Bertolazzi, L., et al. (2025). LLMs instead of human judges? A large scale empirical study across 20 NLP evaluation tasks. ACL 2025.
Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv:2404.04475.
Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks.
Calderon, N., Reichart, R., & Dror, R. (2025). The alternative annotator test for LLM-as-a-judge: How to statistically justify replacing human annotators with LLMs. ACL 2025.
Thakur, A. S., Choudhary, K., Ramayapally, V. S., Vaidyanathan, S., & Hupkes, D. (2025). Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), ACL 2025.
Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. UIST 2024.
Kim, S., Suk, J., Longpre, S., et al. (2024). Prometheus 2: An open source language model specialized in evaluating other language models. EMNLP 2024.
Zadrozny, B., & Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. KDD 2002.
Verga, P., Hofstätter, S., Althammer, S., et al. (2024). Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv:2404.18796.
Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. NeurIPS 2017.
Jung, J., Brahman, F., & Choi, Y. (2025). Trust or escalate: LLM judges with provable guarantees for human agreement. ICLR 2025.
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., & Zrnic, T. (2023). Prediction-powered inference. Science, 382(6671).
Angelopoulos, A. N., Duchi, J. C., & Zrnic, T. (2023). PPI++: Efficient prediction-powered inference. arXiv:2311.01453.
Boyeau, P., Angelopoulos, A. N., Yosef, N., Malik, J., & Jordan, M. I. (2025). AutoEval done right: Using synthetic data for model evaluation. ICML 2025.