1. Evaluation as Measurement

How evaluations fail

The use of LLMs as evaluators gained traction as studies reported that strong models agree with human preferences about as often as humans agree with one another. On the MT-Bench benchmark, for example, when tied votes were excluded, the agreement between a strong LLM judge (GPT-4) and expert annotators reached 85%, slightly above the 81% agreement observed among the experts themselves; with ties included, both figures were lower (66% and 63%)[1]. Subsequent large-scale comparisons have been more cautious: across 20 natural language processing tasks, the correlation between LLM judges and human annotators varied widely across datasets, judged properties, and judge models[2]. In parallel, research on human annotation has shown that disagreement among annotators is often systematic rather than random noise[3]. Taken together, these findings suggest that neither the human labels nor the automated judge can be assumed reliable without examination.

The following composite scenario illustrates how such problems arise in practice. It concerns Mise, a fictional cooking assistant that answers questions about recipes, substitutions, techniques, and food safety, which serves as the running example for the handbook. A team wishes to determine whether a new version of Mise is better than the current one. Crowdworkers are asked “Is this response helpful? (1–5)” for 2,000 conversations, and the mean rating rises from 3.9 to 4.2. The team then replaces the crowdworkers with an LLM judge that agrees with the crowd on 85% of items and runs it on every subsequent release. Six months later, a user reports that Mise advised them that rice left at room temperature overnight was safe to eat.

Each step in this process had a defensible rationale, and each also left a gap:

Additional annotation volume, a larger judge model, or a more sophisticated aggregation method would not have corrected these problems, because they originated in the design of the measurement and compounded across steps. The gaps also fall into recognizable groups: an unclear target (what was being measured), an unexamined human reference (whether the human labels were consistent and correct), an unexamined judge (whether the automated evaluator was consistent), and an unexamined relationship between the two (whether the judge agreed with humans on the items that mattered). The remainder of this preface addresses these groups in order, beginning with the target.

Start from the decision

The first gap in the scenario, an unclear target, is the most consequential because every later step inherits it. The difficulty becomes concrete when a single response is assessed against several plausible targets. The figure below shows one Mise conversation; selecting a different property changes the verdict, even though the conversation does not change.

In measurement terms, each of these properties is a different construct: an abstract quality that cannot be observed directly and must be operationalized through a concrete procedure, such as a rubric applied by annotators[4]. Choosing among constructs is not a technical question; it depends on what the evaluation is for. A team deciding whether a release is safe to ship needs the food-safety construct, while a team optimizing engagement may care about satisfaction. For this reason, the most reliable starting point for an evaluation is a written statement of the decision it will inform.

The decision-ready brief fixes five elements, each of which constrains a later design choice:

1. Construct: the property of interest (for example, “fully answers the user’s question”), stated precisely enough that two people could disagree about whether a given response satisfies it.
2. Unit of analysis: a single response, a conversational turn, a whole conversation, or a user. Rating responses and rating conversations produce different numbers from the same logs.
3. Population: the traffic, languages, and time window that the final number will claim to describe.
4. Estimand: the quantity to be estimated, such as “the proportion of eligible responses that fully answer the question” or “the probability that v4 is preferred over v3 in a head-to-head comparison.”
5. Decision rule and tolerance: which result leads to which action, and how much statistical uncertainty is acceptable.

Different purposes lead to substantially different designs, even when the underlying data are the same. The figure below summarizes how four common purposes change the sampling plan, the label format, and the main driver of annotation volume.

Fixing the construct answers the question of what is measured. It does not yet answer what a correct label for that construct would be, or how disagreement among annotators should be interpreted. That depends on the kind of reference the construct admits, which is the subject of the next section.

Three kinds of reference

Consider three questions about Mise responses. Whether “350°F is approximately 175°C” is correct can be checked against a conversion formula. Whether a response “fully answers the question” depends on how a rubric defines “fully.” Whether one reply “feels more encouraging to a nervous beginner” depends on the reader. If two annotators disagree, the appropriate response differs in each case: in the first, one of them is wrong; in the second, either one is wrong or the rubric is incomplete; in the third, the disagreement may itself be the finding. It is therefore useful to classify the evaluation question before designing the annotation task, because the classification determines what a reference label means.

The boundary between rubric-interpreted and subjective questions is partly a design decision. Röttger et al. describe two annotation paradigms: a prescriptive paradigm, in which annotators are instructed to apply a guideline’s view, and a descriptive paradigm, in which annotators are asked to report their own view[5]. Both are legitimate, but combining them unintentionally within one task tends to produce data that are difficult to interpret, since some annotators follow the guideline and others follow their personal judgment. For genuinely subjective tasks, a substantial body of work argues that annotator disagreement carries information and should not be discarded[6][3].

With a construct fixed and the kind of reference identified, the target of the evaluation is defined. The remaining gaps from Section 1.1 concern the instruments that measure this target, namely the human annotators and the LLM judge, and the relationship between them.

The three-way alignment model

In the scenario of Section 1.1, the team summarized the evaluation with one number: the judge “agreed with humans 85% of the time.” That single number combined three separate questions. Were the human labels themselves consistent? Was the judge consistent with itself? And did the judge agree with the human labels on the items relevant to the decision? A high value could hide a failure in any of the three, and a low value would not indicate which one had failed.

Measurement theory offers a useful distinction here. Reliability refers to whether a measurement procedure produces consistent results when repeated; validity refers to whether it measures the intended construct[7][4]. A procedure can be highly reliable and still invalid, for example when every annotator consistently applies the same mistaken rule. Applied to an evaluation that combines human annotators and an LLM judge, this distinction suggests examining three forms of agreement separately, each anchored to the decision and construct defined in Sections 1.2 and 1.3:

1. Alignment within humans: whether annotators apply the rubric consistently with one another, with themselves over time, and in the way the rubric intends.
2. Alignment within the judge: whether the judge produces the same verdict across repeated runs, swapped presentation orders, paraphrased prompts, and irrelevant edits to the input.
3. Alignment between humans and the judge: whether the judge agrees with the human reference closely enough, on the items and subgroups that matter for the decision.

Separating the three is useful because each has distinct failure modes, distinct metrics, and distinct remedies. When the judge and the human labels disagree, the decomposition indicates where to look: at the human reference, at the judge, or at the rubric that both are meant to follow. The figure below summarizes each component and where it is treated in the handbook.

Two consequences of this framework recur throughout the handbook.

Agreement between humans and the judge is bounded by agreement among humans

If two trained annotators agree with each other at a chance-corrected agreement of Cohen’s κ = 0.55 (a statistic defined in Section 3.2, where 1 indicates perfect agreement and 0 indicates agreement no better than chance), a judge cannot meaningfully be expected to agree with individual annotators at κ = 0.9, because there is no single human answer for it to match. A judge whose agreement with humans approaches the level of human–human agreement is already performing comparably to an additional annotator. The informative comparison is therefore between the judge and this human ceiling, rather than between the judge and perfect agreement; Zheng et al. use exactly this comparison when assessing LLM judges[1]. Section 6.1 develops the idea further.

High agreement does not establish validity

Agreement is evidence of reliability, not of validity. If all annotators have learned the same incorrect rule (for example, “a polite and confident answer is helpful”), they will agree closely, and a judge tuned to match them will agree closely as well. All three alignments can then appear excellent while the measurement targets the wrong construct. This is why the handbook treats the decision and the construct as the anchor for every alignment, and why independent checks against expert-verified items appear in several later chapters.

Running example and reading paths

Abstract guidance on evaluation is difficult to apply without a concrete case, so every chapter uses the same fictional product. Mise, the cooking assistant introduced in Section 1.1, is evaluated on three tasks. They were chosen so that together they cover the three kinds of reference described in Section 1.3, and each chapter refers to them by the names below.

Public datasets for practice

Readers who wish to apply the techniques to real data may use the following public resources, which correspond closely to the three tasks. Each provides multiple human judgments per item, which is a prerequisite for studying agreement.

• MT-Bench human judgments: approximately 3,300 expert pairwise preferences over responses from six models to 80 multi-turn questions[1]. Suitable for pairwise preference experiments and position-bias analysis.
• SummEval: 1,600 news summaries rated for coherence, consistency, fluency, and relevance by both experts and crowdworkers[8]. Suitable for examining divergence between expert and crowd ratings.
• HelpSteer2: approximately 20,000 responses rated on five attributes (helpfulness, correctness, coherence, complexity, and verbosity) on a 0–4 scale[9]. Suitable for pointwise, multi-criterion rubrics similar to the answer-quality task.
• Chatbot Arena: large-scale crowdsourced pairwise preferences collected from real users[10]. Suitable for studying descriptive preference data at scale.

Reading paths

Each chapter is designed to be read independently in 10–15 minutes and begins with a short note on the context and prerequisites it assumes. The following paths are suggested for common roles.

The first component of the framework, alignment within humans, begins with the design of the annotation task itself. Chapter 2 describes how to turn a decision brief into a rubric, a sample, and an annotation interface, and how to pilot them before collecting data at scale.

Summary & quiz

References

1. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks.
2. Bavaresco, A., Bernardi, R., Bertolazzi, L., et al. (2025). LLMs instead of human judges? A large scale empirical study across 20 NLP evaluation tasks. ACL 2025.
3. Plank, B. (2022). The “problem” of human label variation: On ground truth in data, modeling and evaluation. EMNLP 2022.
4. Jacobs, A. Z., & Wallach, H. (2021). Measurement and fairness. FAccT 2021.
5. Röttger, P., Vidgen, B., Hovy, D., & Pierrehumbert, J. (2022). Two contrasting data annotation paradigms for subjective NLP tasks. NAACL 2022.
6. Aroyo, L., & Welty, C. (2015). Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1).
7. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
8. Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021). SummEval: Re-evaluating summarization evaluation. TACL, 9.
9. Wang, Z., Dong, Y., Delalleau, O., et al. (2024). HelpSteer2: Open-source dataset for training top-performing reward models. NeurIPS 2024 Datasets and Benchmarks.
10. Chiang, W.-L., Zheng, L., Sheng, Y., et al. (2024). Chatbot Arena: An open platform for evaluating LLMs by human preference. ICML 2024.