2. Designing the Annotation Task

Write a decision brief

Consider a common sequence of events. A team collects 3,000 quality ratings of Mise responses and presents the average at a review meeting. The first question from the audience is whether the sample includes recipe-generation traffic. It does not, because no one specified that it should. The team must now either relabel or present a number that the audience does not trust. Nothing was wrong with the raters or the statistics; the request itself was underspecified.

Section 1.2 argued that an evaluation should start from the decision it informs. The practical artifact for this is a one-page decision brief: a short document that the requester, the person running the annotation, and the consumers of the results agree on before work begins. Its main value is the discussion it forces. A complete brief answers six questions:

1. Purpose: which decision the labels will inform.
2. Unit: what one annotated item is (a response, a pair of responses, a conversation).
3. Population: which traffic, languages, and time window the result must describe.
4. Measure: the quantity to be estimated.
5. Decision rule: which result would change the decision.
6. Tolerance: how much statistical uncertainty is acceptable.

The builder below assembles these fields into a brief and flags the fields that are missing or too vague to design against.

Three practices make decision briefs effective:

• Involve the consumer of the result. The product manager who will read “82% of responses fully answered the question” has a working definition of “fully.” That definition should be elicited before data collection, not after. If two stakeholders define the construct differently, the team has identified its first rubric boundary at no cost.
• Draft the report sentence first. Write the sentence the final report is expected to contain, with blanks for the numbers (for example, “On [population], v4 fully answers [__]% of questions, so the team will [ship / not ship]”). If the sentence cannot be written, a field of the brief is missing.
• Fix the decision rule before seeing data. A threshold chosen after the results are known tends to be set just below whatever value was observed.

Once the brief names a measure such as “the share of responses that fully answer the question,” the next problem is that the measure is still only a phrase. The following section describes how to turn it into instructions that different people apply in the same way.

Turn a construct into an executable rubric

Suppose a user asks Mise, “Can I use olive oil instead of butter in banana bread? How much?” and the response explains that olive oil works well but gives no quantity. One careful rater labels the response Fully because the main question was answered; another labels it Partially because the second question was ignored. Both raters are reasonable. The disagreement arises because the label name “fully answers” does not specify what to do when a message contains more than one question.

A label name is not an instruction. An executable rubric is one that two trained people, reading it independently, would apply in the same way on most items, and that tells them how to proceed on the items where the decision is difficult. The contrast below shows an ambiguous rubric and an executable one for the answer-quality task.

The executable version is longer, but each added phrase prevents a specific and predictable disagreement. Selecting a highlighted phrase in the figure below shows which disagreement it prevents.

How specific should a rubric be?

Specificity involves a genuine tradeoff. Each added rule increases agreement on the cases it anticipates, but it can make the rubric brittle on cases it does not anticipate. A rubric with 40 special cases is difficult to remember, slow to apply, and tends to be applied literally in situations its author did not foresee. A rubric with three one-line definitions generalizes better but leaves room for each rater to substitute a personal construct.

A workable compromise is to keep the definitions short and principled and to place the specifics in a separate, growing list of worked boundary examples. Raters learn the principle from the definition and calibrate its edges from the examples. When a new type of difficult case appears, the team adds an example rather than a new rule.

The “Cannot tell” option

Whether to offer an escape option is one of the most consequential small decisions in a rubric. Without one, raters who encounter genuinely broken items are forced to guess, which adds noise. With a loosely defined one, the option absorbs uncertainty: difficult items receive “Cannot tell,” and the rubric never reveals where its boundaries lie.

The distinction matters for later analysis: missing evidence is a property of the item, whereas rater uncertainty is a property of the rater–item pair. Combining the two in one option makes neither measurable.

Version the rubric and extend it from disagreements

A rubric should be managed like code. It should carry a version number, each label should be stored with the version under which it was collected, and a short changelog should record substantive changes (for example, “v1.2: an answer for a different situation now counts as Partially”). Labels collected under different versions are not directly comparable, so the version must be known for every label.

The most productive source of boundary examples is the team’s own disagreements. Each time two raters disagree and an adjudicator resolves the case, the adjudicator should ask whether the disagreement was a rater error or a gap in the rubric. If the rubric was silent, the item becomes a new worked example. After several rounds, the example list becomes a map of the places where the construct is genuinely difficult.

Finally, the rubric should state whether the task is prescriptive (raters apply the rubric’s view) or descriptive (raters report their own view)[2]. The answer-quality task is prescriptive; the pairwise preference task is closer to descriptive. A rubric that does not state which stance applies will receive a mixture of both.

A precise rubric defines how each item should be labeled. It does not determine which items are labeled, and that choice determines what the final number can claim to describe. Sampling is the subject of the next section.

Sample for the decision

Suppose the Mise team draws 1,000 conversations at random from production traffic and asks raters to label them for food safety. Because unsafe responses make up roughly 3% of traffic, the sample will contain about 30 unsafe responses. That is adequate for estimating the overall unsafe rate, but it is far too few to characterize how the assistant fails on safety questions. If the team instead fills the sample with safety questions, it can study failures in detail, but the raw unsafe rate in that sample no longer describes production.

The example illustrates the most common sampling error in evaluation work. It is rarely a flawed technique; more often it is the use of one sample for two purposes that require opposite designs. The two purposes are summarized below.

Both sets are legitimate. They should be kept separate (or the selection probabilities should be recorded so that they can be separated later), and reports should state which set each number comes from.

How many labels are needed to estimate a rate?

For a representative set and a proportion near p, a standard approximation for the half-width of a 95% confidence interval (CI) is:

margin ≈ 1.96 × √ p(1 − p) / n 

Assumes independent items and p not too close to 0 or 1. For rare rates, an exact or Wilson interval is more appropriate.

The key property is the square root: halving the margin requires four times as many labels. The calculator below shows the relationship for different values of n and p.

Rare classes require separate calculations

For the food-safety task, unsafe responses make up about 3% of Mise traffic. When the goal is to measure how well a person or system detects unsafe responses, the relevant quantity is not the total sample size but the number of unsafe items the sample contains. The calculator below gives the expected count and the probability of obtaining at least k unsafe items in a random sample.

Oversample, then reweight

The standard response to rare classes is stratified sampling with oversampling. The population is divided into strata (for example, “safety-related question” and “all other questions”), the rare stratum is sampled at a higher rate, and each stratum’s sampling rate is recorded. When a population rate is reported, each item is weighted by the inverse of its selection probability[3]. The demonstration below shows the error that results when the weights are omitted.

Record-keeping that prevents later problems

• Quotas. Cap the number of items any single day, user, or prompt template can contribute, so that one widely shared recipe does not make up 15% of the sample.
• Deduplication. Near-duplicate prompts (for example, “how long to boil an egg” in 40 phrasings) consume budget without adding information, particularly in diagnostic sets.
• Documentation. Record the random seed, the query that defined the population, and every exclusion with its reason. A datasheet-style record[4] allows others, including the same team months later, to reproduce or reweight the sample.
• Clustering. Five turns from one conversation, or ten questions from one frequent user, are not five or ten independent observations. They share context and style, so the effective sample size is smaller than the row count.

At this point the team knows what to label, how to label it, and which items to include. What remains is the medium through which raters see each item and record a judgment. As the next section shows, that medium affects the labels as much as the rubric does.

The interface is part of the instrument

The annotation interface is often treated as infrastructure that merely records judgments. The evidence suggests otherwise. The question format, the order of the options, the visible context, and the presence of a suggested answer all change the labels collected. In the study cited at the start of this chapter, Kern et al. varied only how an annotation task was presented and observed differences in both the resulting labels and the models trained on them[1]. An interface is therefore part of the measurement instrument and deserves the same design attention as the rubric.

The figure below presents the same Mise item in five question formats. Each mock interface is interactive. Each format makes certain judgments easy to express and others impossible.

Small interface choices with large effects

Rotate option order

Survey research has documented response-order effects for decades: under some conditions, respondents favor options presented first, and the effect is stronger when respondents are fatigued or the task is demanding[5][6]. Annotators labeling item 380 of 400 fit this description closely. For nominal labels, the option in the top position should be randomized or rotated, per item or per rater. For ordered scales, the direction should be reversed for half of the raters rather than shuffled, since a shuffled 1–5 scale is confusing. In either case, the displayed order should be recorded so that an order effect can be tested later.

Randomize sides in pairwise tasks

The new model should never appear consistently on the same side. Sides should be randomized per item, the assignment stored, and the data checked afterwards for whether position A wins more often than expected across all items. This is the same position bias that Section 5.2 tests for in LLM judges; human raters exhibit it as well.

Show sufficient context and keep the evidence visible

Raters can judge only what they can see. If the user’s question appeared in turn 2 and the item displays only turn 5, the answer-quality question cannot be answered. Similarly, if a long response pushes the label controls below the visible area, raters tend to answer before they finish reading. Where possible, the question, the evidence, and the controls should appear in one view, with the relevant part of the input highlighted.

Avoid pre-filled suggestions when building reference data

Displaying a model’s suggested label makes annotation faster, but it also anchors judgments: people adjust insufficiently away from a value or option they have already been shown[7]. The exercise below demonstrates the effect.

Model assistance is not always inappropriate. In high-volume production labeling, a suggestion can be a reasonable tradeoff, provided its effect has been measured. For reference data that will later be used to validate an LLM judge (Section 6.1), however, pre-filling is particularly risky: the judge would be evaluated against labels it helped produce.

Request the correction, not only the verdict

A label of “Partially” indicates that something is wrong but not what should be fixed. A short follow-up field substantially increases the value of each label at little additional cost:

The follow-up should be structured where possible (a list of which question was missed is easier to analyze than free text) and conditional, so that it appears only for the labels that require it.

Each design element discussed so far (brief, rubric, sample, and interface) can be reasoned about in advance, but their interactions usually become visible only when real raters encounter real items. The final section therefore concerns how to test the full design before committing the budget.

Pilot before scaling

Consider a team that launches 3,000 answer-quality labels without a pilot. After two thirds of the items are complete, an analyst notices that several raters have been interpreting “Partially” as “mostly good” rather than “at least one question missed.” The labels collected so far follow two different definitions and cannot be pooled. The team must clarify the rubric, retrain the raters, and relabel the affected items, which costs roughly two thirds of the original labeling budget in addition to the time lost.

A pilot tests the brief, rubric, sample, interface, and instructions together on a small number of items before the main budget is spent. It is among the most frequently skipped steps in annotation projects, usually because of schedule pressure, and it is often the step that would have protected the schedule. The calculator below models the tradeoff in expected cost.

Typical systematic problems include raters reading “Partially” as “mostly good,” a population query that silently excludes non-English traffic, an interface that truncates long responses, and a rubric that does not address messages containing several questions. None of these is rare, and each is inexpensive to detect on 50 items and expensive to detect on 3,000.

A staged pilot

Pilots are most effective when run in stages, each with a different audience that detects a different class of problem. Select a stage below for details.

What to examine in pilot data

Pilot data are too limited to estimate quantities precisely. Its purpose is to reveal symptoms:

• Time per item. A cluster of implausibly fast items often indicates that a rater is not reading the response. A long tail can indicate that the interface hides context or that the rubric is difficult to apply. A mocked distribution is shown below.
• Escape-option rate. If “Cannot tell” exceeds a few percent, either the inputs are broken or the option is being used to express uncertainty.
• Confusion between adjacent labels. In the answer-quality task, most disagreement is between Fully and Partially. These cases are the source of the next boundary examples.
• Raters’ questions. Each question a pilot rater asks corresponds to a sentence missing from the instructions. All such questions should be collected in one place.

Pilot checklist

A completed pilot shows that the design can be executed. It does not yet show that the labels it produces are trustworthy. Chapter 3 addresses that question: how to select and train raters, how to measure agreement beyond chance, how to detect rater bias, and how to combine several labels per item into a reference label.

Summary & quiz

References

1. Kern, C., Eckman, S., Beck, J., Chew, R., Ma, B., & Kreuter, F. (2023). Annotation sensitivity: Training data collection methods affect model performance. Findings of EMNLP 2023.
2. Röttger, P., Vidgen, B., Hovy, D., & Pierrehumbert, J. (2022). Two contrasting data annotation paradigms for subjective NLP tasks. NAACL 2022.
3. Lohr, S. L. (2021). Sampling: Design and Analysis (3rd ed.). CRC Press.
4. Gebru, T., Morgenstern, J., Vecchione, B., et al. (2021). Datasheets for datasets. Communications of the ACM, 64(12).
5. Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2).
6. Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3).
7. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157).