Inter-annotator agreement (IAA) is the measurement of how consistently multiple annotators apply the same labels to the same data, and it has become the default health check for annotation programs: run a sample through two labelers, compute a number, treat a high number as proof that the data is trustworthy.
This blog covers how to design an IAA protocol that does real diagnostic work: which statistic to use for which label type, how to build the calibration process the measurement depends on, how to distinguish noise-like disagreement from pattern-like disagreement, and how the resulting signal should actually change what a program does next. Text annotation services and model evaluation services are the two capabilities most directly involved.
Key Takeaways
- A high agreement score does not guarantee training-worthy data. Whether disagreement behaves like random noise or like a learnable pattern matters more than the raw statistic, and a protocol that only reports the number misses the distinction that actually predicts downstream model quality.
- The right statistic depends on the label type, not convention. Cohen’s kappa suits two annotators on a fixed category set; Krippendorff’s alpha handles more than two annotators, missing labels, and ordinal or interval label types that kappa was never built for.
- Low agreement is a diagnostic, not a verdict. A category that calibrates poorly is usually telling you the guideline is ambiguous or the category boundary is genuinely contested, and the fix is guideline iteration or adjudication (routing disputed items to a senior reviewer for a final call), not simply retraining annotators.
- Agreement should be measured and reported per category, never as one blended score. A single aggregate number hides exactly the categories where disagreement concentrates, which are also usually the categories where the model will struggle most.
- The protocol has to specify what happens after the number, not just how to compute it. Adjudication paths, guideline revision triggers, and distributional labeling (recording the spread of judgments on contested items instead of forcing one answer) all need to be decided in advance, or a low score just sits there without changing anything.
What IAA Actually Measures, and What It Does Not
Raw percent agreement, the fraction of items where annotators matched, is intuitive and misleading, because it does not correct for the agreement you would expect by chance alone. A task with two labels where 90 percent of items obviously belong to one category will show high raw agreement even from annotators guessing. Guessing the majority label is simply right most of the time. Chance-corrected statistics address this by comparing observed agreement to the agreement expected under random labeling. That is why they are the standard for any task with meaningfully skewed label distributions, a description that fits most real annotation programs.
What none of these statistics measure directly is whether the data is good for machine learning, which is a different question than whether humans agree. That is the distinction the Reidsma and Carletta finding turns on. Agreement measures the reliability of the labeling process. A machine learner, though, is specifically vulnerable to a kind of unreliability that agreement statistics do not flag: disagreement that is not random but follows an exploitable pattern, which the model will learn as if it were signal.
Choosing the Right Statistic for the Label Type
Cohen’s Kappa: The Default for Two Annotators, Fixed Categories
Cohen’s kappa fits the common case cleanly: two annotators, a fixed set of categorical labels, no missing data. It is the right default for straightforward classification tasks with two labelers and should be the first thing reached for in that situation. Its limitations show up exactly outside that situation: it was not built for more than two annotators, it does not have a standard extension for missing labels, and it treats all disagreements as equally wrong even when some categories are conceptually closer than others.
Krippendorff’s Alpha: The Generalization That Handles Real Programs
Krippendorff’s alpha was built to handle what kappa was not: any number of annotators, missing labels, and different data types (nominal, ordinal, or interval) through a single flexible framework. Missing labels are the normal case once a program uses distributed annotation pools rather than a fixed pair. Most production annotation programs outgrow Cohen’s kappa’s assumptions quickly, with multiple annotators rotating through a task, incomplete overlap between them, and severity scales that are ordinal rather than purely categorical. That is why alpha is usually the more defensible choice at real operational scale, even though kappa remains more common in casual usage.
Weighted Measures for Ordinal and Graded Labels
Neither statistic in its basic form distinguishes a near-miss from a wildly wrong label. On a severity scale, an annotator who says moderate when the correct label is high should count as less wrong than one who says minimal, and weighted variants of both kappa and alpha exist specifically to encode that: disagreements between adjacent categories are penalized less than disagreements between distant ones. Any protocol using an ordinal or graded label set without a weighting scheme is measuring agreement as if every wrong answer were equally wrong, which understates reliability on exactly the categories where near-miss disagreement is most common and least concerning.
Distinguishing Noise-Like Disagreement From Pattern-Like Disagreement
This is the diagnostic step most protocols skip entirely, and it is the one the research says matters most for downstream model quality. Noise-like disagreement looks random: it does not correlate with any feature of the input, any particular annotator, or any particular time period. A model trained on it treats the disagreement as label noise, which common training procedures tolerate reasonably well at moderate levels, though that tolerance varies by task, model, and noise rate. Pattern-like disagreement correlates with something: a specific annotator systematically applying a stricter standard, a particular input feature that reliably splits annotators, a category boundary that is contested in a consistent, learnable direction. A model trained on pattern-like disagreement learns the pattern. And because the pattern reflects an unresolved ambiguity in the labeling rather than a real signal in the task, the model inherits the ambiguity as if it were ground truth.
The practical test is to break down agreement by annotator, by input feature, and by time period rather than reporting only the aggregate. A category with acceptable overall agreement that splits sharply along one annotator or one input characteristic is exhibiting pattern-like disagreement even though the topline number looks fine, and it is worth the extra analysis pass specifically because the topline number will not surface it.
What the Protocol Should Specify Beyond the Statistic
A complete IAA protocol answers several questions that the raw statistics do not. What triggers adjudication: at what score, or after how many conflicting labels, does a case route to a senior reviewer for a final call rather than getting resolved by majority vote or left ambiguous? What triggers guideline revision: a category with persistently low agreement across calibration rounds is usually telling you the instructions are ambiguous rather than that the annotators need more training, and the fix is rewriting the guideline with worked examples for the boundary that keeps getting crossed. And should contested categories get forced to a single label, or preserved as distributional labels that record the actual spread of judgment? That choice should be made deliberately per category rather than defaulted to consensus everywhere. A protocol silent on all three produces a number without producing a decision.
A Worked Example at the Boundary
Here is what a worked example looks like in practice. In a content moderation taxonomy, the item “you people never listen” kept splitting annotators between Harassment and Not Harassment: one group read “you people” as targeting a protected group, the other as generic frustration. The adjudicator’s ruling was that without surrounding context indicating a protected group, the phrase alone is generic, so the label is Not Harassment. That reasoning went into the guideline as a worked example, paired with a contrast case where “you people” follows an explicit ethnic reference and the label flips to Harassment. Agreement on the category moved from contested to stable in the next calibration round, not because annotators got smarter, but because the boundary finally had an example sitting on it.
How Digital Divide Data Can Help
Everything above is method, and it is portable: a team can implement all of it internally. This section is for readers weighing whether to build that muscle alone or with a partner. Either way, the same components decide whether IAA actually improves model quality: the right statistic for the label type, agreement measured and reported per category, and a defined path from a low score to a guideline fix or an adjudication decision. Producing those is the work we do.
Calibration and measurement. Text annotation teams run structured calibration rounds with the statistic matched to the label type, reporting agreement per category and per annotator so pattern-like disagreement surfaces instead of hiding inside an aggregate.
The evaluation layer that closes the loop. Model evaluation services build the held-out sets and adjudication workflows that turn a low agreement score into a guideline revision or a distributional label decision, not just a flagged number.
If your program can show its per-category agreement numbers, this discipline exists. If it can’t, that’s the starting point. Talk to an expert.
Conclusion
An inter-annotator agreement protocol earns its keep when it changes what a program does, not when it produces a number that clears a threshold. The statistic has to match the label type, the measurement has to be broken out by category rather than blended into one aggregate, and the protocol has to specify in advance what happens when agreement comes in low: adjudication, guideline revision, or a deliberate decision to preserve disagreement as signal rather than force consensus.
The test for any annotation program’s IAA protocol is direct: the last time a category came in below target, what changed as a result? If the honest answer is nothing, the protocol is measuring reliability and doing nothing with what it finds, which means it is not actually protecting model quality. It is producing a number that looks like it is.
References
Artstein, R., & Poesio, M. (2008). Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555–596. https://aclanthology.org/J08-4004/
Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433. https://doi.org/10.1111/j.1468-2958.2004.tb00738.x
Reidsma, D., & Carletta, J. (2008). Reliability measurement without limits. Computational Linguistics, 34(3), 319–326. https://aclanthology.org/J08-3001/
Q1. What counts as a “good” kappa or alpha score? We keep hearing 0.8 as the bar.
Treat 0.8 as a starting heuristic, not a pass-fail line, and the Reidsma and Carletta finding is precisely why: a score at or above the conventional threshold does not guarantee the data is fit for training if the disagreement it contains follows an exploitable pattern rather than looking like noise. A more defensible bar is task-specific and category-specific: categorical, low-ambiguity labels should calibrate high, often above 0.8, while genuinely comparative or judgment-heavy categories can be legitimately useful for training at lower scores, provided the disagreement has been checked for pattern versus noise. A single universal threshold applied to every category in a taxonomy is usually wrong for most of them.
Q2. We have more than two annotators rotating through tasks with incomplete overlap. Which statistic should we use?
Krippendorff’s alpha, and this is close to the textbook case it was designed for. Cohen’s kappa assumes a fixed pair of annotators labeling the same complete set of items, an assumption that breaks the moment annotators rotate and overlap only partially. Alpha handles any number of annotators and tolerates missing data by design, computing agreement from whatever overlapping judgments actually exist rather than requiring a complete matrix. Programs that default to kappa out of familiarity and then struggle to make it work with rotating annotator pools are usually fighting the tool rather than the problem; switching to alpha resolves the mismatch directly.
Q3. A specific clause or content category keeps coming in with low agreement, no matter how much we retrain annotators. What now?
Stop retraining annotators and start rewriting the guideline, because persistently low agreement after repeated training is a strong signal that the ambiguity lives in the instructions, not in annotator skill. Pull the specific disputed cases from that category, have someone with the authority to make a final call adjudicate each one, and write the resulting reasoning into the guideline as a worked example precisely at the boundary that keeps getting crossed. If agreement still does not improve after a guideline revision informed by real disputed cases, consider whether the category is genuinely contested rather than poorly specified, in which case a distributional label that preserves the range of judgment may serve the downstream use case better than forcing an artificial consensus.
Q4. How do we tell if our disagreement is the noise-like kind or the pattern-like kind that the Reidsma and Carletta research warns about?
Break the disagreement down along three axes before concluding anything from the aggregate score: by annotator, to see whether one labeler systematically diverges from the others; by input feature, to see whether disagreement clusters around a particular kind of case rather than spreading evenly; and by time, to see whether agreement drifted as guidelines evolved or as different annotator cohorts rotated through. Noise-like disagreement will not show a clean pattern along any of these axes. Pattern-like disagreement usually will, most often as one annotator applying a consistently different standard or one input characteristic that reliably splits judgment. Finding a pattern is actionable, since it usually points directly at a guideline gap or a training gap; finding none is itself useful information, since it means the topline agreement score is probably a fair description of the data’s reliability.
Q5. Should disagreement ever be kept in the final dataset instead of being resolved to a single label?
For genuinely contested categories, yes, and forcing consensus in those cases can cost you real signal. Some judgments, whether content crosses a subjective severity threshold, whether a clause is market-standard or a negotiated deviation, reflect legitimate variation in expert judgment rather than a resolvable error, and collapsing that variation into one adjudicated label discards information a downstream model or decision system could use, such as calibrating its own confidence to match the level of human disagreement on similar cases. The decision should be made deliberately per category during protocol design, not applied as a blanket policy: categories with a clear correct answer should be adjudicated to one label, while categories with legitimate, persistent expert disagreement are often better served by distributional labels that preserve the spread.
Udit Khanna leads the delivery of scalable AI and data solutions at Digital Divide Data, with a deep specialization in Physical AI. With a background in presales, solutioning, and customer success, he brings a mix of technical depth and business fluency, helping global enterprises move their AI projects from prototype to real-world deployment without losing momentum.
