Data annotation: labels, disagreement and review evidence

Data annotation adds labels or other task-specific information to examples so people or software can use them for training, evaluation or analysis. A label might identify a message’s requested action, an entity’s text span or an object’s image region. The annotation is a decision under a written policy; it is not automatically a verified real-world outcome.

Useful annotation keeps the input, label rule, reviewer decision and uncertainty together. Disagreement can reveal a vague instruction rather than a careless reviewer. This page works through eight invented messages before any model prediction or evaluation score is involved.

Labels, unlabelled inputs and outcomes

An unlabelled input contains the example without the requested reference annotation. A labelled input pairs that example with an annotation defined by the task. The same sentence can receive a topic label in one dataset and a requested-action label in another; neither label is meaningful without its scheme.

An outcome is something that happened outside the annotation process. A message asking for a delivery-date change does not prove that the date changed. Keep observed outcomes in a different field with their own evidence. Otherwise a training or evaluation dataset can confuse what a person requested with what the organisation actually did.

Reference labels are sometimes called gold labels. That name describes their role in comparison, not an assurance that they contain no mistakes. Before treating them as an answer key, check the policy, the source version and any unresolved disagreements.

A compact labelbook for requested actions

Synthetic annotation exercise. The messages and both reviewers below are fictional. Annotator A and Annotator B work independently using labelbook actions_v1. Its unit is one complete message, and it permits one label per message. Neither reviewer sees the other’s first decision.

The labelbook needs examples and exclusions because ordinary words can describe different operations. “Stop” might concern a booking, a reminder or a recurring message. Treating every occurrence as cancellation would create apparent consistency while applying the wrong meaning.

Eight messages, two independent annotation passes

The first pass has six agreements out of eight messages. This is a count derived from an authored table, not an annotation-service result or model benchmark. It establishes neither correctness nor chance-adjusted agreement. Six matching mistakes would still be six agreements.

A04 and A06 matter because both mention a change-related word without requesting that action. Their agreed information labels follow the task’s requested-action rule. A simple keyword policy would miss that distinction even though the annotation table itself remains easy to read.

A disagreement ledger and adjudication rule

Adjudication checks the original message against the labelbook, records why the first passes differed and retains an unresolved state where the source is insufficient. For this exercise, a third fictional reviewer applies the existing exclusions; majority voting is unnecessary because the policy already resolves the relevant boundary.

Store message ID, exact text revision, labelbook version, both first labels, final label and decision reason. Do not overwrite A’s initial cancel labels with review and then report perfect initial agreement. If a later labelbook adds a notification-preference category, create a new annotation revision rather than silently redefining the older labels.

Related: Computer vision: distinguish image evidence, recognition and inference.

How does data annotation differ from prediction and evaluation?

Text classification predicts or assigns task labels to text. Annotation prepares the reference decisions against which such outputs may be compared. Keep predicted_label distinct from reference_label so a model suggestion cannot quietly become its own answer key.

AI agent evaluation checks task behaviour, while RAG evaluation also needs source-evidence labels. Those pages cover scoring and workflow observations. The annotation question comes earlier: can a reviewer justify the expected result from this input under this rule?

AI Flowi Workflow capability boundary

For proposal-generation interfaces, consult Structured Output in the catalogue and the Language Model reference; their status is documented in the catalogue; not production-tested by Ai Flowi.

Those references do not establish a native annotation workspace or adjudication mechanism. If a model suggests labels for actions_v1, hide those suggestions during the independent human passes. Otherwise A and B could copy the same proposal, making their apparent agreement evidence of a shared suggestion rather than independent interpretation. Provider assistance requires an authorised account and any necessary API key, with the provider’s usage billed separately.

Before revising the labelbook, check which missing distinction caused A08. Adding a reminder-preference category could resolve that message while leaving A07 ambiguous. Write an inclusion example for stopping reminders and an exclusion for stopping the booking. Then re-annotate affected messages under actions_v2, keeping the older results. The policy revision repairs a category gap; it cannot supply the absent referent in “leave it.”

Common mistakes and related concepts

Do not hide disagreement, treat review as a confirmed action or use later service outcomes to rewrite what the original message asked. For a small collection, independent manual passes and a written ledger can cover the annotation task. No training implementation or paid annotation service is asserted here.

Synthetic data explains invented inputs, NLP distinguishes language tasks, and machine learning pipelines explain where reference labels enter a learning lifecycle.

By Ai Flowi · Published 10 October 2026