How to measure annotation quality: IoU, agreement and gold sets
By the Annoting team · Updated · 5 min read
Annotation quality is measured with Intersection over Union (IoU) for boxes and masks — the overlap between a label and a verified reference divided by their combined area — plus inter-annotator agreement for classification, accuracy on hidden gold-standard items, and the missed-object rate. A useful quality report shows all of these per class, not one blended average.
Intersection over Union (IoU)
IoU compares two shapes: the area where they overlap divided by the area they cover together. Identical shapes score 1.0; shapes that do not touch score 0. It is the standard measure for box, polygon and mask quality, comparing an annotator's label with a reviewer-corrected or gold-standard one.
Boxes on large, clear objects routinely reach very high IoU; small objects and fuzzy edges score lower because a few pixels matter more. Look at IoU per class and per object size, not only the overall mean.
Agreement and gold standards
For classification and attributes, measure how often independent annotators agree (raw agreement or Cohen's kappa) and how they score on hidden gold-standard items with known answers. Gold items mixed into normal work keep scores honest because annotators cannot tell which images are being checked.
The metric most reports miss: missed objects
IoU only scores labels that exist. An object nobody labeled never appears in the average, yet a missed object teaches the model that the object is background. Ask for recall against reviewer audits — the share of real objects that received a label.
What to ask an annotation provider
- Which metrics do you measure, and on what share of the data?
- Is review done by a different person than the annotator?
- Do you use hidden gold-standard items?
- Are metrics reported per class and shared with us?
- What happens to work that fails the threshold?
Annoting services mentioned in this guide