How to measure annotation quality: IoU, agreement and gold sets

By the Annoting team · Updated · 5 min read

Short answer

Annotation quality is measured with Intersection over Union (IoU) for boxes and masks — the overlap between a label and a verified reference divided by their combined area — plus inter-annotator agreement for classification, accuracy on hidden gold-standard items, and the missed-object rate. A useful quality report shows all of these per class, not one blended average.

Intersection over Union (IoU)

IoU compares two shapes: the area where they overlap divided by the area they cover together. Identical shapes score 1.0; shapes that do not touch score 0. It is the standard measure for box, polygon and mask quality, comparing an annotator's label with a reviewer-corrected or gold-standard one.

Boxes on large, clear objects routinely reach very high IoU; small objects and fuzzy edges score lower because a few pixels matter more. Look at IoU per class and per object size, not only the overall mean.

Agreement and gold standards

For classification and attributes, measure how often independent annotators agree (raw agreement or Cohen's kappa) and how they score on hidden gold-standard items with known answers. Gold items mixed into normal work keep scores honest because annotators cannot tell which images are being checked.

The metric most reports miss: missed objects

IoU only scores labels that exist. An object nobody labeled never appears in the average, yet a missed object teaches the model that the object is background. Ask for recall against reviewer audits — the share of real objects that received a label.

What to ask an annotation provider

  • Which metrics do you measure, and on what share of the data?
  • Is review done by a different person than the annotator?
  • Do you use hidden gold-standard items?
  • Are metrics reported per class and shared with us?
  • What happens to work that fails the threshold?

Annoting services mentioned in this guide

FAQ

Related questions

What is a good IoU for annotations?

It depends on object size and edge clarity. Agree a threshold per annotation type during scoping, and judge it per class — a single blended number hides weak classes.

How does Annoting measure quality?

Every job passes automated checks and a human reviewer. Hidden ground-truth frames keep annotator scores honest, and clients receive real IoU figures on reviewed work.

What is inter-annotator agreement?

The rate at which independent annotators give the same label to the same item. Low agreement usually points to an unclear guideline rather than careless annotators.

Have a dataset to label?

Tell us what you're building. We'll scope the annotation with you and send a plan and a per-label quote.