How to write data annotation guidelines that annotators follow
By the Annoting team · Updated · 6 min read
A good annotation guideline defines every class with examples and counter-examples, gives explicit rules for edge cases (occlusion, truncation, minimum size, reflections, groups), shows correct and incorrect labels visually, and is updated whenever reviewers find a recurring disagreement. Its job is to make two annotators make the same decision on the same image.
Why guidelines decide dataset quality
Most annotation errors are not careless mistakes; they are reasonable people deciding differently because the rule was not written down. Is a person on a poster a person? Does a box include the side mirror? Inconsistent answers teach the model noise. The guideline removes the decision from the annotator's judgement.
What every guideline should contain
- Purpose — one paragraph on what the model will do, so annotators can resolve unforeseen cases sensibly.
- Class list — each class with a definition, 3–5 positive examples and 2–3 look-alike negatives.
- Shape rules — how tight a box is, polygon vertex tolerance, whether occluded parts are included.
- Minimum size — objects smaller than N pixels are skipped or flagged.
- Occlusion and truncation — label the visible part or the full extent, and which flags to set.
- Special cases — reflections, images of objects (posters, screens), groups and crowds, blur.
- Attributes — any extra fields, with allowed values.
- Visual do/don't pairs — a correct and an incorrect label side by side for each tricky rule.
Calibrate before production
Have several annotators label the same 50–100 images, then compare. Every disagreement is either a missing rule or an unclear one. Fix the guideline, repeat until agreement is stable, and only then start the full dataset.
Keep it alive during the project
Reviewers will find cases nobody anticipated. Add them to the guideline with a date and an example, and tell the team. Never change a class definition mid-project without deciding what happens to already-labeled data — otherwise one class quietly becomes two.
Annoting services mentioned in this guide