How to write data annotation guidelines that annotators follow

By the Annoting team · Updated · 6 min read

Short answer

A good annotation guideline defines every class with examples and counter-examples, gives explicit rules for edge cases (occlusion, truncation, minimum size, reflections, groups), shows correct and incorrect labels visually, and is updated whenever reviewers find a recurring disagreement. Its job is to make two annotators make the same decision on the same image.

Why guidelines decide dataset quality

Most annotation errors are not careless mistakes; they are reasonable people deciding differently because the rule was not written down. Is a person on a poster a person? Does a box include the side mirror? Inconsistent answers teach the model noise. The guideline removes the decision from the annotator's judgement.

What every guideline should contain

  • Purpose — one paragraph on what the model will do, so annotators can resolve unforeseen cases sensibly.
  • Class list — each class with a definition, 3–5 positive examples and 2–3 look-alike negatives.
  • Shape rules — how tight a box is, polygon vertex tolerance, whether occluded parts are included.
  • Minimum size — objects smaller than N pixels are skipped or flagged.
  • Occlusion and truncation — label the visible part or the full extent, and which flags to set.
  • Special cases — reflections, images of objects (posters, screens), groups and crowds, blur.
  • Attributes — any extra fields, with allowed values.
  • Visual do/don't pairs — a correct and an incorrect label side by side for each tricky rule.

Calibrate before production

Have several annotators label the same 50–100 images, then compare. Every disagreement is either a missing rule or an unclear one. Fix the guideline, repeat until agreement is stable, and only then start the full dataset.

Keep it alive during the project

Reviewers will find cases nobody anticipated. Add them to the guideline with a date and an example, and tell the team. Never change a class definition mid-project without deciding what happens to already-labeled data — otherwise one class quietly becomes two.

Annoting services mentioned in this guide

FAQ

Related questions

How long should an annotation guideline be?

As long as the edge cases require. Simple classification can fit on a page; a detailed segmentation project can run to 20 or more pages, mostly visual examples.

Who should write the annotation guidelines?

The ML team that owns the model, with the annotation lead. The ML team knows what the model needs; the annotation lead knows where annotators will disagree.

Does Annoting help write guidelines?

Yes. Scoping a project with us includes agreeing classes, edge cases and annotation rules in writing before labeling starts, then updating them as review finds new cases.

Have a dataset to label?

Tell us what you're building. We'll scope the annotation with you and send a plan and a per-label quote.