What is data annotation? A practical guide for AI teams
By the Annoting team · Updated · 7 min read
Data annotation is the process of adding labels to raw data — images, video, text or audio — so a machine-learning model can learn from it. In computer vision that means drawing boxes, outlines, masks or keypoints on objects and tagging each with a class. The labeled data becomes the ground truth the model is trained and evaluated against.
Why models need annotated data
Most production computer-vision models are trained with supervised learning: they are shown thousands of examples with the correct answer attached and adjust until their predictions match. The annotation is that correct answer. A detector learns what a forklift looks like because people drew a box around every forklift in its training set.
That makes annotation quality a ceiling on model quality. A model trained on loose boxes predicts loose boxes; a model trained on a dataset where half the small objects were missed learns to miss small objects. More data rarely fixes inconsistent data.
The main types of image and video annotation
Each annotation type answers a different question, and costs a different amount of human time per image:
| Type | What it marks | Trains |
|---|---|---|
| Classification | One or more labels for the whole image | Image classifiers |
| Bounding boxes | A rectangle around each object | Object detection (YOLO, DETR) |
| Oriented boxes | A rotated rectangle matching the object's angle | Aerial and rotated detection |
| Polygons / instance masks | The exact outline of each object | Instance segmentation |
| Semantic segmentation | A class for every pixel | Scene segmentation |
| Keypoints | Ordered landmarks such as body joints | Pose estimation |
| Video tracking | Objects across frames with persistent IDs | Multi-object tracking |
How a data annotation project runs
Whether done in-house or by a provider, a well-run project has the same stages:
- Scoping — decide the classes, the annotation type and what the model actually needs to see.
- Guidelines — write down the rules, with examples of edge cases: occlusion, tiny objects, ambiguous classes.
- Calibration — annotators label a pilot batch, disagreements are reviewed and the guideline is tightened.
- Production labeling — the full dataset is labeled, often with model pre-labels that people correct.
- Quality assurance — automated checks plus human review; failing work goes back for rework.
- Delivery — export in the format the training pipeline reads, such as COCO, YOLO or Pascal VOC.
How annotation quality is measured
For shapes, the standard measure is Intersection over Union (IoU): how much an annotator's box or mask overlaps a reviewer-verified one. For classification, it is agreement between annotators or accuracy against hidden gold-standard examples. Missed objects are tracked separately, because an object that was never labeled does not show up in an IoU average.
Ask any provider which of these they measure, on what sample, and whether you see the numbers.
Doing it in-house or outsourcing
Small or highly specialised datasets are often labeled by the team that builds the model. As volume grows, most teams move to a managed annotation provider, which supplies trained annotators, reviewers and tooling and charges per label. The trade-off is control versus throughput; a clear guideline is what makes outsourcing work.
Annoting services mentioned in this guide