What is data annotation? A practical guide for AI teams

By the Annoting team · Updated · 7 min read

Short answer

Data annotation is the process of adding labels to raw data — images, video, text or audio — so a machine-learning model can learn from it. In computer vision that means drawing boxes, outlines, masks or keypoints on objects and tagging each with a class. The labeled data becomes the ground truth the model is trained and evaluated against.

Why models need annotated data

Most production computer-vision models are trained with supervised learning: they are shown thousands of examples with the correct answer attached and adjust until their predictions match. The annotation is that correct answer. A detector learns what a forklift looks like because people drew a box around every forklift in its training set.

That makes annotation quality a ceiling on model quality. A model trained on loose boxes predicts loose boxes; a model trained on a dataset where half the small objects were missed learns to miss small objects. More data rarely fixes inconsistent data.

The main types of image and video annotation

Each annotation type answers a different question, and costs a different amount of human time per image:

TypeWhat it marksTrains
ClassificationOne or more labels for the whole imageImage classifiers
Bounding boxesA rectangle around each objectObject detection (YOLO, DETR)
Oriented boxesA rotated rectangle matching the object's angleAerial and rotated detection
Polygons / instance masksThe exact outline of each objectInstance segmentation
Semantic segmentationA class for every pixelScene segmentation
KeypointsOrdered landmarks such as body jointsPose estimation
Video trackingObjects across frames with persistent IDsMulti-object tracking

How a data annotation project runs

Whether done in-house or by a provider, a well-run project has the same stages:

  • Scoping — decide the classes, the annotation type and what the model actually needs to see.
  • Guidelines — write down the rules, with examples of edge cases: occlusion, tiny objects, ambiguous classes.
  • Calibration — annotators label a pilot batch, disagreements are reviewed and the guideline is tightened.
  • Production labeling — the full dataset is labeled, often with model pre-labels that people correct.
  • Quality assurance — automated checks plus human review; failing work goes back for rework.
  • Delivery — export in the format the training pipeline reads, such as COCO, YOLO or Pascal VOC.

How annotation quality is measured

For shapes, the standard measure is Intersection over Union (IoU): how much an annotator's box or mask overlaps a reviewer-verified one. For classification, it is agreement between annotators or accuracy against hidden gold-standard examples. Missed objects are tracked separately, because an object that was never labeled does not show up in an IoU average.

Ask any provider which of these they measure, on what sample, and whether you see the numbers.

Doing it in-house or outsourcing

Small or highly specialised datasets are often labeled by the team that builds the model. As volume grows, most teams move to a managed annotation provider, which supplies trained annotators, reviewers and tooling and charges per label. The trade-off is control versus throughput; a clear guideline is what makes outsourcing work.

Annoting services mentioned in this guide

FAQ

Related questions

What is data annotation in simple terms?

It is labeling raw data so a computer can learn from it — for example drawing a box around every car in a photo and tagging it "car", so a model learns to find cars.

Is data annotation the same as data labeling?

Yes. The two terms are used interchangeably. Some teams say "labeling" for simple tags and "annotation" for drawn shapes, but there is no strict difference.

What is the most common type of image annotation?

Bounding boxes, because object detection is the most widely deployed vision task and boxes are the fastest spatial label to produce.

How long does it take to annotate a dataset?

It depends on the annotation type and object density. Classification can run to thousands of images per annotator per day; dense segmentation may be a few dozen. A provider can estimate once they see a sample.

Have a dataset to label?

Tell us what you're building. We'll scope the annotation with you and send a plan and a per-label quote.