Bounding box vs polygon vs segmentation: which annotation do you need?

By the Annoting team · Updated · 6 min read

Short answer

Use bounding boxes when you need to find and count objects, polygons (instance segmentation) when you need each object's exact shape or area, and semantic segmentation when every pixel of the scene needs a class. Boxes are the fastest and cheapest; segmentation is the most precise and takes several times longer per image.

Side-by-side comparison

Bounding boxPolygon / instanceSemantic segmentation
MarksRectangle per objectOutline per objectClass per pixel
Separates instancesYesYesNo
Captures shapeNoYesYes
Relative effortLowestMedium–highHighest
TrainsObject detectionInstance segmentationSemantic segmentation
Typical useCounting, tracking, presenceMeasurement, grasping, defectsScene parsing, land cover

When bounding boxes are enough

If the downstream question is "is it there, how many, and roughly where", boxes carry all the information the model needs. Counting vehicles, detecting PPE, monitoring shelves and tracking people all run well on box-trained detectors. Boxes are also easiest to review, which keeps quality high at volume.

Boxes struggle with long, thin or diagonal objects — a box around a diagonal pipe is mostly background. For top-down imagery, oriented (rotated) boxes fix most of that at little extra cost.

When you need polygons or masks

Choose instance segmentation when shape or area matters: measuring a defect, estimating fruit size, guiding a robot gripper, or separating overlapping objects precisely. Polygons and masks are two encodings of the same thing; most tools convert between them on export.

A useful trick: masks can always be turned into boxes, but boxes cannot be turned into masks. If you expect to need segmentation later, labeling polygons now avoids relabeling.

When you need semantic segmentation

Semantic segmentation suits classes that are regions rather than countable things — road, sky, water, crop, tissue. It answers "what is everywhere" and gives clean per-class area. It does not separate two touching objects of the same class; if you need both, panoptic segmentation combines semantic and instance labels.

A quick decision guide

  • Only need a yes/no or category for the image → classification.
  • Need to find, count or track objects → bounding boxes.
  • Objects are rotated in top-down imagery → oriented bounding boxes.
  • Need each object's shape, size or a clean cut-out → polygons / instance segmentation.
  • Need every pixel classified → semantic segmentation.
  • Need posture or landmark positions → keypoints.

Annoting services mentioned in this guide

FAQ

Related questions

Is segmentation better than bounding boxes?

It is more precise, not universally better. If your model only needs to locate and count objects, boxes give comparable results at a fraction of the labeling cost.

Can I convert polygons to bounding boxes?

Yes — a box is just the min and max of the polygon's points, so any segmentation dataset can produce a detection dataset. The reverse is not possible without relabeling.

What is panoptic segmentation?

Panoptic segmentation combines semantic and instance segmentation: every pixel gets a class, and countable objects also get individual instance IDs.

Have a dataset to label?

Tell us what you're building. We'll scope the annotation with you and send a plan and a per-label quote.