Comparing the Effects of Annotation Type on Machine Learning Detection Performance

James F. Mullen Jr., Franklin R. Tanner, Phil A. SalleeWorkshops @ CVPR 2019 · DOI

Introduction

In the field of computer vision, data annotation is often the most significant bottleneck in the development pipeline. Manually clicking around the edges of objects in satellite imagery to create perfect polygon masks is a grueling and expensive process. This research investigates a critical question: is this level of precision actually necessary for high-performing models?

A research team from Raytheon addressed this in their study, comparing high-fidelity polygon annotations against cheaper alternatives like bounding boxes and simple point centroids. Their findings suggest that for many object detection tasks, we may be overpaying for precision that neural networks do not strictly require.

Annotation Types and the Cost-Benefit Tradeoff

Supervised machine learning requires labeled data, typically categorized into three levels of fidelity:

  1. Polygons: Highly accurate but slow to produce (requiring many clicks).
  2. Bounding Boxes: Faster (two clicks) but include "noise" pixels that are not part of the object.
  3. Centroids (Points): The fastest (one click) but provide the least spatial information.
Visual comparison of polygon, bounding box, and point annotations on a vehicle target.

The study identifies a lack of quantitative data regarding the cost-benefit tradeoff between these methods. While researchers often default to polygons, the time and financial investment required can be up to six times higher than simpler methods.

Methodology

The researchers used the Overhead Imagery Research Dataset (OIRDS), which contains approximately 1,000 images and 2,000 vehicle targets. They modified the Overfeat network architecture to perform pixel-wise classification (segmenting "car" vs. "non-car").

To ensure a fair comparison, they handled low-fidelity labels strategically. Bounding boxes were defined as the tightest box around the object. For centroids, they extrapolated a circular "point-target" region around each point, sized to match the area of an average target. Three identical versions of the network were then trained, varying only the type of ground-truth mask used.

Key Results and Performance

The models were evaluated using the Area under the Curve (AuC) of the Receiver Operating Characteristic (ROC). To determine true utility, all three models were evaluated against the "gold standard" polygon truth.

ROC curves comparing the performance of models trained on different annotation types against the polygon ground truth.

Quantitatively, the performance gap was remarkably slim:

  • Polygon-trained model: 0.948 AuC
  • Bounding Box-trained model: 0.943 AuC
  • Point Target-trained model: 0.941 AuC

The difference between the most expensive method (polygons) and the cheapest (centroids) was only 0.007 AuC. Qualitatively, the networks trained on boxes or circles did not just output those shapes; they learned to "paint" the actual shape of the vehicles, effectively ignoring the label noise.

Discussion and Insights

The study reinforces the idea that deep neural networks are inherently robust to label noise. Based on commercial annotation rates, polygons take roughly six times longer to produce than centroids. The paper argues that for many applications, a negligible drop in accuracy is a fair trade for an 85% reduction in annotation costs.

Limitations

The authors note that these results are based on the OIRDS dataset, which focuses on relatively uniform objects (cars) in nadir (top-down) imagery. More complex, amorphous shapes or oblique imagery—where bounding boxes include significantly more background noise—might still benefit from the precision of polygons.

Future Direction: The researchers suggest a "hybrid" approach: starting with cheap centroid annotations to train a base model, then selectively adding high-fidelity polygons only where the model demonstrates difficulty.

Summary updated

BibTeX
@inproceedings{jr2019comparing,
  author = {James F. Mullen Jr. and Franklin R. Tanner and Phil A. Sallee},
  title = {{Comparing the Effects of Annotation Type on Machine Learning Detection Performance}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00114},
}
Figure