Full or Weak Annotations? An Adaptive Strategy for Budget-Constrained Annotation Campaigns

Javier Gamazo Tejero, Martin S. Zinkernagel, Sebastian Wolf, Raphael Sznitman, Pablo Márquez-Neila

Introduction

Semantic segmentation is a critical task in computer vision, requiring the assignment of a class label to every pixel in an image. While deep learning models have achieved remarkable success in this area, they are notoriously data-hungry. The primary bottleneck is the high cost of "strong" annotations—pixel-wise masks—which can take hundreds of seconds per image. In contrast, "weak" annotations, such as image-level tags, are significantly cheaper (often by a factor of 100) but provide less granular information.

Researchers facing a fixed budget $X$ must solve the "Budget Allocation Problem": how many expensive masks versus cheap tags should be acquired to maximize model performance? Since the optimal ratio varies by dataset and total budget, arbitrary fixed-ratio strategies (e.g., 80% masks, 20% tags) often lead to suboptimal results. This paper proposes an adaptive, online strategy to dynamically determine the optimal annotation mix during a campaign.

Methodology: Adaptive Budget Allocation

The proposed framework treats the annotation campaign as an iterative optimization problem. Instead of committing the entire budget upfront, the process is divided into several steps.

Iterative optimization framework using Gaussian Processes to model the utility surface and allocate budget between weak and strong annotations.

At each time step $t$, the system models the relationship between the number of weak/strong annotations and the expected model performance (e.g., Dice score) using a Gaussian Process (GP). This "utility surface" is guided by a logarithmic mean prior, reflecting the reality of diminishing returns as data volume increases. By estimating both performance and uncertainty, the system employs Expected Improvement (EI)—a Bayesian optimization technique—to select the next batch of annotations. This allows the algorithm to explore promising annotation ratios and exploit known high-performing combinations, effectively identifying a "Pareto front" of strategies that offer the best accuracy for a given cost.

Experimental Evaluation

The method was evaluated across four diverse datasets representing different challenges:

  • Augmented PASCAL VOC 2012: General natural objects.
  • Cityscapes: Urban scenes for autonomous driving.
  • SUIM: Underwater imagery for marine robotics.
  • OCT (Optical Coherence Tomography): Specialized medical imaging requiring high domain expertise.

Results and Discussion

The adaptive strategy consistently achieved performance levels near the "best-possible" fixed strategy, which can typically only be identified through exhaustive, post-hoc experimentation.

Performance comparison of the proposed adaptive strategy (Ours) against various fixed-ratio baselines across four datasets.

Key findings include:

  1. Dataset Dependency: There is no universal "golden ratio." For instance, the OCT dataset benefited from a high proportion of strong annotations (90%), while the SUIM dataset showed better gains with more weak labels.
  2. Dynamic Pivoting: The optimal strategy is a "moving target." A ratio that works well for a small budget may become inefficient as the budget grows. The adaptive framework successfully "pivots" its strategy as more data is collected.
  3. Robustness and Limitations: The system proved robust across different cost ratios and iteration counts, outperforming naive baselines. However, reliance on a logarithmic prior can be a limitation if the true performance surface deviates significantly, as observed in some SUIM experiments where performance dipped slightly at higher budgets.

Conclusion and Future Work

This work introduces a practical, "plug-and-play" tool for researchers to optimize annotation campaigns under budget constraints. By modeling the utility of different annotation types, the system removes the guesswork from budget allocation. Future research will explore integrating more diverse weak supervision types (e.g., bounding boxes, scribbles) and combining this strategy-level optimization with "active learning" to select the most informative individual images for labeling.

Summary updated

Abstract

Annotating new datasets for machine learning tasks is tedious, time-consuming, and costly. For segmentation applications, the burden is particularly high as manual delineations of relevant image content are often extremely expensive or can only be done by experts with domain-specific knowledge. Thanks to developments in transfer learning and training with weak supervision, segmentation models can now also greatly benefit from annotations of different kinds. However, for any new domain application looking to use weak supervision, the dataset builder still needs to define a strategy to distribute full segmentation and other weak annotations. Doing so is challenging, however, as it is a priori unknown how to distribute an annotation budget for a given new dataset. To this end, we propose a novel approach to determine annotation strategies for segmentation datasets, whereby estimating what proportion of segmentation and classification annotations should be collected given a fixed budget. To do so, our method sequentially determines proportions of segmentation and classification annotations to collect for budget-fractions by modeling the expected improvement of the final segmentation model. We show in our experiments that our approach yields annotations that perform very close to the optimal for a number of different annotation budgets and datasets.
BibTeX
@inproceedings{tejero2023full,
  author = {Javier Gamazo Tejero and Martin S. Zinkernagel and Sebastian Wolf and Raphael Sznitman and Pablo Márquez-Neila},
  title = {{Full or Weak Annotations? An Adaptive Strategy for Budget-Constrained Annotation Campaigns}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01095},
}
Figure