Introduction
Semantic segmentation is a critical task in computer vision, requiring the assignment of a class label to every pixel in an image. While deep learning models have achieved remarkable success in this area, they are notoriously data-hungry. The primary bottleneck is the high cost of "strong" annotations—pixel-wise masks—which can take hundreds of seconds per image. In contrast, "weak" annotations, such as image-level tags, are significantly cheaper (often by a factor of 100) but provide less granular information.
Researchers facing a fixed budget $X$ must solve the "Budget Allocation Problem": how many expensive masks versus cheap tags should be acquired to maximize model performance? Since the optimal ratio varies by dataset and total budget, arbitrary fixed-ratio strategies (e.g., 80% masks, 20% tags) often lead to suboptimal results. This paper proposes an adaptive, online strategy to dynamically determine the optimal annotation mix during a campaign.
Methodology: Adaptive Budget Allocation
The proposed framework treats the annotation campaign as an iterative optimization problem. Instead of committing the entire budget upfront, the process is divided into several steps.
At each time step $t$, the system models the relationship between the number of weak/strong annotations and the expected model performance (e.g., Dice score) using a Gaussian Process (GP). This "utility surface" is guided by a logarithmic mean prior, reflecting the reality of diminishing returns as data volume increases. By estimating both performance and uncertainty, the system employs Expected Improvement (EI)—a Bayesian optimization technique—to select the next batch of annotations. This allows the algorithm to explore promising annotation ratios and exploit known high-performing combinations, effectively identifying a "Pareto front" of strategies that offer the best accuracy for a given cost.
Experimental Evaluation
The method was evaluated across four diverse datasets representing different challenges:
- Augmented PASCAL VOC 2012: General natural objects.
- Cityscapes: Urban scenes for autonomous driving.
- SUIM: Underwater imagery for marine robotics.
- OCT (Optical Coherence Tomography): Specialized medical imaging requiring high domain expertise.
Results and Discussion
The adaptive strategy consistently achieved performance levels near the "best-possible" fixed strategy, which can typically only be identified through exhaustive, post-hoc experimentation.
Key findings include:
- Dataset Dependency: There is no universal "golden ratio." For instance, the OCT dataset benefited from a high proportion of strong annotations (90%), while the SUIM dataset showed better gains with more weak labels.
- Dynamic Pivoting: The optimal strategy is a "moving target." A ratio that works well for a small budget may become inefficient as the budget grows. The adaptive framework successfully "pivots" its strategy as more data is collected.
- Robustness and Limitations: The system proved robust across different cost ratios and iteration counts, outperforming naive baselines. However, reliance on a logarithmic prior can be a limitation if the true performance surface deviates significantly, as observed in some SUIM experiments where performance dipped slightly at higher budgets.
Conclusion and Future Work
This work introduces a practical, "plug-and-play" tool for researchers to optimize annotation campaigns under budget constraints. By modeling the utility of different annotation types, the system removes the guesswork from budget allocation. Future research will explore integrating more diverse weak supervision types (e.g., bounding boxes, scribbles) and combining this strategy-level optimization with "active learning" to select the most informative individual images for labeling.
Summary updated