Abstract
In this paper, we identify an important reproducibility challenge in the image-to-set prediction literature that impedes proper comparisons among published methods, namely, researchers use different evaluation protocols to assess their contributions. To alleviate this issue, we introduce an image-to-set prediction benchmark suite built on top of five public datasets of increasing task complexity that are suitable for multi-label classification (VOC, COCO, NUS-WIDE, ADE20k and Recipe1M). Using the benchmark, we provide an in-depth analysis where we study the key components of current models, namely the choice of the image representation backbone as well as the set predictor design. Our results show that (1) exploiting better image representation backbones leads to higher performance boosts than enhancing set predictors, and (2) modeling both the label co-occurrences and ordering has a slight positive impact in terms of performance, whereas explicit cardinality prediction only helps when training on complex datasets, such as Recipe1M.
Introduction
Image-to-set prediction, commonly known as multi-label classification, involves identifying all relevant objects or concepts within a single image. Unlike standard single-label classification, these models must output an unordered set of labels of varying sizes. Despite its ubiquity, the field faces a reproducibility crisis. The lack of standardized benchmarks—including inconsistent dataset splits, evaluation metrics, and pre-trained backbones—makes it difficult to verify if new methodologies represent genuine progress or are merely artifacts of better tuning or stronger encoders.
Methodology
To address these challenges, the authors propose a systematic analysis that decouples the two main components of image-to-set models: the image representation backbone and the set predictor.
The study evaluates a diverse range of architectures:
- Backbones: Standard encoders such as ResNet-50, ResNet-101, and ResNeXt-101-32x8d.
- Set Predictors: Ranging from simple Feed-forward (FF) networks using various loss functions (Binary Cross-Entropy, soft IoU, Target Distribution) to complex auto-regressive models like LSTMs and Transformers.
A key contribution of this work is the use of HYPERBAND, an automated bandit-based algorithm, to ensure that every model is compared under an equivalent hyperparameter tuning budget, eliminating the "tuning bias" often found in literature.
Datasets
The benchmark suite utilizes five public datasets of increasing complexity to test the limits of these models:
- Pascal VOC 2007 & MS COCO: Standard benchmarks with relatively small label dictionaries (<100 classes).
- NUS-WIDE: Focuses on more abstract concepts.
- ADE20k: A scene parsing dataset with 150 classes.
- Recipe1M: The most challenging task, requiring the prediction of ingredients from food images with a massive dictionary of 1,486 classes, many of which are visually subtle or invisible.
Key Insights and Results
The extensive benchmarking revealed several critical findings:
The Backbone is King
The most significant performance improvements consistently came from upgrading the image representation backbone rather than refining the set-prediction logic. For instance, moving from a ResNet-50 to a ResNeXt-101-32x8d backbone typically yielded a larger Overall F1 (O-F1) boost than switching between different set predictor architectures.
Simplicity and Tuning
A simple feed-forward baseline trained with Binary Cross-Entropy (FF-BCE) proved remarkably robust. When properly tuned, it often matched or outperformed much more complex models on standard datasets. This suggests that many "state-of-the-art" claims in previous work may have been driven by hyperparameter optimization rather than architectural innovation.
Complexity and Cardinality
Explicitly predicting the number of labels (cardinality) only provided a clear advantage on highly complex datasets like Recipe1M, where the number of items per image varies significantly. Additionally, while sets are unordered, auto-regressive models (LSTMs/Transformers) can exploit implicit label ordering to improve results in specific contexts like ADE20k.
Conclusion
This research clarifies the current landscape of image-to-set prediction by providing a rigorous, unified benchmark. The findings suggest that researchers should shift their focus toward improving image representations specifically tailored for multi-label scenarios. To facilitate future work, the authors have released their code, best-performing models, and standardized dataset splits.
Summary updated