Elucidating image-to-set prediction: An analysis of models, losses and datasets.

Luis Pineda, Amaia Salvador, Michal Drozdzal, Adriana RomeroarXiv 2019 · OpenReview

Abstract

In this paper, we identify an important reproducibility challenge in the image-to-set prediction literature that impedes proper comparisons among published methods, namely, researchers use different evaluation protocols to assess their contributions. To alleviate this issue, we introduce an image-to-set prediction benchmark suite built on top of five public datasets of increasing task complexity that are suitable for multi-label classification (VOC, COCO, NUS-WIDE, ADE20k and Recipe1M). Using the benchmark, we provide an in-depth analysis where we study the key components of current models, namely the choice of the image representation backbone as well as the set predictor design. Our results show that (1) exploiting better image representation backbones leads to higher performance boosts than enhancing set predictors, and (2) modeling both the label co-occurrences and ordering has a slight positive impact in terms of performance, whereas explicit cardinality prediction only helps when training on complex datasets, such as Recipe1M.

Introduction

Image-to-set prediction, commonly known as multi-label classification, involves identifying all relevant objects or concepts within a single image. Unlike standard single-label classification, these models must output an unordered set of labels of varying sizes. Despite its ubiquity, the field faces a reproducibility crisis. The lack of standardized benchmarks—including inconsistent dataset splits, evaluation metrics, and pre-trained backbones—makes it difficult to verify if new methodologies represent genuine progress or are merely artifacts of better tuning or stronger encoders.

Methodology

To address these challenges, the authors propose a systematic analysis that decouples the two main components of image-to-set models: the image representation backbone and the set predictor.

The study evaluates a diverse range of architectures:

  • Backbones: Standard encoders such as ResNet-50, ResNet-101, and ResNeXt-101-32x8d.
  • Set Predictors: Ranging from simple Feed-forward (FF) networks using various loss functions (Binary Cross-Entropy, soft IoU, Target Distribution) to complex auto-regressive models like LSTMs and Transformers.
Overview of the evaluated model properties and architectures, including Feed-forward (b), LSTM (c), and Transformer (d) designs.

A key contribution of this work is the use of HYPERBAND, an automated bandit-based algorithm, to ensure that every model is compared under an equivalent hyperparameter tuning budget, eliminating the "tuning bias" often found in literature.

Datasets

The benchmark suite utilizes five public datasets of increasing complexity to test the limits of these models:

  • Pascal VOC 2007 & MS COCO: Standard benchmarks with relatively small label dictionaries (<100 classes).
  • NUS-WIDE: Focuses on more abstract concepts.
  • ADE20k: A scene parsing dataset with 150 classes.
  • Recipe1M: The most challenging task, requiring the prediction of ingredients from food images with a massive dictionary of 1,486 classes, many of which are visually subtle or invisible.

Key Insights and Results

The extensive benchmarking revealed several critical findings:

The Backbone is King

The most significant performance improvements consistently came from upgrading the image representation backbone rather than refining the set-prediction logic. For instance, moving from a ResNet-50 to a ResNeXt-101-32x8d backbone typically yielded a larger Overall F1 (O-F1) boost than switching between different set predictor architectures.

Comparison of O-F1 improvement when upgrading the set predictor versus upgrading the image backbone across the five benchmark datasets.

Simplicity and Tuning

A simple feed-forward baseline trained with Binary Cross-Entropy (FF-BCE) proved remarkably robust. When properly tuned, it often matched or outperformed much more complex models on standard datasets. This suggests that many "state-of-the-art" claims in previous work may have been driven by hyperparameter optimization rather than architectural innovation.

Complexity and Cardinality

Explicitly predicting the number of labels (cardinality) only provided a clear advantage on highly complex datasets like Recipe1M, where the number of items per image varies significantly. Additionally, while sets are unordered, auto-regressive models (LSTMs/Transformers) can exploit implicit label ordering to improve results in specific contexts like ADE20k.

Conclusion

This research clarifies the current landscape of image-to-set prediction by providing a rigorous, unified benchmark. The findings suggest that researchers should shift their focus toward improving image representations specifically tailored for multi-label scenarios. To facilitate future work, the authors have released their code, best-performing models, and standardized dataset splits.

Summary updated

Abstract

In this paper, we identify an important reproducibility challenge in the image-to-set prediction literature that impedes proper comparisons among published methods, namely, researchers use different evaluation protocols to assess their contributions. To alleviate this issue, we introduce an image-to-set prediction benchmark suite built on top of five public datasets of increasing task complexity that are suitable for multi-label classification (VOC, COCO, NUS-WIDE, ADE20k and Recipe1M). Using the benchmark, we provide an in-depth analysis where we study the key components of current models, namely the choice of the image representation backbone as well as the set predictor design. Our results show that (1) exploiting better image representation backbones leads to higher performance boosts than enhancing set predictors, and (2) modeling both the label co-occurrences and ordering has a slight positive impact in terms of performance, whereas explicit cardinality prediction only helps when training on complex datasets, such as Recipe1M. To facilitate future image-to-set prediction research, we make the code, best models and dataset splits publicly available at: https://github.com/facebookresearch/image-to-set.
BibTeX
@misc{pineda2019elucidating,
  author = {Luis Pineda and Amaia Salvador and Michal Drozdzal and Adriana Romero},
  title = {{Elucidating image-to-set prediction: An analysis of models, losses and datasets.}},
  year = {2019},
  eprint = {1904.05709},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.05709},
}
Figure