Caption supervision enables robust learners: a controlled study of distributionally robust model training

Benjamin Feuer, Ameya Joshi, Chinmay HegdeICLR 2023 · OpenReview

Abstract

Vision language models like CLIP are robust to natural distribution shifts, in part because CLIP learns on unstructured data using a technique called caption supervision; the model inteprets image-linked texts as ground-truth labels. In a carefully controlled comparison study, we show that CNNs trained on a standard cross-entropy loss can also benefit from caption supervision, in some cases even more than VL models, on the same data. To facilitate future experiments with high-accuracy caption-supervised models, we introduce CaptionNet, one piece of which is a class-balanced, fully supervised dataset with over 50,000 new human-labeled ImageNet-compliant samples which includes web-scraped captions. In a series of experiments on CaptionNet, we show how the choice of loss function, data filtration and supervision strategy enable robust computer vision.
BibTeX
@inproceedings{feuer2023caption,
  author = {Benjamin Feuer and Ameya Joshi and Chinmay Hegde},
  title = {{Caption supervision enables robust learners: a controlled study of distributionally robust model training}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Rkk51I-BpMH},
}
Figure