Caption supervision enables robust learners: a controlled study of distributionally robust model training
Lists
Only you see this
Abstract
Vision language models like CLIP are robust to natural distribution shifts, in part because CLIP learns on unstructured data using a technique called caption supervision; the model inteprets image-linked texts as ground-truth labels. In a carefully controlled comparison study, we show that CNNs trained on a standard cross-entropy loss can also benefit from caption supervision, in some cases even more than VL models, on the same data. To facilitate future experiments with high-accuracy caption-supervised models, we introduce CaptionNet, one piece of which is a class-balanced, fully supervised dataset with over 50,000 new human-labeled ImageNet-compliant samples which includes web-scraped captions. In a series of experiments on CaptionNet, we show how the choice of loss function, data filtration and supervision strategy enable robust computer vision.
BibTeX
@inproceedings{feuer2023caption,
author = {Benjamin Feuer and Ameya Joshi and Chinmay Hegde},
title = {{Caption supervision enables robust learners: a controlled study of distributionally robust model training}},
booktitle = {International Conference on Learning Representations},
year = {2023},
url = {https://openreview.net/forum?id=Rkk51I-BpMH},
}