Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement
Lists
Only you see this
Abstract
We propose Dataset Reinforcement, a strategy to improve a dataset once such that the accuracy of any model architecture trained on the reinforced dataset is improved at no additional training cost for users. We propose a Dataset Reinforcement strategy based on data augmentation and knowledge distillation. Our generic strategy is designed based on extensive analysis across CNN- and transformer-based models and performing large-scale study of distillation with state-of-the-art models with various data augmentations. We create a reinforced version of the ImageNet training dataset, called ImageNet+, as well as reinforced datasets CIFAR-100+, Flowers-102+, and Food-101+. Models trained with ImageNet+ are more accurate, robust, and calibrated, and transfer well to downstream tasks (e.g., segmentation and detection). As an example, the accuracy of ResNet-50 improves by 1.7% on the ImageNet validation set, 3.5% on ImageNetV2, and 10.0% on ImageNet-R. Expected Calibration Error (ECE) on the ImageNet validation set is also reduced by 9.9%. Using this backbone with Mask-RCNN for object detection on MS-COCO, the mean average precision improves by 0.8%. We reach similar gains for MobileNets, ViTs, and Swin-Transformers. For MobileNetV3 and Swin-Tiny, we observe significant improvements on ImageNet-R/A/C of up to 20% improved robustness. Models pretrained on ImageNet+ and fine-tuned on CIFAR-100+, Flowers-102+, and Food-101+, reach up to 3.4% improved accuracy. The code, datasets, and pretrained models are available at https://github.com/apple/ml-dr.
BibTeX
@misc{faghri2023reinforce,
author = {Fartash Faghri and Hadi Pouransari and Sachin Mehta and Mehrdad Farajtabar and Ali Farhadi and Mohammad Rastegari and Oncel Tuzel},
title = {{Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement}},
year = {2023},
eprint = {2303.08983},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2303.08983},
}Loading related content...