Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta

TL;DR

This paper investigates the impact of dramatically increasing dataset size on deep learning performance in computer vision. By leveraging the massive JFT-300M dataset, they demonstrate that performance scales logarithmically with data volume, and that representation learning with larger datasets still holds significant promise, even with noisy labels.

What Problem Does This Paper Address?

The deep learning revolution in computer vision has been largely attributed to three factors: increased model capacity, more computational power (GPUs), and the availability of large labeled datasets. While model architectures and GPU capabilities have advanced significantly since 2012, the size of the largest publicly available datasets (like ImageNet) has remained relatively stagnant. This raises a crucial question: what would happen if we scaled up the training data volume by 10x or even 100x? Would performance see a proportional increase, or would we hit diminishing returns much sooner? The authors aim to shed light on this "mystery" by exploring the relationship between enormous datasets and visual deep learning performance.

Methods

The core of this paper's methodology involves using a massive, noisy dataset (JFT-300M) for representation learning (pre-training) and then evaluating these learned representations on various downstream vision tasks.

  • Representation Learning: They exploit the JFT-300M dataset, which contains over 375 million noisy labels for 300 million images. This data is used to train a standard ResNet-101 model.
  • Downstream Task Evaluation: The pre-trained models are then evaluated on several standard computer vision tasks:
  • Image Classification: Fine-tuning on ImageNet.
  • Object Detection: Using the Faster RCNN framework on COCO and PASCAL VOC datasets.
  • Semantic Segmentation: Using the DeepLab framework on PASCAL VOC.
  • Human Pose Estimation: Using the G-RMI Pose framework.
  • Ablative Studies: The paper includes experiments to analyze the impact of:
  • Dataset size (sampling subsets of JFT-300M).
  • Model capacity (using different ResNet architectures).
  • Number of training epochs.
  • The effect of noisy labels and long-tail distributions.
  • De-duplication of validation sets.

Datasets

  • JFT-300M: The primary dataset used for pre-training. It contains over 300 million images with more than 375 million noisy labels across 18,291 categories. The labels are automatically generated and can be noisy (estimated 20% error).
  • ImageNet (ILSVRC 2012): Used for fine-tuning and as a baseline for image classification.
  • COCO (Common Objects in Context): Used for object detection and human pose estimation evaluation.
  • PASCAL VOC (Visual Object Classes): Used for object detection and semantic segmentation evaluation.
  • FastEval14k: A validation set derived from JFT-300M, used for monitoring training progress.

Key Insights and Ideas

  • Logarithmic Scaling with Data: Performance on vision tasks increases logarithmically with the volume of training data. This means that while gains diminish, they don't plateau immediately, suggesting substantial improvements are still possible with larger datasets.
  • Representation Learning Still Powerful: Even with a massive dataset like JFT-300M, representation learning (pre-training) remains highly beneficial. Training a better base model significantly improves performance on various downstream tasks.
  • Data Scale Can Overpower Noise: The study suggests that the sheer scale of data can compensate for label noise. The performance gains observed with JFT-300M indicate that large, noisy datasets can still yield effective representations.
  • Capacity Matters for Large Data: To fully leverage the benefits of very large datasets, models with higher capacity (e.g., deeper ResNets) are necessary. Smaller models show less improvement when trained on massive datasets compared to larger ones.
  • Long-Tail Data is Manageable: The long-tail distribution of the JFT-300M dataset did not adversely affect the training convergence of ConvNets.

Key Results

  • Image Classification: Fine-tuning a JFT-300M pre-trained ResNet-101 on ImageNet achieved a significant boost in performance compared to training from scratch or using an ImageNet pre-trained model. For example, top-1 accuracy on ImageNet was 79.2% for JFT-300M fine-tuning, compared to 76.4% for MSRA checkpoint and 77.5% for random initialization.
  • Object Detection (COCO):
    • JFT-300M pre-trained models significantly outperformed ImageNet baselines. For instance, on COCO 'test-dev', the 'ImageNet+300M' model achieved 58.0 AP, compared to 53.6 AP for the 'ImageNet' baseline.
  • The performance gain was more pronounced with larger models (e.g., ResNet-152 showed a larger gain than ResNet-50 when using JFT-300M).
  • Performance scaled logarithmically with data size, with gains observed even up to 300 million images.
  • Object Detection (PASCAL VOC): JFT-300M checkpoints provided substantial improvements, with a 5.1% and 5.0% boost in [email protected] for '300M' and 'ImageNet+300M' respectively, compared to the ImageNet baseline.
  • Semantic Segmentation (PASCAL VOC): The JFT-300M checkpoint outperformed the ImageNet checkpoint by 1.7% points in mean IOU.
  • Human Pose Estimation (COCO): JFT-300M initialization led to better performance than ImageNet initialization, with '300M' achieving 64.8 AP and 'ImageNet+300M' achieving 64.4 AP, compared to 62.4 AP for the ImageNet baseline.

Limitations and Future Work

  • Hyperparameter Tuning: The authors acknowledge that the reported performance might underestimate the true impact of data due to the extensive effort required for hyperparameter tuning. Training a JFT model for just 4 epochs took 2 months on 50 GPUs, suggesting that finding optimal schedules for massive datasets is a significant challenge.
  • Dataset Bias: The paper mentions the potential for dataset bias and the need for de-duplication, which they performed for their experiments.
  • Future Work: The authors hope their work inspires the vision community to invest more in building larger datasets and to not undervalue the role of data. They suggest further exploration into the impact of data scale, class number, and label noise on representation learning. They also hint at the possibility of developing better learning schedules for massive datasets.

Summary updated

Abstract

The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10x or 100x? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between `enormous data' and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.
BibTeX
@inproceedings{sun2017revisiting,
  author = {Chen Sun and Abhinav Shrivastava and Saurabh Singh and Abhinav Gupta},
  title = {{Revisiting Unreasonable Effectiveness of Data in Deep Learning Era}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.97},
}
Figure