CINIC-10 is not ImageNet or CIFAR-10

Luke Nicholas Darlow, Elliot J. Crowley, Antreas Antoniou, Amos J. StorkeyarXiv 2018 · OpenReview

Introduction

In the landscape of computer vision research, datasets like CIFAR-10 and ImageNet represent two ends of a spectrum. CIFAR-10, with its 60,000 low-resolution images, is ideal for rapid prototyping but is increasingly considered "solved." Conversely, ImageNet offers over a million high-resolution images, providing a rigorous benchmark that requires substantial computational resources. CINIC-10 is introduced as a "Goldilocks" dataset, bridging this gap by augmenting CIFAR-10 with 210,000 downsampled images from ImageNet, resulting in a benchmark that is 4.5 times larger and significantly more challenging while remaining computationally accessible.

Methodology

The construction of CINIC-10 involved a systematic three-stage process. First, the original CIFAR-10 data was reformatted into a standard directory structure compatible with modern deep learning frameworks. Second, the authors mapped synonym sets (synsets) from the ImageNet database to the ten existing CIFAR-10 classes. Finally, relevant ImageNet images were downsampled to 32x32 pixels using a "Box" algorithm to maintain consistency with CIFAR-10's resolution.

Sample images from the 'cat' class in the CINIC-10 dataset, illustrating the diversity and noise inherent in downsampled ImageNet data.

This process yielded a total of 270,000 images, which were partitioned into three equal splits of 90,000 images each for training, validation, and testing. This dedicated validation set addresses a common shortcoming in CIFAR-10 research, where validation sets are often inconsistently derived from training data.

Dataset Characteristics and Distribution Shift

A key feature of CINIC-10 is the introduction of distribution shift. Because the dataset draws from two distinct sources—the original CIFAR-10 and the Fall 2011 release of ImageNet—models must generalize across varying data distributions. Analysis of the color intensity distributions reveals subtle but measurable differences between the two source components.

Comparison of color intensity distributions between CIFAR-10 and ImageNet contributors within CINIC-10.

Furthermore, the authors note that downsampling ImageNet images to 32x32 resolution reduces their information capacity, effectively making the classification task harder than the original high-resolution ImageNet task. This forces models to learn from noisier and less distinct features.

Benchmarking Results

To establish a baseline, several popular neural network architectures were evaluated on CINIC-10. The results consistently showed higher error rates compared to standard CIFAR-10 benchmarks, confirming the increased difficulty of the new dataset.

Model Parameters Test Error (%)
ResNeXt29_2x64d 9.2M 8.55 ± 0.15
DenseNet-121 7.0M 8.74 ± 0.16
GoogLeNet 6.2M 8.83 ± 0.12
ResNet-18 11.2M 9.73 ± 0.05
VGG-16 14.7M 12.23 ± 0.16
MobileNet 3.2M 18.00 ± 0.16

While sophisticated architectures like ResNeXt and DenseNet performed well, lightweight models such as MobileNet struggled significantly, likely due to the increased noise and lower resolution of the augmented data.

Conclusion

CINIC-10 provides a necessary intermediate benchmark for the computer vision community. By combining the scale of ImageNet with the accessibility of CIFAR-10, it offers a more rigorous testing ground for neural architecture search and studies on model generalization. While the dataset contains some inherent noise due to its diverse source material, it serves as a valuable tool for developing more robust and efficient machine learning models.

Summary updated

Abstract

In this brief technical report we introduce the CINIC-10 dataset as a plug-in extended alternative for CIFAR-10. It was compiled by combining CIFAR-10 with images selected and downsampled from the ImageNet database. We present the approach to compiling the dataset, illustrate the example images for different classes, give pixel distributions for each part of the repository, and give some standard benchmarks for well known models. Details for download, usage, and compilation can be found in the associated github repository.
BibTeX
@misc{darlow2018cinic,
  author = {Luke Nicholas Darlow and Elliot J. Crowley and Antreas Antoniou and Amos J. Storkey},
  title = {{CINIC-10 is not ImageNet or CIFAR-10}},
  year = {2018},
  eprint = {1810.03505},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.03505},
}
Figure