94% on CIFAR-10 in 3.29 Seconds on a Single GPU

Keller JordanarXiv 2024 · DOI · arXiv

Introduction

In deep learning research, iteration speed is a critical bottleneck. While CIFAR-10 is a relatively small dataset, researchers often require training thousands of models for tasks like hyperparameter optimization, data attribution studies, or analyzing training variance. Standard training recipes, however, are rarely optimized for absolute wall-clock speed on a single GPU. This paper introduces "airbench," a methodology that reaches 94% accuracy on CIFAR-10 in just 3.29 seconds on a single NVIDIA A100 GPU—representing a 1.9× speedup over the previous state-of-the-art.

Methodology

The airbench approach moves beyond simply increasing hardware resources, focusing instead on squeezing maximum algorithmic efficiency from the training process. The methodology incorporates several key innovations:

  • Alternating Flip: Standard training utilizes random horizontal flips, which can lead to redundant data exposure in short training runs. Airbench employs a deterministic alternating flip: after a random first epoch, images flip their state every subsequent epoch. This ensures the model sees both the original and mirrored versions of every image every two epochs, maximizing unique data exposure.
  • Frozen Patch-Whitening: The initial layer is a 2x2 convolution initialized to perform a whitening transformation based on data eigenvectors and then frozen. This technique alone significantly accelerated training.
  • Identity (Dirac) Initialization: Convolutions are initialized as partial identity transforms to facilitate information flow during the early stages of training.
  • Optimization Refinements: The system uses Nesterov SGD with a triangular learning rate schedule, the Lookahead optimizer, and a "scalebias" trick that specifically increases the learning rate for BatchNorm biases.
Impact of individual features on training speed. The chart shows the relative importance of various techniques, with scalebias and dirac initialization providing the largest gains in efficiency.

Results and Scaling

The airbench methodology was evaluated across multiple configurations to demonstrate its scaling properties. The flagship variant, airbench94, achieves 94.01% accuracy in 3.29 seconds when compiled with torch.compile. Higher accuracy targets were also reached efficiently: airbench95 (95.01% in 10.4s) and airbench96 (96.05% in 46.3s).

CIFAR-10 scaling performance. The plot illustrates the trade-off between computational cost (PFLOPs) and test error across the three airbench variants.

Beyond CIFAR-10, the methodology demonstrated strong generalization. The airbench96 configuration outperformed a standard ResNet-18 on CIFAR-100 by 1.7% without additional hyperparameter tuning. Furthermore, the training process proved remarkably stable, with distribution-wise variance 5× lower than standard training recipes.

Discussion and Limitations

While the wall-clock speeds are unprecedented, there are practical considerations for deployment. The use of torch.compile introduces a one-time overhead of several minutes, meaning these speeds are most beneficial when training large batches of models where compilation costs are amortized. Additionally, while the "alternating flip" trick is highly effective for CIFAR, its benefits diminish on larger datasets like ImageNet where existing augmentation diversity (such as heavy random resized cropping) is already high. Finally, while the algorithmic principles are general, specific timings are optimized for the NVIDIA A100 architecture.

Summary updated

Abstract

CIFAR-10 is among the most widely used datasets in machine learning, facilitating thousands of research projects per year. To accelerate research and reduce the cost of experiments, we introduce training methods for CIFAR-10 which reach 94% accuracy in 3.29 seconds, 95% in 10.4 seconds, and 96% in 46.3 seconds, when run on a single NVIDIA A100 GPU. As one factor contributing to these training speeds, we propose a derandomized variant of horizontal flipping augmentation, which we show improves over the standard method in every case where flipping is beneficial over no flipping at all. Our code is released at https://github.com/KellerJordan/cifar10-airbench.
BibTeX
@misc{jordan2024cifar,
  author = {Keller Jordan},
  title = {{94\% on CIFAR-10 in 3.29 Seconds on a Single GPU}},
  year = {2024},
  eprint = {2404.00498},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.00498},
}
Figure