Efficient Model for Image Classification With Regularization Tricks

Taehyeon Kim, Jonghyup Kim, Seyoung YunNeurIPS 2019 · OpenReview

Deep Neural Networks (DNNs) are notoriously resource-hungry, making them difficult to deploy on edge devices with limited memory and computational power. While many techniques exist to shrink models—such as pruning, quantization, and efficient architecture design—few studies explore how to effectively combine these methods into a single, cohesive pipeline.

Researchers from KAIST tackled this exact problem for the MicroNet Challenge 2019, which tasked participants with designing the most efficient model possible for CIFAR-100 classification while maintaining at least 80% top-1 accuracy. The challenge utilized a strict scoring metric that penalized both the number of parameters and the number of mathematical operations (FLOPS). By combining a custom architecture with novel regularization and pruning tricks, the team achieved a massive 370x improvement in efficiency over standard baselines, winning 2nd and 3rd place in the challenge.

A Four-Stage Pipeline for Maximum Efficiency

To reach their aggressive efficiency targets, the authors employed a sequential, four-stage optimization pipeline:

Four-stage optimization pipeline

1. Architecture Design (C-EfficientNet) The team adapted the well-known EfficientNet architecture for the smaller resolution of CIFAR-100 images, dubbing it "C-EfficientNet." They utilized Mobile Inverted Bottleneck (MBConv) blocks equipped with Squeeze-and-Excitation (SE) modules. To further reduce computational overhead, they replaced the standard Swish activation function with HardSwish.

2. Structural Regularization This stage introduced the core "tricks" that allowed the highly constrained model to generalize effectively:

  • Orthogonality Regularization (ORN): The authors used the Spectral Restricted Isometry Property (SRIP) to penalize weight matrices that are not orthogonal. Crucially, they discovered that applying this regularization only to 1x1 (pointwise) convolutions yielded the best results, suggesting that these bottleneck layers are the most critical points for spectral regularization.
  • Adaptive Label Smoothing (ALS): Standard label smoothing adds uniform noise to labels. ALS, instead, generates a noise vector based on class-wise similarity. By calculating the cosine similarity of the weight vectors in the final classifier layer, the model learns the relationships between similar categories (e.g., different types of trees). This teaches the network that not all "wrong" answers are equally wrong.

3. Data Augmentation Before moving to compression, the team boosted the model's baseline accuracy using state-of-the-art augmentation techniques, specifically FastAutoAugment and CutMix.

4. Sparse Training To hit the final efficiency targets, the network was pruned using a modified version of the Lottery Ticket Hypothesis. The authors found that the standard exponential pruning rate and re-initialization steps actually interfered with the spectral properties learned during the ORN phase. Instead, they used a linear pruning rate for better stability and removed the re-initialization step entirely.

Key Results and Ablation

The final submitted model achieved a remarkable challenge score of 0.0054 (where lower is better), representing a 370x improvement over the baseline. It successfully maintained >80% top-1 accuracy on CIFAR-100 and was the only submission to finish in the top 10% for both parameter storage and computation simultaneously.

Ablation study of ORN and ALS

The ablation study highlights just how impactful the regularization tricks were. As shown in the table above, adding ORN and ALS to the baseline C-EfficientNet-3.0 (expansion ratio 3.0) improved accuracy from 73.02% to 79.31%—a massive jump achieved before any advanced data augmentation or pruning was even applied.

Limitations and Future Work

While this pipeline proved highly successful for the CIFAR-100 task, the authors note that these specific regularization tricks (like ALS and 1x1-specific ORN) have yet to be extensively tested on larger-scale neural network structures or in different domains like NLP. Future work will likely involve scaling these techniques to ImageNet-level models and investigating the theoretical underpinnings of why 1x1 convolutions benefit so disproportionately from orthogonality constraints.

Summary updated

Abstract

In the MicroNet Challenge 2019, competitors attempted to design the neural network architecture with fewer resource budgets, e.g., the number of parameters and FLOPS. In this study, we describe the...
BibTeX
@inproceedings{kim2019efficient,
  author = {Taehyeon Kim and Jonghyup Kim and Seyoung Yun},
  title = {{Efficient Model for Image Classification With Regularization Tricks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=U9cHMK29IY},
}
Figure