Consistency Regularization for Certified Robustness of Smoothed Classifiers

Jongheon Jeong, Jinwoo Shin

Abstract

Deep neural networks are susceptible to adversarial attacks, where small input perturbations lead to incorrect predictions. Randomized smoothing has emerged as a leading technique for providing certified robustness guarantees. However, training effective base classifiers for smoothing is often computationally intensive and hyperparameter-sensitive. This paper introduces a simple consistency regularization technique that forces a classifier to produce stable predictions under Gaussian noise. Our method achieves state-of-the-art certified robustness with significantly lower training costs and improved scalability.

Introduction

The fragility of deep learning models to adversarial examples remains a critical challenge for deployment in safety-critical systems. While empirical defenses are often bypassed by adaptive attacks, randomized smoothing offers a mathematically rigorous alternative. It transforms any base classifier into a "smoothed" version that is guaranteed to be robust within a specific $\ell_2$ radius.

The performance of the smoothed classifier is fundamentally tied to the base model's behavior under heavy noise. Existing training paradigms, such as SmoothAdv (adversarial training) and MACER (soft-radius optimization), are effective but suffer from high computational costs and training instability. We propose a more efficient approach centered on the principle of prediction consistency.

Methodology: Consistency Regularization

We hypothesize that a sufficient condition for high certified robustness is prediction consistency. If a model's output for an input $x$ remains invariant regardless of the specific realization of Gaussian noise $\delta \sim \mathcal{N}(0, \sigma^2 I)$, the resulting smoothed classifier will naturally exhibit a large certified radius.

To enforce this, we introduce a consistency regularization term, $L^{con}$, added to the standard cross-entropy loss. This term consists of two primary components:

  1. KL-Divergence Minimization: We minimize the divergence between the model’s average prediction across multiple noisy passes and each individual noisy prediction. This encourages the model to reach a stable consensus.
  2. Entropy Penalty: To prevent the model from achieving consistency through high-entropy (uniform) predictions, we penalize the entropy of the output distribution.
Comparison of log-probability gap distributions across different training methods.

As shown in the histograms above, consistency regularization (green) leads to a much narrower distribution of log-probability gaps compared to standard Gaussian augmentation or even specialized methods like SmoothAdv. This concentration indicates that the model is significantly more stable under noise.

Experimental Results

We evaluated our method on MNIST, CIFAR-10, and ImageNet using various architectures, including ResNet and DenseNet.

Robustness and Accuracy

On CIFAR-10 with $\sigma=0.50$, our method achieved an Average Certified Radius (ACR) of 0.720, surpassing both the Gaussian baseline (0.525) and MACER (0.691). Notably, consistency regularization is complementary to existing techniques. Adding it to SmoothAdv further improved performance, indicating that consistency captures robustness features distinct from those targeted by adversarial training.

Certified accuracy curves for $\sigma = 0.50$ on CIFAR-10.

Efficiency and Scalability

A major advantage of our approach is its computational efficiency. By avoiding the expensive inner loops required for adversarial sample generation, our method is approximately 2.7x to 3x faster than MACER and SmoothAdv. This efficiency allowed us to scale easily to ImageNet, where we improved the ACR from 0.875 to 0.982 at $\sigma=1.0$.

Discussion and Conclusion

While our method significantly reduces training time, inference remains a bottleneck due to the sampling requirements of randomized smoothing. Future work could explore ways to accelerate the certification process itself. Additionally, while we focused on $\ell_2$ robustness via Gaussian noise, the framework is theoretically extensible to other noise distributions for different threat models.

In conclusion, consistency regularization provides a simple, stable, and highly effective path toward building certified robust models, bridging the gap between theoretical guarantees and practical training constraints.

Summary updated

Abstract

A recent technique of randomized smoothing has shown that the worst-case (adversarial) l2-robustness can be transformed into the average-case Gaussian-robustness by "smoothing" a classifier, i.e., by considering the averaged prediction over Gaussian noise. In this paradigm, one should rethink the notion of adversarial robustness in terms of generalization ability of a classifier under noisy observations. We found that the trade-off between accuracy and certified robustness of smoothed classifiers can be greatly controlled by simply regularizing the prediction consistency over noise. This relationship allows us to design a robust training objective without approximating a non-existing smoothed classifier, e.g., via soft smoothing. Our experiments under various deep neural network architectures and datasets show that the "certified" l2-robustness can be dramatically improved with the proposed regularization, even achieving better or comparable results to the state-of-the-art approaches with significantly less training costs and hyperparameters.
BibTeX
@inproceedings{jeong2020consistency,
  author = {Jongheon Jeong and Jinwoo Shin},
  title = {{Consistency Regularization for Certified Robustness of Smoothed Classifiers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=JBitBSqhisU},
}
Figure