Guardian of the Ensembles: Introducing Pairwise Adversarially Robust Loss for Resisting Adversarial Attacks in DNN Ensembles

Shubhi Shukla, Subhadeep Dalui, Manaar Alam, Shubhajit Datta, Arijit Mondal, Debdeep Mukhopadhyay, Partha Pratim ChakrabartiWACV 2025

TL;DR

This paper introduces Pairwise Adversarially Robust Loss (PARL), a novel training method for ensemble deep learning models that significantly enhances their resilience to adversarial attacks. By enforcing diversity in the decision boundaries and internal representations of ensemble members, PARL achieves state-of-the-art robustness with minimal impact on clean accuracy and reduced training time compared to existing methods.

What Problem Does This Paper Address?

Deep learning models, despite their impressive performance, are notoriously vulnerable to adversarial examples (AEs) – subtly perturbed inputs designed to fool the model. While ensemble methods offer a promising defense by combining multiple models, their effectiveness is often limited if the individual models share similar decision boundaries. This similarity allows adversaries to craft AEs that transfer effectively across the ensemble, undermining its robustness. Existing ensemble defenses often struggle to enforce sufficient diversity or come at a significant cost to clean accuracy. This paper addresses the challenge of creating ensembles that are inherently robust to adversarial attacks by systematically promoting diversity among their constituent models.

Methods

The core contribution of this paper is the Pairwise Adversarially Robust Loss (PARL) function. PARL aims to create ensembles where each member model has distinct decision boundaries. This is achieved by simultaneously optimizing the outputs and gradients of each layer with respect to network parameters across all models in the ensemble during training.

Key aspects of PARL include:

  • Gradient Diversity: PARL encourages the gradients of the loss function with respect to intermediate layer parameters to point in different directions for different models when processing the same input. This is quantified by minimizing the cosine similarity of these gradients.
  • Representation Diversity: Beyond gradients, PARL also enforces diversity in the internal feature representations by minimizing the correlation between the outputs of intermediate convolutional layers across different models. This captures both the direction and magnitude of feature activations.
  • Pairwise Penalty: The PARL loss function incorporates a penalty term that combines the gradient and representation diversity metrics for every pair of models within the ensemble. This ensures that all model pairs contribute to the overall diversity.
  • Layer-wise Control: The authors demonstrate that the degree of diversity can be controlled by applying the PARL objective to a specific number of initial convolution layers, allowing for a trade-off between robustness and clean accuracy.

Datasets

The paper evaluates PARL on the following standard image classification datasets:

  • CIFAR-10
  • CIFAR-100
  • Tiny Imagenet

Key Insights and Ideas

  • Diversity is Key: The fundamental insight is that diverse decision boundaries within an ensemble are crucial for resisting adversarial attacks. If models are too similar, an attack on one easily transfers to others.
  • Gradients and Outputs Tell the Story: Focusing on gradients and intermediate layer outputs, rather than just final predictions, provides a deeper understanding of how models learn and where their vulnerabilities lie. This allows for more targeted diversity enforcement.
  • Layer-wise Diversity Tuning: The ability to control the number of layers influenced by PARL offers a practical way to balance robustness and accuracy, allowing practitioners to tailor the defense to specific needs.
  • Beyond Gradient Orthogonality: While gradient orthogonality is important, incorporating correlation of intermediate layer outputs provides a more comprehensive measure of feature representation diversity, leading to better overall robustness.

Key Results

  • State-of-the-Art Robustness: PARL significantly improves robustness against black-box transfer attacks. On CIFAR-10 with a Resnet20 model and perturbation strength $\epsilon=0.07$, PARL achieved a 24.8% increase in robust accuracy compared to the state-of-the-art ensemble defense method EIO.
  • Minimal Clean Accuracy Drop: PARL maintains competitive clean accuracy, often surpassing other methods. For instance, compared to TRADES, PARL showed similar robust accuracy but a 3.68% higher clean accuracy.
  • Reduced Training Time: PARL offers a substantial reduction in training time. For the CIFAR-10 Resnet20 model, PARL achieved its results in nearly one-third of the training time compared to EIO.
  • Effective Against Query-Based Attacks: PARL also demonstrates robustness against query-based black-box attacks like the Square Attack, requiring significantly more queries for successful misclassifications compared to surrogate models.
  • Ablation Studies: The paper shows that enforcing diversity in the first 4-6 convolution layers yields the best results, and that both gradient similarity and output correlation terms in the loss function are critical for achieving high robustness.

Limitations and Future Work

  • Hyperparameter Tuning: The performance of PARL is influenced by the hyperparameter $\gamma$, which balances the accuracy-robustness trade-off. While the paper suggests $\gamma=0.25$ as a good default, optimal tuning might be dataset and architecture-dependent.
  • Ensemble Size: While the paper primarily uses ensembles of three models, it briefly explores increasing the number of classifiers, suggesting potential for further gains with larger ensembles, though at the cost of increased training time.
  • Future Work: The authors suggest that further research could explore applying PARL to different architectures and datasets, as well as investigating its effectiveness against adaptive adversaries that are aware of the PARL defense. The potential for combining PARL with other defense mechanisms also remains an open avenue.

Summary updated

BibTeX
@inproceedings{shukla2025guardian,
  author = {Shubhi Shukla and Subhadeep Dalui and Manaar Alam and Shubhajit Datta and Arijit Mondal and Debdeep Mukhopadhyay and Partha Pratim Chakrabarti},
  title = {{Guardian of the Ensembles: Introducing Pairwise Adversarially Robust Loss for Resisting Adversarial Attacks in DNN Ensembles}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure