Achieving robustness in classification using optimal transport with hinge regularization

Mathieu Serrurier

TL;DR

This paper introduces a novel framework for training robust neural networks by leveraging optimal transport theory. By reformulating classification as an optimal transport problem with a hinge regularization, they achieve certifiable robustness guarantees against adversarial attacks without sacrificing accuracy. This approach also provides interpretable adversarial examples that visually highlight the model's decision boundaries.

What Problem Does This Paper Address?

Deep neural networks are notoriously vulnerable to adversarial attacks – small, imperceptible perturbations to input data can cause misclassifications. While constraining the Lipschitz constant of a network is known to improve robustness, it often makes training more difficult with standard loss functions. This paper tackles the challenge of building inherently robust classifiers by theoretically grounding robustness in optimal transport, specifically by integrating a Lipschitz constraint directly into the learning objective.

Methods

The core of the proposed method is a new loss function for binary classification based on a hinge-regularized version of the Kantorovich-Rubinstein dual formulation of the Wasserstein distance. This formulation, when minimized by a 1-Lipschitz neural network, theoretically guarantees adversarial robustness. The authors prove that this hinge-regularized loss still corresponds to an optimal transportation problem with a solvable dual formulation. They also establish geometric properties of the optimal solution and extend the approach to multi-class classification. To enforce the 1-Lipschitz constraint on neural networks, they employ spectral normalization for dense layers and propose a tighter estimation for convolutional layers.

Datasets

The paper evaluates their approach on standard benchmark datasets:

  • MNIST [21]
  • CIFAR-10 [20]
  • CelebA dataset [23] (for the binary eyeglasses detection task)

Key Insights and Ideas

  • Optimal Transport as a Robustness Framework: The paper elegantly frames adversarial robustness not just as a defense mechanism but as an intrinsic property derived from optimal transport theory. The "transport plan" between data distributions directly relates to how perturbations affect classification.
  • Hinge Regularization for Certifiable Robustness: Introducing a hinge loss into the optimal transport formulation provides a clear margin for classification, leading to provable robustness guarantees. This is a key theoretical contribution.
  • Interpretable Adversarial Examples: The method generates adversarial examples that are not just misclassifications but also visually meaningful, showing how inputs are "transported" across the decision boundary. This offers a form of model interpretability.
  • Theoretical Link Between Lipschitz and Robustness: The work solidifies the theoretical connection between a network's Lipschitz constant and its robustness, providing a principled way to achieve both.

Key Results

  • Improved Robustness: Experiments show that the proposed hinge-KR approach achieves state-of-the-art robustness against various adversarial attacks (FGSM, PGD, DeepFool, CW) across different datasets, often outperforming existing methods.
  • Comparable Accuracy: Crucially, the proposed method achieves these robustness gains without a significant drop in classification accuracy compared to non-robust baselines.
  • Interpretable Attacks: Adversarial examples generated by the model for the CelebA dataset are visually interpretable, highlighting specific facial features (eyes, nose) that are perturbed to cause misclassification. This provides insights into the model's decision-making process.

Limitations and Future Work

  • Computational Cost: While inference is not significantly impacted, the training process for the proposed framework can be more computationally expensive due to the optimal transport formulation and the enforcement of Lipschitz constraints.
  • Generalization to Other Tasks: The paper focuses on classification. Extending this framework to other tasks like object detection or segmentation, where optimal transport might be more complex to apply, could be a future direction.
  • Further Exploration of Activation Functions: The paper mentions several norm-preserving activation functions and their impact. Further research into optimizing these for robustness and efficiency could be beneficial.
  • Theoretical Analysis of Multi-class Extension: While the multi-class approach is presented, further theoretical analysis of its robustness guarantees compared to the binary case could be valuable.

Summary updated

Abstract

Adversarial examples have pointed out Deep Neural Networks vulnerability to small local noise. It has been shown that constraining their Lipschitz constant should enhance robustness, but make them harder to learn with classical loss functions. We propose a new framework for binary classification, based on optimal transport, which integrates this Lipschitz constraint as a theoretical requirement. We propose to learn 1-Lipschitz networks using a new loss that is an hinge regularized version of the Kantorovich-Rubinstein dual formulation for the Wasserstein distance estimation. This loss function has a direct interpretation in terms of adversarial robustness together with certifiable robustness bound. We also prove that this hinge regularized version is still the dual formulation of an optimal transportation problem, and has a solution. We also establish several geometrical properties of this optimal solution, and extend the approach to multi-class problems. Experiments show that the proposed approach provides the expected guarantees in terms of robustness without any significant accuracy drop. The adversarial examples, on the proposed models, visibly and meaningfully change the input providing an explanation for the classification.
BibTeX
@inproceedings{serrurier2021achieving,
  author = {Mathieu Serrurier},
  title = {{Achieving robustness in classification using optimal transport with hinge regularization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00057},
}
Figure