OTJR: Optimal Transport Meets Optimal Jacobian Regularization for Adversarial Robustness

Binh Minh Le, Shahroz Tariq, Simon S. Woo

Abstract

Adversarial robustness remains a critical challenge for Deep Neural Networks (DNNs). While Adversarial Training (AT) and Jacobian Regularization (JR) are established defense mechanisms, their effective integration has been hindered by the inefficiency of random projections in traditional JR. This paper introduces OTJR (Optimal Transport with Jacobian Regularization), a framework that leverages Sliced Wasserstein (SW) distance to identify optimal trajectories in latent space. By aligning adversarial and clean distributions and regularizing the model along these optimal paths, OTJR achieves superior robustness across standard and real-world benchmarks, including CIFAR-10 and CIFAR-100.

Introduction

Deep Neural Networks are notoriously fragile; tiny, imperceptible adversarial perturbations can easily trick them into misclassification. Current defenses primarily follow two paradigms: Adversarial Training (AT), which trains models on perturbed samples, and Jacobian Regularization (JR), which mathematically "silences" a model's sensitivity by ensuring small input changes do not lead to large output changes. However, traditional JR methods often rely on random projections to estimate the Jacobian norm, which is computationally inefficient and lacks directionality. OTJR addresses this gap by using Optimal Transport theory to guide the regularization process.

Methodology

OTJR bridges the gap between AT and JR by utilizing the Sliced Wasserstein (SW) distance. Unlike computationally heavy alternatives, the SW distance allows for the efficient comparison of clean and adversarial distributions within the model's latent space.

The core innovation of OTJR is the concept of "Optimal Trajectories." Instead of regularizing the Jacobian along arbitrary random directions, OTJR uses the SW distance to determine the specific direction an adversarial sample must move to align with the clean distribution. This informed trajectory provides a more effective target for regularization than traditional random guessing.

Comparison of conventional Jacobian Regularization using random projections versus the proposed OTJR using optimal trajectories in the latent space.

The training objective is a unified loss function comprising standard cross-entropy loss on adversarial samples, a Jacobian regularization term focused on the identified optimal trajectories, and the SW distance itself to pull the latent distributions together.

Key Insights and Results

The authors evaluated OTJR on standard benchmarks (CIFAR-10/100, Tiny-IMAGENET) and curated "WEB" datasets to simulate real-world online attacks.

Key findings include:

  • Superior Robustness: OTJR achieved 52.57% accuracy on CIFAR-10 under the rigorous AutoAttack benchmark, outperforming TRADES, PGD-AT, and SAT.
  • Gradient Silencing: The authors found that JR acts as an agnostic stabilizer. As shown in Figure 3, models utilizing JR exhibit significantly lower and more stable gradient norms compared to standard PGD-AT, effectively "silencing" the model's sensitivity to input noise.
Gradient norm comparison between PGD-AT and Jacobian Regularization (JR) over CIFAR-10 samples, demonstrating the gradient silencing effect.
  • Scalability: By using Sliced Wasserstein distance, OTJR scales effectively to complex datasets like CIFAR-100, where previous Optimal Transport-based defenses often failed to converge within reasonable timeframes.
  • Decision Boundaries: Visualization indicates that OTJR produces much larger and more stable "decision cells" around data points, making it significantly harder for an attacker to push a sample across a classification boundary.

Conclusion

OTJR demonstrates that the synergy between Optimal Transport and Jacobian Regularization can significantly enhance adversarial robustness. By replacing random projections with optimal trajectories, the framework makes regularization "aware" of specific attack vectors. While OTJR introduces some computational overhead due to Jacobian calculations, its performance gains and scalability make it a compelling defense strategy for modern DNNs. Future work may involve combining OTJR with other techniques like Weight Perturbation (AWP) to further push the robustness ceiling.

Summary updated

BibTeX
@inproceedings{le2024otjr,
  author = {Binh Minh Le and Shahroz Tariq and Simon S. Woo},
  title = {{OTJR: Optimal Transport Meets Optimal Jacobian Regularization for Adversarial Robustness}},
  booktitle = {ACCV},
  year = {2024},
  doi = {10.1007/978-981-96-0963-5\_7},
}
Figure