Interpreting Robustness Proofs of Deep Neural Networks

Debangshu Banerjee, Avaljot Singh, Gagandeep Singh

Introduction

Deep Neural Networks (DNNs) are increasingly deployed in safety-critical domains, yet they remain largely "black boxes." While formal verification provides mathematical guarantees that a network is robust to specific input perturbations, these proofs are often composed of high-dimensional geometric constraints that offer little insight into why a network is robust. This lack of interpretability creates a significant barrier to trust. This paper introduces SuPFEx (Sufficient Proof Feature Extraction), an algorithm designed to bridge the gap between formal verification and human-understandable explanations.

The Challenge of Proof Interpretability

Formal verification ensures that for all inputs within a defined region (e.g., a 2% pixel perturbation), the network's output remains constant. However, current tools do not explain which visual cues the proof relies on. A proof might hold due to semantically meaningful features, such as the shape of a digit, or it might rely on "spurious features"—statistical artifacts in the background pixels that do not generalize. Standard interpretability methods like saliency maps are insufficient because they explain individual predictions rather than the infinite set of inputs covered by a formal proof.

Methodology: SuPFEx

SuPFEx identifies a minimal subset of "proof features" that are sufficient to maintain a formal robustness guarantee. The method follows three main steps:

  1. Feature Identification: Neurons in the penultimate layer are treated as features. For a given input region, a "proof feature" is defined by the range of values a specific neuron can take.
  2. Greedy Extraction: SuPFEx employs a greedy pruning strategy. It iteratively removes the least important features and re-runs the formal verifier to ensure the proof still holds. This process continues until a minimal, sufficient set of features is identified.
  3. Visualization: To interpret these features, the algorithm calculates the mean gradient of the extracted proof features with respect to the input pixels, generating a heat map that highlights the pixels most critical to the proof.

Experimental Evaluation

The authors evaluated SuPFEx on MNIST and CIFAR-10 datasets across various training regimes:

  • Standard Training: Often results in proofs relying on spurious background noise.
  • Adversarial Training (PGD): Improves robustness but may still rely on non-human-aligned features.
  • Certifiable Training (CROWN-IBP): Highly conservative; proofs may ignore meaningful object parts to ensure safety.
  • COLT (Combined Adversarial and Certifiable Training): Found to produce the most "human-aligned" proofs, focusing on semantically relevant parts of the image.

Key Results

SuPFEx significantly reduces the complexity of robustness proofs. By isolating the "load-bearing" neurons, the algorithm can reduce the number of features required for a proof by orders of magnitude.

Comparison of proof feature counts between a baseline and SuPFEx, demonstrating a significant reduction in proof complexity.

Qualitatively, SuPFEx reveals that as models become more robust through specialized training like COLT, the focus of the proof shifts from disorganized pixel clouds to clean outlines of the target objects. Furthermore, the identified features remain consistent across different mathematical verifiers (e.g., IBP, DeepZ), indicating that SuPFEx captures intrinsic properties of the network's robustness.

Conclusion and Future Work

SuPFEx provides a novel lens into the formal guarantees of DNNs, revealing that different training methods secure robustness through vastly different mechanisms. While currently limited by the computational cost of repeated verification calls, future work aims to scale SuPFEx to larger architectures and datasets like ImageNet by optimizing the feature selection process.

Summary updated

BibTeX
@inproceedings{banerjee2024interpreting,
  author = {Debangshu Banerjee and Avaljot Singh and Gagandeep Singh},
  title = {{Interpreting Robustness Proofs of Deep Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=Ev10F9TWML},
}
Figure