Interactive Analysis of CNN Robustness

Stefan Sietzen, Mathias Lechner, Judy Borowski, Ramin M. Hasani, Manuela Waldner

Introduction

Convolutional Neural Networks (CNNs) have achieved remarkable success on standard image recognition benchmarks, often rivaling human performance. However, these models remain notoriously "brittle" when faced with out-of-distribution data or semantic perturbations. To bridge the gap between high benchmark accuracy and real-world reliability, we present Perturber, a web-based interactive tool designed to "stress-test" CNNs in real-time through 3D scene manipulation.

The Challenge of Model Brittleness

Traditional robustness testing is often an offline, batch-oriented process. Researchers generate perturbed datasets, run inference, and analyze aggregate metrics hours later. This latency hinders the development of an intuitive understanding of model failure modes. A model might correctly classify a dog in a standard pose but fail when the camera angle shifts or the background changes. Disentangling whether a model relies on the geometric shape of an object or its surface texture is difficult without fluid, hypothesis-driven exploration.

Methods: The Perturber Architecture

Perturber addresses these challenges by providing a visual analytics (VA) playground built on three core technical pillars:

  • 3D Scene Manipulation: Utilizing Three.js, the tool renders 3D objects (e.g., dogs, cats, vehicles), allowing users to independently control geometry (rotation, zoom), scene parameters (lighting, background), and shape (morphing).
  • Real-Time Inference: By leveraging TensorFlow.js, Perturber executes model inference directly in the browser. This enables a fluid feedback loop where top-5 predictions update instantly as users adjust sliders.
  • Neuron Activation Views: To provide transparency, the tool visualizes internal feature maps. It combines feature visualizations (synthetic images representing what a neuron "prefers") with live activation maps to show which parts of the input are triggering specific internal representations.

Key Insights: Texture Bias and the Robustness Paradox

Interactive exploration with Perturber has revealed critical vulnerabilities in standard CNN architectures that are often obscured by aggregate metrics.

The Texture Bias

A prominent finding is the "texture bias" of standard CNNs. When the texture is removed from a 3D dog model—leaving only a smooth silhouette—standard models often misclassify the object as a "piggy bank." In contrast, robust variants (such as shape-biased or adversarially trained models) are more likely to maintain a canine-related classification, demonstrating a more human-like reliance on geometry.

Figure 1: The Perturber interface comparing standard and adversarially trained models. The bottom row illustrates the 'texture bias' where a textureless dog is misclassified as a piggy bank by the standard model, while the robust model identifies a Siberian Husky.

The Robustness Paradox

Counter-intuitively, models trained to be robust against pixel-level adversarial noise often become more vulnerable to semantic changes. Perturber allows researchers to observe that these "robust" models can be flipped by simple yaw rotations or background swaps that do not affect standard models as severely. This suggests that current robustification techniques may trade one type of vulnerability for another.

Results and Evaluation

Perturber achieves high performance, maintaining up to 40+ FPS on modern hardware, which is essential for effective manual exploration. In case studies with machine learning experts, the tool successfully helped researchers replicate known literature findings and generate new hypotheses about model behavior.

Furthermore, quantitative verification using the Background Challenge dataset confirmed that observations made within the tool generalize: "robustified" networks achieved only 12.3% accuracy compared to 22.3% for standard models when backgrounds were varied, confirming the tool's ability to highlight real-world performance gaps.

Conclusion and Future Work

Perturber demonstrates the power of interactive visual analytics in uncovering the hidden vulnerabilities of CNNs. By moving from static evaluation to interactive manipulation, researchers can better understand and eventually mitigate the brittleness of modern computer vision models. Future work will focus on automated neuron discovery to highlight significant activation changes and extending support to encoder-decoder architectures used in tasks like semantic segmentation.

Summary updated

BibTeX
@article{sietzen2021interactive,
  author = {Stefan Sietzen and Mathias Lechner and Judy Borowski and Ramin M. Hasani and Manuela Waldner},
  title = {{Interactive Analysis of CNN Robustness}},
  journal = {Computer Graphics Forum},
  year = {2021},
  doi = {10.1111/cgf.14418},
}
Figure