DeepView: Visualizing the behavior of deep neural networks in a part of the data space

Alexander Schulz, Fabian Hinder, Barbara Hammer

Abstract

Machine learning algorithms using deep architectures have been able to implement increasingly powerful and successful models. However, they also become increasingly more complex, more difficult to comprehend and easier to fool. So far, most methods in the literature investigate the decision of the model for a single given input datum. In this paper, we propose to visualize a part of the decision function of a deep neural network together with a part of the data set in two dimensions with discriminative dimensionality reduction. This enables us to inspect how different properties of the data are treated by the model, such as outliers, adversaries or poisoned data. Further, the presented approach is complementary to the mentioned interpretation methods from the literature and hence might be even more useful in combination with those.

Introduction

Deep Neural Networks (DNNs) are notoriously difficult to interpret. While methods like saliency maps explain individual decisions, they fail to provide a holistic view of the model’s decision landscape. In high-dimensional spaces, standard dimensionality reduction (DR) tools like PCA or t-SNE focus on data topology but ignore the classifier's logic. Consequently, a misclassified point might appear correctly clustered based on pixel similarity, obscuring model errors. DeepView addresses this by creating a visualization where spatial layout directly reflects the model's internal decision function.

Methods

DeepView utilizes two core innovations to map high-dimensional decision boundaries into 2D space:

  1. Fisher UMAP (Discriminative DR): Instead of Euclidean distance, DeepView employs a metric based on the Fisher Information Metric, specifically the Jensen-Shannon divergence between class probabilities. This ensures that points the model considers "class-wise" similar are placed together, regardless of raw input similarity.
  2. Inverse Mapping ($\pi^{-1}$): To visualize the decision boundary across the entire 2D plane, the system must determine the classifier's output for every coordinate. DeepView uses a radial basis function network to approximate an inverse mapping from 2D back to the high-dimensional input space, allowing it to "probe" the classifier across a regular grid.
DeepView mapping process from high-dimensional space to a 2D decision boundary visualization.

Key Insights and Applications

The resulting visualizations provide a "classifier's eye" view of the data:

  • Visualizing Decision "Pockets": The tool reveals unexpected regions of misclassification. For instance, a "car" decision pocket might exist within a "ship" region, highlighting specific tuning weaknesses.
  • Adversarial and Poisoning Detection: Adversarial examples appear as isolated points in incorrect color territories or near jagged boundaries. Similarly, poisoned data samples cluster in ways that deviate from natural distributions, unmasking backdoor attacks.
2D visualization of the decision boundary between two classes, showing the model's confidence levels.

Experimental Results

The researchers evaluated DeepView on CIFAR-10 (ResNet-20) and Fashion-MNIST (ConvNet).

  • Layout Accuracy ($Q_{kNN}$): On CIFAR-10, DeepView’s Fisher UMAP achieved 96.3% accuracy in matching the model's labels, compared to only 18.3% for standard Euclidean UMAP.
  • Boundary Fidelity: The inverse mapping ($Q_{data}$) maintained over 98% accuracy, ensuring the 2D background colors reliably represent high-dimensional model behavior.
  • Performance: The process is computationally feasible, taking approximately 90 seconds for standard datasets on a consumer GPU.

Limitations and Future Work

While effective, the current distance calculation scales at $O(n^2)$, requiring approximation for larger datasets. Additionally, the inverse mapping is an approximation of a non-invertible projection. Future research aims to extend this framework to temporal data, such as speech and text sequences.

Code is available at https://github.com/LucaHermes/DeepView.

Summary updated

Abstract

Machine learning algorithms using deep architectures have been able to implement increasingly powerful and successful models. However, they also become increasingly more complex, more difficult to comprehend and easier to fool. So far, most methods in the literature investigate the decision of the model for a single given input datum. In this paper, we propose to visualize a part of the decision function of a deep neural network together with a part of the data set in two dimensions with discriminative dimensionality reduction. This enables us to inspect how different properties of the data are treated by the model, such as outliers, adversaries or poisoned data. Further, the presented approach is complementary to the mentioned interpretation methods from the literature and hence might be even more useful in combination with those. Code is available at https://github.com/LucaHermes/DeepView .
BibTeX
@inproceedings{schulz2020deepview,
  author = {Alexander Schulz and Fabian Hinder and Barbara Hammer},
  title = {{DeepView: Visualizing the behavior of deep neural networks in a part of the data space}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=QHkeVnaaA3F},
}
Figure