Abstract
Machine learning algorithms using deep architectures have become increasingly powerful but also more complex and difficult to comprehend. Most existing interpretation methods focus on single-input decisions. We propose DeepView, a tool that visualizes the decision function of a deep neural network (DNN) alongside the dataset in two dimensions using discriminative dimensionality reduction. This allows for the inspection of model behavior regarding outliers, adversaries, and poisoned data. Code is available at https://github.com/LucaHermes/DeepView.
Introduction
Deep neural networks (DNNs) are often criticized as "black boxes." While local explanation methods like saliency maps provide insights into individual predictions, they fail to offer a global perspective on the model's decision landscape. Understanding where class boundaries lie and how a model generalizes in regions devoid of training data is crucial for identifying vulnerabilities. The primary obstacle is the high dimensionality of the input space, which standard dimensionality reduction (DR) techniques like t-SNE or UMAP struggle to project without losing the classifier's underlying logic.
Methodology
DeepView addresses the dimensionality challenge through two primary innovations: Fisher UMAP and a learned inverse mapping.
Fisher UMAP (Discriminative DR)
Standard DR techniques typically preserve Euclidean distances, which may not reflect the model's internal logic. DeepView utilizes Fisher UMAP, which employs a metric based on the Fisher Information Matrix. Specifically, it calculates the Jensen-Shannon divergence between the probability distributions produced by the model for different inputs. This ensures that points the model perceives as similar are clustered together in the 2D projection, regardless of their raw pixel-level similarity.
Inverse Mapping and Boundary Visualization
To visualize decision boundaries, the tool must predict class labels for every point in the 2D space. DeepView implements a mathematically grounded inverse mapping ($\pi^{-1}$) that projects 2D coordinates back into the high-dimensional input space. This allows the system to "probe" the model at any pixel in the visualization, coloring the background to represent predicted classes and confidence levels (entropy).
Key Insights and Applications
DeepView provides a model-centric view of the data, revealing phenomena that traditional visualizations obscure.
- Decision "Pockets" and Overfitting: The tool can identify small, isolated regions where the model's classification flips unexpectedly. These "pockets" often indicate overfitting or adversarial vulnerabilities.
- Adversarial and Poisoning Detection: By visualizing the decision landscape, researchers can spot adversarial examples hidden within class territories. Furthermore, DeepView can separate "poisoned" samples (e.g., images with a hidden trigger) into distinct clusters, even if they appear normal to human observers.
- Interactive Exploration: The inverse mapping enables users to click on any point in the 2D plot to see a reconstructed image representing the model's "perception" of that coordinate.
Results
The effectiveness of DeepView was evaluated on benchmark datasets including CIFAR-10 and Fashion-MNIST. A key metric, $Q_{kNN}$, was used to measure how well the 2D projection preserves the model's labels. On CIFAR-10, Fisher UMAP achieved a score of 96.3%, significantly outperforming standard Euclidean UMAP (18.3%). This demonstrates that Fisher UMAP successfully captures the discriminative logic of the DNN. In practical tests, DeepView successfully visualized adversarial "ships" hidden within "bird" class territory, showing exactly how the decision boundary had been warped.
Conclusion
DeepView offers a powerful, global perspective on DNN behavior, complementing existing local explanation methods. While challenges remain regarding the scalability of the $O(N^2)$ distance calculations and the fidelity of inverse mapping reconstructions, it represents a significant step toward making deep learning models more transparent and robust.
Summary updated