NeuroView: Explainable Deep Network Decision Making

C. J. Barberan, Randall Balestriero, Richard G. BaraniukarXiv 2021 · arXiv

Introduction

Deep neural networks (DNs) have achieved remarkable success across various domains, yet they remain largely "black boxes." This opacity poses significant risks in high-stakes applications where understanding the why behind a decision is as important as the decision itself. Traditional interpretability methods are often post-hoc or limited to local pixel-level explanations. NeuroView introduces a novel architectural family designed for intrinsic interpretability, establishing a direct causal link between internal features and final classifications without compromising performance.

The Challenge of Opaque Architectures

Current deep learning models are difficult to audit. A model might classify an image correctly but for the wrong reasons—for instance, identifying a "digit" based on background color rather than shape. This lack of transparency makes it hard to detect hidden biases. The authors of NeuroView argue that true understanding requires seeing how units across the entire depth of a network coordinate to reach a conclusion.

Methods: Designing for Interpretability

NeuroView modifies standard architectures (like ResNet or VGG) to create a "glass box" model. The transformation involves three primary architectural shifts:

  1. Global Connectivity: Instead of relying solely on the final layer for inference, NeuroView concatenates the outputs of every nonlinear unit from every layer into a single, massive feature vector.
  2. Soft Vector Quantization (Soft VQ): To simplify these activations, a sigmoid function is applied, mapping them to a [0, 1] range. This effectively treats internal units as "on/off" feature detectors.
  3. Wide Linear Classifier: This combined vector feeds into a single, wide linear classifier. The linearity is key: the weights ($W$) assigned to each input provide a direct, quantitative measure of how much each specific internal unit contributes to a class prediction.
Evolution of the NeuroView architecture from standard inference (a) to global connectivity with Soft VQ and pooling (d).

To manage computational overhead, dimensionality reduction (such as Max or Mean pooling) is used to ensure the final layer's input size remains proportional to the total number of units.

Key Insights and Interpretability

The primary advantage of NeuroView is its ability to map abstract internal units to human-understandable concepts. By analyzing the weights of the final linear layer, researchers can identify exactly which features—such as "sky," "textures," or "whiskers"—drive a specific classification.

Interpretability in action: identifying specific concepts like "Grass" or "Whiskers" and their corresponding weights for class prediction.

Causal Mapping and Bias Detection

Because the connection is linear, the model reveals both positive and negative contributions. For example, the presence of a "sky" concept might actively decrease the probability of an image being classified as a "ballroom." This allows for rigorous bias detection. In tests using Neural Style Transfer, the authors found that models could develop "texture aversion," where early-layer units detecting specific colors or textures were assigned high negative weights to avoid misclassification based on stylistic noise.

Experimental Results

The researchers validated NeuroView across diverse datasets, including Places10 (scenes), ModelNet40 (3D objects), various textures, and UrbanSound8k (audio).

  • Performance Parity: NeuroView consistently matched or exceeded the accuracy of standard "black box" networks. On the Places10 dataset, a NeuroView-modified VGG11 achieved 94.0% accuracy, surpassing the standard version's 92.9%. This refutes the common belief that interpretability requires a trade-off in performance.
  • Robustness via Perturbation: Through "Counterfactual Accuracy" tests, the team confirmed model dependencies. For the "aquarium" class, removing the red color channel actually increased accuracy, proving the model was correctly focusing on blue-tinted textures rather than simple color values.

Conclusion and Future Directions

NeuroView demonstrates that deep networks can be both high-performing and transparent. By architectural design, it provides a roadmap for auditing AI decisions in real-time. Future work aims to scale this approach to massive datasets like ImageNet and explore automated concept labeling for specialized fields like medical imaging. Additionally, the weights provided by NeuroView could eventually be used to prune redundant units, leading to more efficient, as well as more understandable, AI systems.

Summary updated

BibTeX
@misc{barberan2021neuroview,
  author = {C. J. Barberan and Randall Balestriero and Richard G. Baraniuk},
  title = {{NeuroView: Explainable Deep Network Decision Making}},
  year = {2021},
  eprint = {2110.07778},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.07778},
}
Figure