Abstract
Deep convolutional neural networks (CNNs) have achieved breakthrough performance in many pattern recognition tasks such as image classification. However, the development of high-quality deep models typically relies on a substantial amount of trial-and-error, as there is still no clear understanding of when and why a deep model works. In this paper, we present a visual analytics approach for better understanding, diagnosing, and refining deep CNNs. We formulate a deep CNN as a directed acyclic graph. Based on this formulation, a hybrid visualization is developed to disclose the multiple facets of each neuron and the interactions between them. In particular, we introduce a hierarchical rectangle packing algorithm and a matrix reordering algorithm to show the derived features of a neuron cluster. We also propose a biclustering-based edge bundling method to reduce visual clutter caused by a large number of connections between neurons. We evaluated our method on a set of CNNs and the results are generally favorable.
Introduction
Despite their breakthrough performance, Deep Convolutional Neural Networks (CNNs) are frequently criticized as "black boxes." Designing and refining these models often involves a "dark art" of manual tuning and trial-and-error. This opacity stems from two primary hurdles: the sheer complexity and scale of modern architectures—comprising hundreds of layers and millions of connections—and the functional opacity of individual neurons. This paper introduces CNNVis, a visual analytics toolkit designed to help experts move beyond simple performance metrics and understand why a model succeeds or fails.
The CNNVis Methodology
To manage the scale of modern CNNs, the authors formulate the network as a Directed Acyclic Graph (DAG). The system employs several novel techniques to condense information without losing critical insights:
- Hierarchical Clustering: Layers and neurons are clustered based on activation patterns, allowing the system to display "representative" neurons that summarize layer-wide behavior.
- Hierarchical Rectangle Packing: To visualize learned features, the system displays the top-5 image patches that maximally activate a neuron cluster. These are organized using a treemap-like layout where size indicates activation strength.
- Matrix Reordering: Neuron activations across classes are shown in a matrix. A divide-and-conquer version of the Held-Karp algorithm reorders rows to reveal how specific neuron groups respond to different data categories.
- Biclustering-based Edge Bundling: To solve the "hairball" problem of dense connections, a weighted biclustering method (based on the Apriori algorithm) bundles edges with similar high-magnitude weights, highlighting the most influential paths.
Visual Analytics in Action
The resulting interface allows researchers to interactively explore the network's internal state. By visualizing the flow of information from early layers to final classification, users can identify structural redundancies or training failures.
Key Insights and Case Studies
The utility of CNNVis was demonstrated through several expert-led case studies using datasets like CIFAR-10 and ImageNet:
- Cluster Purity: A well-trained network typically shows "impure" clusters in early layers (responding to low-level features like edges) and "pure" clusters in deeper layers (responding to specific classes). If deep layers remain impure, it suggests the network is too shallow for the task.
- Identifying Redundancy: Analysis of overfitted or excessively deep networks revealed that many neurons or entire layers often learn nearly identical features. For instance, a 20-layer "DeepCNN" performed worse than a 10-layer "BaseCNN" on CIFAR-10 because the extra layers were largely redundant.
- Training Diagnosis: In one instance, CNNVis helped diagnose a "stuck" training process. Visualization revealed an overwhelming majority of negative weights, which caused the ReLU activation functions to output zero and "kill" the gradient. The expert fixed this by adding Batch Normalization layers.
Conclusion
CNNVis provides a powerful framework for opening the CNN black box. While currently focused on static snapshots of models, future work aims to support real-time visualization during training and adapt the DAG formulation for architectures with feedback loops, such as Recurrent Neural Networks (RNNs). By transforming millions of parameters into interpretable visual patterns, CNNVis enables a more principled approach to deep learning research.
Summary updated