Analysis of functional neural codes of deep learning models

Jung Hoon Lee, Sujith VijayanarXiv 2022 · arXiv · OpenReview

TL;DR

This paper proposes a "functional telescope hypothesis" for deep neural networks (DNNs), suggesting that shallow layers condense high-dimensional inputs into a compressed, homogeneous feature space, while deeper layers expand this space to make final predictions. This compression, while aiding generalization, also appears to be the root cause of DNNs' vulnerability to adversarial attacks.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are powerful but notoriously opaque. Their complex, multi-layered structure makes it difficult to understand how they arrive at their decisions, which is a critical bottleneck for deploying them in high-stakes applications where failures can have severe consequences. Furthermore, DNNs are known to be susceptible to adversarial perturbations – subtle, imperceptible changes to input data that can drastically alter a model's output. This paper aims to shed light on the internal workings of DNNs to understand both their generalization capabilities and their vulnerabilities to these attacks.

Methods

The core of this research utilizes the Self-Organizing Map (SOM), an unsupervised learning algorithm, to analyze the internal representations (or "functional codes") within DNNs. The authors first convert the raw outputs of hidden layers (feature maps) into more abstract vector codes by computing their max, mean, standard deviation, and center of mass. These vector codes are then fed into a SOM, which groups similar codes together, effectively identifying "functional codes." By analyzing how these functional codes are distributed and how they change across different layers of various DNN architectures, the researchers gain insights into the network's internal processing.

Datasets

The study uses the ImageNet dataset for training and evaluating convolutional neural networks (CNNs) like ResNet18, ResNet50, VGG19, and DenseNet121. They also analyze Vision Transformers (ViTs) trained on ImageNet. Additionally, they evaluate ResNet50 on smaller datasets: CIFAR10, CIFAR100, and SVHN.

Key Insights and Ideas

  • The "Functional Telescope" Hypothesis: The paper proposes that DNNs act like telescopes. Shallow layers act as a "condensing lens," mapping diverse inputs into a compact, homogeneous feature space. Deeper layers then act as an "expanding lens," transforming this condensed representation into specific decision variables.
  • Homogeneity in Shallow Layers: Contrary to some expectations, shallow layers of DNNs tend to map inputs to very similar, homogeneous "functional codes." This suggests a strong compression of information.
  • Diversity in Deep Layers: As information progresses through deeper layers, these homogeneous codes are transformed into more diverse representations, leading to distinct outputs for different classes.
  • The Double-Edged Sword of Compression: The homogeneous feature space in shallow layers is crucial for efficient generalization from training data. However, this very compression makes the network susceptible to adversarial perturbations, as small changes in this condensed space can lead to large shifts in the output.
  • Functional Codes Correlate with Labels: The study found that functional codes become more semantically meaningful (i.e., correlated with input labels) in deeper layers.

Key Results

  • Analysis of various CNN architectures (ResNet, VGG, DenseNet) and Vision Transformers consistently showed that earlier layers map inputs to homogeneous functional codes, while later layers map them to more diverse codes.
  • The density of Best Matching Units (BMUs) in the SOM analysis revealed that functional codes are not uniformly distributed, suggesting they represent distinct patterns.
  • Euclidean distances between functional codes of the same class were significantly shorter in deeper layers compared to shallow layers, indicating increased semantic coherence.
  • Adversarial perturbations caused a much larger Euclidean distance between clean and perturbed functional codes in deeper layers compared to random noise, highlighting the vulnerability. For example, adversarial attacks on ResNet18 and ResNet50 caused prediction accuracy to drop from ~77-93% to ~3-6%, while random noise had negligible impact.

Limitations and Future Work

  • Unclear Exact Reason for Compression: While the paper argues that compressed feature space allows for efficient generalization and error landscape estimation, the precise underlying mechanism remains an open question.
  • Focus on Specific Architectures: The study primarily focuses on CNNs and ViTs. Further research could explore other architectures.
  • Future Work: The authors suggest that their "functional telescope hypothesis" can explain why small perturbations in shallow layers can lead to significant changes in deeper layers. This hypothesis could also explain phenomena like "model stitching," where layers from one model can improve another. Further investigation into the exchangeability of functional codes across models is also suggested.

Summary updated

Abstract

Deep neural networks (DNNs), the agents of deep learning (DL), require a massive number of parallel/sequential operations. This makes it difficult to comprehend DNNs' operations and impedes proper diagnosis. Without better knowledge of their internal process, deploying DNNs in high-stakes domains can lead to catastrophic failures. Therefore, to build more reliable DNNs/DL to be deployed in high-stakes real-world problems, it is imperative that we gain insights into DNNs' internal operations underlying their decision-making. Here, we use the self-organizing map (SOM) to analyze DL models' internal codes associated with DNNs' decision-making. Our analyses suggest that shallow layers close to the input layer compress features into condensed space and that deep layers close to the output layer expand feature space. We also found evidence indicating that compressed features may underlie DNNs' vulnerabilities to adversarial perturbations.
BibTeX
@misc{lee2022analysis,
  author = {Jung Hoon Lee and Sujith Vijayan},
  title = {{Analysis of functional neural codes of deep learning models}},
  year = {2022},
  eprint = {2205.10952},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.10952},
}
Figure