N2NSkip: Learning Highly Sparse Networks using Neuron-to-Neuron Skip Connections

Arvind R. Subramaniam, Avinash SharmaarXiv 2020 · arXiv

TL;DR

N2NSkip introduces sparse, learnable "neuron-to-neuron" skip connections to improve the performance of neural networks pruned at initialization. By re-allocating a portion of the parameter budget to these long-range connections, the authors significantly enhance network connectivity and gradient flow, leading to higher accuracy and faster convergence at extreme sparsity levels (up to 98% pruned) without increasing the total parameter count.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously over-parametrized, making them difficult to deploy on edge devices with limited memory and processing power. While network pruning (removing unnecessary weights) helps, traditional "train-then-prune" methods are computationally expensive.

To save time, researchers have turned to pruning at initialization—identifying and removing weights before the network ever sees a training sample. However, these "born-sparse" networks often suffer from poor "wiring." Because so many connections are missing, the flow of information (and gradients during training) is bottlenecked. This leads to slow convergence and lower final accuracy compared to their dense counterparts. The authors ask: Can we prune a network early while maintaining the rich connectivity needed for fast, effective learning?

Methods

The authors propose N2NSkip, a method that emulates the long-range, sparse connections found in the biological brain.

  1. Preliminary Pruning: They start with a standard pruning-at-initialization method, such as Randomized Pruning (RP) or Connection Sensitivity Pruning (CSP).
  2. N2NSkip Insertion: Instead of keeping all remaining weights in sequential layers, they re-allocate a portion of the "parameter budget" to skip connections that jump across non-consecutive layers.
  3. Sparse & Learnable: Unlike standard ResNet skip connections (which are usually identity mappings or dense projections), N2NSkip connections are sparse convolutional or linear layers. They connect individual neurons or channels across layers using learnable weights.
  4. Sparsity Neutrality: The total number of parameters remains the same. For example, a network with 10% density might be transformed into 5% sequential connections and 5% N2NSkip connections.
  5. Heat Diffusion Analysis: To prove these networks are better "wired," the authors treat the DNN as a graph and simulate heat diffusion. By measuring how quickly "heat" (information) spreads from input to output, they can quantitatively compare the connectivity of different architectures.

Datasets

The authors evaluated N2NSkip across several standard benchmarks using VGG19 and ResNet50 architectures:

  • CIFAR-10 & CIFAR-100: Small-scale image classification.
  • ImageNet (LSVRC 2012): Large-scale competitive image classification.

Key Insights and Ideas

  • Connectivity > Raw Parameter Count: The paper suggests that how you connect neurons is just as important as how many connections you have. By introducing "shortcuts," they bypass the bottlenecks created by heavy pruning.
  • Bio-Inspiration: The work draws on cognitive science, noting that brain connectivity follows a lognormal distribution with many long-range skips, rather than a purely sequential "layer-by-layer" structure.
  • Gradient Highways: N2NSkip connections act as highways for gradients. During backpropagation, the training signal can reach earlier layers more effectively, which explains the observed "superior convergence" (faster learning) in the early epochs.
  • A New Metric for DNNs: Using heat diffusion signatures to analyze neural networks is a novel contribution. It provides a mathematical framework to say a network is "well-connected" without relying solely on final accuracy.

Key Results

  • Accuracy Boost at High Sparsity: On CIFAR-100 with a VGG19 at only 2% density (98% pruned), N2NSkip improved accuracy from 55.43% to 61.21% compared to standard random pruning.
  • Faster Convergence: In experiments on ResNet50, N2NSkip networks reached baseline-level accuracy within the first 50 epochs, whereas standard pruned networks lagged significantly behind.
  • ImageNet Success: The benefits scaled to ImageNet. At 20% density, N2NSkip-RP achieved 70.03% accuracy, significantly outperforming the standard RP baseline of 65.32%.
  • Connectivity Validation: The heat diffusion analysis showed that N2NSkip networks have "heat signatures" much closer to the original dense reference models than standard pruned networks do.

Limitations and Future Work

  • Hardware Efficiency: While N2NSkip maintains theoretical sparsity, unstructured or semi-structured sparse skip connections can be difficult to accelerate on current GPU hardware, which favors dense matrix multiplications.
  • Connection Strategy: Currently, N2NSkip connections are added randomly between neurons. The authors suggest that future work could explore more "intelligent" ways to choose which neurons to link.
  • Graph Theory Expansion: The authors believe the field could benefit from deeper explorations into other graph-theoretical properties (like small-worldness or expander graphs) to further optimize sparse architectures.

Summary updated

Abstract

The over-parametrized nature of Deep Neural Networks leads to considerable hindrances during deployment on low-end devices with time and space constraints. Network pruning strategies that sparsify DNNs using iterative prune-train schemes are often computationally expensive. As a result, techniques that prune at initialization, prior to training, have become increasingly popular. In this work, we propose neuron-to-neuron skip connections, which act as sparse weighted skip connections, to enhance the overall connectivity of pruned DNNs. Following a preliminary pruning step, N2NSkip connections are randomly added between individual neurons/channels of the pruned network, while maintaining the overall sparsity of the network. We demonstrate that introducing N2NSkip connections in pruned networks enables significantly superior performance, especially at high sparsity levels, as compared to pruned networks without N2NSkip connections. Additionally, we present a heat diffusion-based connectivity analysis to quantitatively determine the connectivity of the pruned network with respect to the reference network. We evaluate the efficacy of our approach on two different preliminary pruning methods which prune at initialization, and consistently obtain superior performance by exploiting the enhanced connectivity resulting from N2NSkip connections.
BibTeX
@misc{subramaniam2020n,
  author = {Arvind R. Subramaniam and Avinash Sharma},
  title = {{N2NSkip: Learning Highly Sparse Networks using Neuron-to-Neuron Skip Connections}},
  year = {2020},
  eprint = {2208.03662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.03662},
}
Figure