TL;DR
N2NSkip introduces sparse, learnable "neuron-to-neuron" skip connections to improve the performance of neural networks pruned at initialization. By re-allocating a portion of the parameter budget to these long-range connections, the authors significantly enhance network connectivity and gradient flow, leading to higher accuracy and faster convergence at extreme sparsity levels (up to 98% pruned) without increasing the total parameter count.
What Problem Does This Paper Address?
Deep Neural Networks (DNNs) are notoriously over-parametrized, making them difficult to deploy on edge devices with limited memory and processing power. While network pruning (removing unnecessary weights) helps, traditional "train-then-prune" methods are computationally expensive.
To save time, researchers have turned to pruning at initialization—identifying and removing weights before the network ever sees a training sample. However, these "born-sparse" networks often suffer from poor "wiring." Because so many connections are missing, the flow of information (and gradients during training) is bottlenecked. This leads to slow convergence and lower final accuracy compared to their dense counterparts. The authors ask: Can we prune a network early while maintaining the rich connectivity needed for fast, effective learning?
Methods
The authors propose N2NSkip, a method that emulates the long-range, sparse connections found in the biological brain.
- Preliminary Pruning: They start with a standard pruning-at-initialization method, such as Randomized Pruning (RP) or Connection Sensitivity Pruning (CSP).
- N2NSkip Insertion: Instead of keeping all remaining weights in sequential layers, they re-allocate a portion of the "parameter budget" to skip connections that jump across non-consecutive layers.
- Sparse & Learnable: Unlike standard ResNet skip connections (which are usually identity mappings or dense projections), N2NSkip connections are sparse convolutional or linear layers. They connect individual neurons or channels across layers using learnable weights.
- Sparsity Neutrality: The total number of parameters remains the same. For example, a network with 10% density might be transformed into 5% sequential connections and 5% N2NSkip connections.
- Heat Diffusion Analysis: To prove these networks are better "wired," the authors treat the DNN as a graph and simulate heat diffusion. By measuring how quickly "heat" (information) spreads from input to output, they can quantitatively compare the connectivity of different architectures.
Datasets
The authors evaluated N2NSkip across several standard benchmarks using VGG19 and ResNet50 architectures:
- CIFAR-10 & CIFAR-100: Small-scale image classification.
- ImageNet (LSVRC 2012): Large-scale competitive image classification.
Key Insights and Ideas
- Connectivity > Raw Parameter Count: The paper suggests that how you connect neurons is just as important as how many connections you have. By introducing "shortcuts," they bypass the bottlenecks created by heavy pruning.
- Bio-Inspiration: The work draws on cognitive science, noting that brain connectivity follows a lognormal distribution with many long-range skips, rather than a purely sequential "layer-by-layer" structure.
- Gradient Highways: N2NSkip connections act as highways for gradients. During backpropagation, the training signal can reach earlier layers more effectively, which explains the observed "superior convergence" (faster learning) in the early epochs.
- A New Metric for DNNs: Using heat diffusion signatures to analyze neural networks is a novel contribution. It provides a mathematical framework to say a network is "well-connected" without relying solely on final accuracy.
Key Results
- Accuracy Boost at High Sparsity: On CIFAR-100 with a VGG19 at only 2% density (98% pruned), N2NSkip improved accuracy from 55.43% to 61.21% compared to standard random pruning.
- Faster Convergence: In experiments on ResNet50, N2NSkip networks reached baseline-level accuracy within the first 50 epochs, whereas standard pruned networks lagged significantly behind.
- ImageNet Success: The benefits scaled to ImageNet. At 20% density, N2NSkip-RP achieved 70.03% accuracy, significantly outperforming the standard RP baseline of 65.32%.
- Connectivity Validation: The heat diffusion analysis showed that N2NSkip networks have "heat signatures" much closer to the original dense reference models than standard pruned networks do.
Limitations and Future Work
- Hardware Efficiency: While N2NSkip maintains theoretical sparsity, unstructured or semi-structured sparse skip connections can be difficult to accelerate on current GPU hardware, which favors dense matrix multiplications.
- Connection Strategy: Currently, N2NSkip connections are added randomly between neurons. The authors suggest that future work could explore more "intelligent" ways to choose which neurons to link.
- Graph Theory Expansion: The authors believe the field could benefit from deeper explorations into other graph-theoretical properties (like small-worldness or expander graphs) to further optimize sparse architectures.
Summary updated