Abstract
Unstructured neural network pruning is an effective technique that can significantly reduce theoretical model size, computation demand and energy consumption of large neural networks without compromising accuracy. However, a number of fundamental questions about pruning are not answered yet. For example, do the pruned neural networks contain the same representations as the original network? Is pruning a compression or evolution process? Does pruning only work on trained neural networks? What is the role and value of the uncovered sparsity structure? In this paper, we strive to answer these questions by analyzing three unstructured pruning methods (magnitude based pruning, post-pruning re-initialization, and random sparse initialization). We conduct extensive experiments using the Singular Vector Canonical Correlation Analysis (SVCCA) tool to study and contrast layer representations of pruned and original ResNet, VGG, and ConvNet models. We have several interesting observations: 1) Pruned neural network models evolve to substantially different representations while still maintaining similar accuracy. 2) Initialized sparse models can achieve reasonably good accuracy compared to well-engineered pruning methods. 3) Sparsity structures discovered by pruning models are not inherently important or useful.
Introduction
Deep neural networks (DNNs) are notoriously oversized, posing significant challenges for deployment on resource-constrained edge devices. While "pruning"—the removal of non-essential weights—is a standard solution to reduce model size, our theoretical understanding of the process remains limited. This research investigates whether pruning acts as a simple compression of existing knowledge or if it forces the model to undergo an evolutionary process, learning entirely new internal representations. Specifically, the study evaluates three assumptions: whether pruned models retain original representations, if dense pre-training is necessary, and if the specific structure of a pruning mask is inherently valuable.
Methods
The study compares three primary pruning strategies to understand their impact on model representation:
- Magnitude-Based Pruning: Training a full model and subsequently removing the smallest weights.
- Post-Pruning Re-initialization: Extracting the mask from a pruned model, resetting weights to random values, and retraining from scratch.
- Random Sparse Initialization: Creating a random mask and training from scratch.
To analyze the internal state of these models, the researchers employed Singular Vector Canonical Correlation Analysis (SVCCA). This tool quantifies the similarity between mathematical representations of layers across different models, providing a window into how "similar" their internal logic remains after pruning.
Experiments were conducted on CIFAR-10 and CIFAR-100 datasets using ResNet-34, VGG-16, and ConvNet architectures.
Results and Analysis
Evolution Over Compression
A key finding is that as pruning becomes more aggressive, the model's internal representations diverge significantly from the original. At 90% sparsity, the median similarity between pruned and original layers often drops below 50%. This suggests that the model does not merely "squeeze" original knowledge but evolves new ways to solve the task with limited parameters.
The Role of Sparsity Structure
Contrary to the "Lottery Ticket Hypothesis," the specific pattern of the pruning mask appears less critical in unstructured pruning. On CIFAR-10 with ResNet-34 at 90% sparsity, Magnitude Pruning and a completely Random Sparse model both achieved an identical accuracy of 87.2%. This indicates that the "winning ticket" might be more about the existence of sparsity than the specific structure uncovered by complex algorithms.
Architecture Robustness and Learning Dynamics
The study highlights that ResNet, with its skip connections, is significantly more robust to high sparsity than VGG-16. While ResNet maintained over 91% accuracy at 60% random sparsity, VGG-16 collapsed to random guessing (10%). Furthermore, the retraining process exhibits a "top-down" stabilization, where layers closest to the output stabilize first and guide the adaptation of earlier layers.
Conclusion
This investigation demonstrates that unstructured pruning is an evolutionary process rather than simple compression. The resulting models develop fundamentally different internal representations while maintaining performance. The finding that random sparse masks can match engineered ones challenges current intuitions about the necessity of complex pruning pipelines. Future work should explore the unique role of skip connections in enabling sparse training and whether these evolutionary insights can inform the design of inherently sparse architectures.
Summary updated