Is Pruning Compression?: Investigating Pruning Via Network Layer Similarity

Cody Blakeney, Yan Yan, Ziliang ZongWACV 2020 · OpenReview

Abstract

Unstructured neural network pruning is an effective technique that can significantly reduce theoretical model size, computation demand and energy consumption of large neural networks without compromising accuracy. However, a number of fundamental questions about pruning are not answered yet. For example, do the pruned neural networks contain the same representations as the original network? Is pruning a compression or evolution process? Does pruning only work on trained neural networks? What is the role and value of the uncovered sparsity structure? In this paper, we strive to answer these questions by analyzing three unstructured pruning methods (magnitude based pruning, post-pruning re-initialization, and random sparse initialization). We conduct extensive experiments using the Singular Vector Canonical Correlation Analysis (SVCCA) tool to study and contrast layer representations of pruned and original ResNet, VGG, and ConvNet models. We have several interesting observations: 1) Pruned neural network models evolve to substantially different representations while still maintaining similar accuracy. 2) Initialized sparse models can achieve reasonably good accuracy compared to well-engineered pruning methods. 3) Sparsity structures discovered by pruning models are not inherently important or useful.

Introduction

Deep neural networks (DNNs) are notoriously oversized, posing significant challenges for deployment on resource-constrained edge devices. While "pruning"—the removal of non-essential weights—is a standard solution to reduce model size, our theoretical understanding of the process remains limited. This research investigates whether pruning acts as a simple compression of existing knowledge or if it forces the model to undergo an evolutionary process, learning entirely new internal representations. Specifically, the study evaluates three assumptions: whether pruned models retain original representations, if dense pre-training is necessary, and if the specific structure of a pruning mask is inherently valuable.

Methods

The study compares three primary pruning strategies to understand their impact on model representation:

  1. Magnitude-Based Pruning: Training a full model and subsequently removing the smallest weights.
  2. Post-Pruning Re-initialization: Extracting the mask from a pruned model, resetting weights to random values, and retraining from scratch.
  3. Random Sparse Initialization: Creating a random mask and training from scratch.

To analyze the internal state of these models, the researchers employed Singular Vector Canonical Correlation Analysis (SVCCA). This tool quantifies the similarity between mathematical representations of layers across different models, providing a window into how "similar" their internal logic remains after pruning.

The experimental pipeline involves generating layer-wise neuron vectors from various pruning methods and analyzing them via SVCCA to compare original and pruned networks.

Experiments were conducted on CIFAR-10 and CIFAR-100 datasets using ResNet-34, VGG-16, and ConvNet architectures.

Results and Analysis

Evolution Over Compression

A key finding is that as pruning becomes more aggressive, the model's internal representations diverge significantly from the original. At 90% sparsity, the median similarity between pruned and original layers often drops below 50%. This suggests that the model does not merely "squeeze" original knowledge but evolves new ways to solve the task with limited parameters.

Similarity distributions for ResNet 34 show a clear decay in representation similarity as sparsity increases from 30% to 90%.

The Role of Sparsity Structure

Contrary to the "Lottery Ticket Hypothesis," the specific pattern of the pruning mask appears less critical in unstructured pruning. On CIFAR-10 with ResNet-34 at 90% sparsity, Magnitude Pruning and a completely Random Sparse model both achieved an identical accuracy of 87.2%. This indicates that the "winning ticket" might be more about the existence of sparsity than the specific structure uncovered by complex algorithms.

Architecture Robustness and Learning Dynamics

The study highlights that ResNet, with its skip connections, is significantly more robust to high sparsity than VGG-16. While ResNet maintained over 91% accuracy at 60% random sparsity, VGG-16 collapsed to random guessing (10%). Furthermore, the retraining process exhibits a "top-down" stabilization, where layers closest to the output stabilize first and guide the adaptation of earlier layers.

Conclusion

This investigation demonstrates that unstructured pruning is an evolutionary process rather than simple compression. The resulting models develop fundamentally different internal representations while maintaining performance. The finding that random sparse masks can match engineered ones challenges current intuitions about the necessity of complex pruning pipelines. Future work should explore the unique role of skip connections in enabling sparse training and whether these evolutionary insights can inform the design of inherently sparse architectures.

Summary updated

Abstract

Unstructured neural network pruning is an effective technique that can significantly reduce theoretical model size, computation demand and energy consumption of large neural networks without compromising accuracy. However, a number of fundamental questions about pruning are not answered yet. For example, do the pruned neural networks contain the same representations as the original network? Is pruning a compression or evolution process? Does pruning only work on trained neural networks? What is the role and value of the uncovered sparsity structure? In this paper, we strive to answer these questions by analyzing three unstructured pruning methods (magnitude based pruning, post-pruning re-initialization, and random sparse initialization). We conduct extensive experiments using the Singular Vector Canonical Correlation Analysis (SVCCA) tool to study and contrast layer representations of pruned and original ResNet, VGG, and ConvNet models. We have several interesting observations: 1) Pruned neural network models evolve to substantially different representations while still maintaining similar accuracy. 2) Initialized sparse models can achieve reasonably good accuracy compared to well-engineered pruning methods. 3) Sparsity structures discovered by pruning models are not inherently important or useful.
BibTeX
@inproceedings{blakeney2020is,
  author = {Cody Blakeney and Yan Yan and Ziliang Zong},
  title = {{Is Pruning Compression?: Investigating Pruning Via Network Layer Similarity}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=neCJFaSvkp},
}
Figure