Neural Network Pruning Denoises the Features and Makes Local Connectivity Emerge in Visual Tasks

Franco Pellegrini, Giulio BiroliICML 2022 · OpenReview

TL;DR

Neural network pruning is often viewed simply as a way to compress models, but this paper reveals it as a powerful tool for architectural discovery. By iteratively pruning a standard fully connected network (FCN) trained on visual tasks, the authors show that convolutional-like structures—such as local connectivity and Gabor-like filters—emerge naturally from the data, effectively "denoising" the network's learned features.

The Problem: Hard-Coded vs. Emergent Inductive Biases

Modern deep learning relies heavily on "hand-designed" inductive biases. For example, Convolutional Neural Networks (CNNs) are built on the human-derived intuition that images have local structure and translational symmetry. While highly effective, these biases are hard-coded into the architecture before training even begins.

The authors ask a fundamental question: Can these architectural features emerge automatically from a general-purpose model? Specifically, they investigate the "inductive bias" that pruning imprints on a network. They want to know if pruning a "blank slate" model (like an FCN) can recover the structural efficiencies we usually program by hand, and what conditions regarding data size and task type are necessary for this to happen.

Methods and Datasets

The researchers started with a simple FCN (typically 3 hidden layers with 1024 nodes each) and trained it on image classification tasks. They then applied Iterative Magnitude Pruning (IMP) with weight rewinding—the core procedure behind the "Lottery Ticket Hypothesis."

The process involves:

  1. Training the dense network.
  2. Pruning the smallest weights (e.g., 30% per iteration).
  3. Rewinding the remaining weights to their values from early in training.
  4. Retraining and repeating.

To analyze the results, they developed metrics for Node Connectivity (how many inputs a hidden node keeps) and Local Distances (the spatial distribution of those inputs). They compared IMP against one-shot pruning, saliency-based pruning (SNIP), and data-free pruning (SynFlow).

The experiments were primarily conducted on ImageNet32 (a downsampled version of ImageNet with 1.3M images, 1000 classes, 32x32 resolution) and verified on ImageNet64. To test the role of the "task," they also used modified task sets, grouping classes into semantic macro-classes (e.g., "dogs") versus random macro-classes, and reduced datasets to study the impact of data volume.

Key Insights

  • Pruning as a Denoiser: The authors argue that the first round of training suggests potential features, but they are buried in noise. Iterative pruning acts as a filter, "sifting out" the noise and leaving behind structured, meaningful connectivity.
  • Spontaneous Locality: Even though the FCN has no inherent concept of "neighboring pixels," the pruned sub-networks spontaneously developed local receptive fields. Nodes chose to keep connections to pixels that were spatially close to each other in the input image.
  • The "Meaning" Requirement: Structure only emerges when the task is "meaningful" (semantic). When the network was forced to learn random labels (memorization), the pruned masks remained featureless and non-local. This suggests locality is a tool the network discovers specifically to achieve generalization.
  • Data-Driven Evolution: The emergence of structure is a function of the Signal-to-Noise Ratio (SNR). If the dataset is too small, the "signal" of the true features is drowned out by the "noise" of random weight fluctuations, and pruning fails to find structured sub-networks.

Key Results

Performance Boost

Counterintuitively, pruned FCNs actually outperformed their dense counterparts. On ImageNet32, a pruned FCN with only ~1% of its weights remaining achieved over 12% accuracy, compared to less than 9% for the original dense network.

Validation accuracy peaks at extreme sparsity levels, outperforming the dense network.

Gabor-like Filters and IMP Superiority

Visualizing the first-layer masks revealed the emergence of edge detectors and color patches—structures remarkably similar to Gabor filters and the early layers of hand-designed CNNs. Furthermore, Iterative Magnitude Pruning (IMP) was significantly more effective at "distilling" these structures than one-shot or data-free methods, which produced much noisier masks.

Comparison of pruning methods shows IMP uniquely recovers clean, Gabor-like visual features.

Hierarchy

In deeper layers, the network combined these local first-layer features into larger, more complex receptive fields, mimicking the hierarchical processing of the visual cortex.

Limitations and Future Work

  • Computational Efficiency: While this proves structure can emerge, starting with a massive FCN to prune it down to a CNN-like structure is computationally expensive compared to just using a CNN from the start.
  • Weight Sharing: The study shows the emergence of locality, but not weight sharing (the "C" in CNN). In these pruned FCNs, different nodes might learn the same feature at different locations, but they don't share the same weight values.
  • Beyond Vision: The authors suggest this framework could be used to discover unknown inductive biases in other domains, like audio or complex time-series data, where human intuition hasn't yet provided a "perfect" architecture like the CNN.

Summary updated

BibTeX
@inproceedings{pellegrini2022neural,
  author = {Franco Pellegrini and Giulio Biroli},
  title = {{Neural Network Pruning Denoises the Features and Makes Local Connectivity Emerge in Visual Tasks}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=J8ZJCfPCXGQ},
}
Figure