Traditional neural network pruning relies on a simple heuristic: if a weight is small, it must be unimportant. By removing filters with the smallest $L_1$ or $L_2$ norms, researchers have successfully slimmed down models for years. However, this "magnitude-only" approach has a blind spot. It ignores the relationships between filters. If a layer contains two filters with very large weights that perform the exact same mathematical operation, magnitude-based pruning will keep both, wasting half the computation on redundant information.
A growing body of research argues that structural redundancy—the degree to which filters or neurons overlap in function—is a more effective metric for compression than individual weight magnitude. By identifying and removing correlated components, these methods ensure that every remaining filter contributes a unique feature to the network's "visual vocabulary."
The Intuition of Functional Overlap
The core idea is to treat a convolutional layer not as a collection of independent weights, but as a population of feature detectors. Arshdeep Singh and Mark D Plumbley demonstrate that filters often learn nearly identical signatures. In their work on Acoustic Scene Classification, they show that calculating the cosine distance between filters allows us to identify these duplicates. If two filters are highly correlated, one can be removed with almost no loss in accuracy, even if both filters have high magnitudes.
This concept extends beyond simple identity. Shaowu Chen introduces the Weighted Hybrid Criterion (WHC), which uses absolute cosine similarity to identify both identical and "antiphase" (perfectly inverted) filters. Since an inverted filter provides the same information as its original counterpart (just with a flipped sign), it is functionally redundant and a prime candidate for pruning.
From Weights to Feature Maps
While comparing static weights is computationally cheap, it doesn't always capture how a network behaves in practice. Many researchers advocate for activation-based analysis, looking at the "feature maps" produced when the network processes data.
- Rank-based Importance: Mingbao Lin and colleagues proposed HRank, which measures the mathematical rank of feature maps. The intuition is that a filter producing a low-rank feature map is only capturing simple, redundant structures that other filters likely already cover. High-rank maps, conversely, contain more unique information.
- Correlation of Responses: Xavier Suau and Luca Zappella developed Principal Filter Analysis (PFA). By calculating the Pearson correlation between filter activations across a dataset, PFA identifies filters that "fire" in the same patterns. This allows for "filter distillation," where the network is compressed by keeping only the most representative, decorrelated signals.
- Quantified Similarity: Long Huang and others treat feature maps as 3D tensors, using image similarity metrics to prune filters that produce nearly identical outputs (QSFM).
Clustering the Feature Space
If the goal is to keep only unique features, the problem of pruning becomes a problem of clustering. Instead of evaluating filters one by one, we can group them into "functional families."
In the CUP (Cluster Pruning) framework, filters are grouped using agglomerative hierarchical clustering based on their weight similarity. Only one "representative" filter is kept from each cluster. Similarly, Lianqiang Li uses k-means++ to group filters that extract similar features, significantly reducing parameters in models like LeNet-5.
Zi Wang takes this a step further by modeling the entire layer as an undirected graph. In this graph, filters are nodes, and edges represent high similarity. By analyzing the connectivity and "structural redundancy" of this graph, the system can identify which layers are over-capacitated and prune them more aggressively.
Diversity as a Regularizer
If redundancy is the enemy of efficiency, then diversity is the solution. Some researchers have moved from post-hoc pruning to "decorrelation regularization" during training. Wengang Zhou showed that by explicitly penalizing correlation between filters during the learning process, networks naturally develop higher levels of "structured sparsity," making them much easier to compress later.
Zelda Mariet introduced Divnet, which uses Determinantal Point Processes (DPPs)—a probabilistic framework designed to model diversity. By selecting a subset of neurons that are "most diverse" according to the DPP, the network can maintain high performance even after 90% of a layer is removed, because the remaining 10% are guaranteed to be non-redundant.
Why It Works: The Information Bottleneck
The success of these methods can be explained through the Information Bottleneck principle. As Song Guo explores in the APIB framework, an ideal network should maximize the information it carries about the target label while minimizing the total information it stores (the "bottleneck").
Redundant filters represent "mutual information" between channels that doesn't add new predictive power. By using tools like the Hilbert-Schmidt Independence Criterion (HSIC), researchers can statistically prove which channels are redundant. This shifts pruning from a heuristic "guessing game" into a rigorous optimization of the network's information density.