Deep convolutional neural networks (CNNs) are notoriously over-parameterized. To deploy these models on resource-constrained edge hardware or mobile devices, we must compress them. Historically, the dominant paradigm for this has been magnitude-based pruning: calculating the norm of a filter's weights and deleting the smallest ones under the assumption that smaller weights contribute less to the final output.
However, this heuristic has a fundamental flaw. Imagine a convolutional layer containing two filters that are completely identical, but both happen to have very large weight magnitudes. A magnitude-based pruner will dutifully keep both, wasting computation on a duplicate operation. Conversely, a filter might have small weights but extract a highly unique, irreplaceable feature; magnitude pruning would discard it.
To solve this, a powerful paradigm shifts the focus from magnitude to similarity. By identifying and removing redundant filters that perform duplicate functions, similarity-based pruning maximizes feature diversity and preserves the network's expressive power with a fraction of the computational cost.
Measuring Redundancy in Weight Space
How do we mathematically determine if two filters are "doing the same thing"? The most direct approach is to analyze the static weights of the filters themselves. By treating each filter as a vector or a point in a high-dimensional space, we can use geometric and statistical relationships to uncover redundancy.
If we look at the geometric distribution of weights within a layer, filters that cluster closely together are functionally redundant. For instance, He et al. utilize Minkowski distance to measure the spatial proximity of filter weights, demonstrating that geometric similarity is a far more reliable indicator of redundancy than weight norms.
A prominent manifestation of this geometric intuition is Filter Pruning via Geometric Median (FPGM), introduced by Liu et al.. Instead of searching for the smallest filters, the authors calculate the geometric median of all filters within a layer. The filters lying closest to this central tendency are the most redundant because their representational contributions are already captured by the surrounding ensemble.
Magnitude Pruning: [Small] ✗ [Large Unique] ✓ [Large Duplicate A] ✓ [Large Duplicate B] ✓
Similarity Pruning: [Small] ✓ [Large Unique] ✓ [Large Duplicate A] ✓ [Large Duplicate B] ✗
Other weight-space approaches leverage statistical correlation or graph theory:
- Correlation Matrices: Wang et al. propose calculating the Pearson correlation coefficient between weight vectors to explicitly target duplicate functions, prioritizing information diversity over individual weight magnitudes.
- Graph Connectivity: Wang et al. model a convolutional layer as an undirected graph where edges connect filters with high weight similarity. Analyzing the connectivity of this graph reveals clusters of structural redundancy across the layer, allowing the network to be trimmed systematically.
Measuring Redundancy in Activation Space
While static weight analysis is computationally efficient, filters with different weights can still produce highly correlated outputs depending on the input data distribution. To capture this dynamic behavior, researchers look at activation-based redundancy.
Instead of analyzing the weights, we analyze the output feature maps generated by the filters. In Principal Filter Analysis (PFA), developed by Suau et al., the statistical correlation between output responses is measured. By performing eigenvalue decomposition on the covariance matrix of these activations, the authors quantify the unique information contribution of each filter, systematically pruning those that generate overlapping feature maps.
Similarly, Singh utilizes the cosine distance between filter activation signatures to eliminate informational overlap. This passive pruning strategy ensures feature diversity in the pruned model, addressing the core limitation of norm-based methods that retain multiple similar filters simply because they are large. To scale this up without massive data requirements, Singh et al. later introduced a Nystrom approximation to estimate the similarity matrix from only a few columns, drastically accelerating the compression process.
Another elegant lens on activation redundancy is rank analysis. Ma et al. evaluate filter importance by calculating the mathematical rank of their generated feature maps. Low-rank maps indicate that a filter's output contains little to no meaningful or unique feature information, making it a prime candidate for removal without requiring exhaustive layer-wise sensitivity testing.
Preserving the Geometry of Feature Space
When we prune filters, we risk collapsing the underlying manifold of the feature space, which can severely degrade model performance on downstream tasks. To prevent this, pruning algorithms must be aware of the local neighborhood structure.
Global Collapse vs. Local Preservation
Global Pruning: [ Cluster A ] --> [ Truncated A ] (Unique sub-features lost)
Local Pruning: [ Cluster A ] --> [ Compact A ] (Geometric structure intact)
Wang et al. introduced Progressive Local Filter Pruning (PLFP) to address this challenge. Instead of evaluating a filter's global importance across the entire layer, PLFP examines its relationship with its $k$-nearest neighbors in the feature space. If a filter is highly similar to its immediate neighbors, it is considered replaceable. This localized approach ensures that unique, isolated feature clusters are preserved, preventing the global feature space from collapsing—a property that is especially critical for high-precision tasks like image retrieval.
Key Foundational Work
Deep Boosting: Joint Feature Selection and Analysis Dictionary Learning in Hierarchy (2015) — contributed 2 concept elements to this direction.
Constructing Hidden Units Using Examples and Queries (1990) — contributed 1 concept elements to this direction.
Learning Compact Convolutional Neural Networks with Nested Dropout (2015) — contributed 1 concept elements to this direction.