Abstract
Deep Neural Networks (DNNs) are often over-parameterized, leading to significant computational waste during deployment. While model pruning and Neural Architecture Search (NAS) are standard optimization techniques, they lack a formal metric to quantify structural redundancy. We introduce REDTEST, a framework centered on the Model Structural Redundancy Score (MSRS). By leveraging Centered Kernel Alignment (CKA) to compare intermediate representations, MSRS identifies layers that perform nearly identical functions. Our experiments demonstrate that redundancy-aware pruning can reduce FLOPs by over 50% with negligible impact on accuracy, providing a definitive stopping criterion for model optimization.
Introduction
As DNNs scale, they become increasingly expensive to train and deploy, particularly on resource-constrained devices. Current metrics like parameter count, FLOPs, and latency measure complexity but fail to identify if a model is "over-engineered." For example, two layers might produce nearly identical feature maps, rendering one of them redundant. Without a quantitative measure of this redundancy, developers cannot easily determine how much a model can be compressed before performance degrades.
Methods: The REDTEST Framework
The core of REDTEST is the Model Structural Redundancy Score (MSRS), which quantifies "wasted space" within an architecture through a multi-step process:
- Measuring Layer Similarity: The framework uses Centered Kernel Alignment (CKA) to compare Intermediate Representations (IRs) across layers. CKA is robust to different layer shapes, allowing for a consistent comparison of feature maps throughout the network.
- Scoring Redundancy: Rather than a simple average of similarities, MSRS employs a scaled hyperbolic tangent function. This weights high-similarity pairs more heavily, ensuring that nearly identical layers contribute significantly more to the final redundancy score.
- Redundancy-aware Optimization: This metric is integrated into two key workflows:
- NAS: The search algorithm penalizes architectures with high MSRS, favoring efficiency.
- Layer Pruning: The framework identifies adjacent layers with high similarity, removes the redundant "later" layer, and fine-tunes the resulting compact model.
Key Insights
- Redundancy vs. Complexity: Deeper models are often just more redundant. For instance, ResNet110 and ResNet56 achieve similar accuracy on CIFAR-10, but ResNet110 exhibits a much higher MSRS, indicating many layers are effectively "idling."
- The "Triangular" Pattern: In deep models, similarity matrices often reveal a triangular pattern where successive layers perform increasingly incremental refinements. This visual cue serves as a clear signal for potential pruning.
- MSRS as a Progress Bar: MSRS provides a clear stopping criterion. A high score suggests that pruning or a more efficient NAS search will likely yield high rewards without sacrificing accuracy.
Experimental Results
REDTEST was evaluated across standard benchmarks including CIFAR-10/100 and ImageNet.
- Superior Pruning: On ResNet56 (CIFAR-10), REDTEST removed 52.6% of FLOPs and 45% of parameters while slightly increasing top-1 accuracy from 93.11% to 93.40%. This outperformed several state-of-the-art pruning methods.
- Efficient NAS: Redundancy-aware NAS found models with significantly lower structural redundancy (30.37 vs 34.84 MSRS) while maintaining higher accuracy than standard latency-aware models.
- Scalability: The metric remains effective for massive architectures like ResNet1202, where redundancy scores scale proportionally with depth, confirming its utility for large-scale model analysis.
Conclusion
REDTEST and the MSRS metric provide a much-needed quantitative lens for understanding DNN efficiency. By identifying functional overlap between layers, the framework enables more aggressive and effective model compression. Future work will explore applying MSRS to Transformer architectures and integrating redundancy penalties directly into the training phase to prevent "bloat" from the outset.
Summary updated