Abstract: With the rise in edge-computing devices, there has been an increasing demand to deploy energy and resource-efficient models. A large body of research has been devoted to developing methods that can reduce the size of the model considerably without affecting the standard metrics such as top-1 accuracy. However, these pruning approaches tend to result in a significant mismatch in other metrics such as fairness across classes and explainability. To combat such misalignment, we propose a novel multi-part loss function inspired by the knowledge-distillation literature. Through extensive experiments, we demonstrate the effectiveness of our approach across different compression algorithms, architectures, tasks as well as datasets. In particular, we obtain up to $4.1\times$ reduction in the number of prediction mismatches between the compressed and reference models, and up to $5.7\times$ in cases where the reference model makes the correct prediction; all while making no changes to the compression algorithm, and minor modifications to the loss function. Furthermore, we demonstrate how inducing simple alignment between the predictions of the models naturally improves the alignment on other metrics including fairness and attributions. Our framework can thus serve as a simple plug-and-play component for compression algorithms in the future.
Introduction: The Hidden Cost of Model Compression
As deep learning models migrate to edge devices like smartphones, sensors, and autonomous vehicles, model compression techniques such as pruning have become essential. Traditionally, the success of these techniques is measured by "Top-1 Accuracy." However, accuracy is a "leaky" metric. Two models can achieve identical accuracy scores while exhibiting vastly different behaviors. A compressed model might maintain its overall score but fail disproportionately on specific demographic groups or rely on irrelevant background features for its predictions.
This phenomenon is known as a lack of functional equivalence. To quantify this, we track Compression Impacted Exemplars (CIEs)—individual test images where the compressed model's prediction disagrees with the original "teacher" model. High CIE counts indicate that the compressed model is not merely a smaller version of the original, but a fundamentally different and potentially less reliable entity.
Methodology: A Multi-Part Loss Framework
We propose a "plug-and-play" framework that modifies the fine-tuning stage of model compression. Rather than training the student model solely on ground-truth labels using standard Cross-Entropy, we introduce a multi-part loss function designed to enforce functional alignment:
- Standard Cross-Entropy ($L_{CE}$): Ensures the model learns the ground-truth labels.
- Logit Pairing ($L_{MSE}$): Minimizes the Mean Squared Error between the raw output scores (logits) of the original and compressed models. This forces the student to mimic the teacher's uncertainty and relative class scores.
- Hard Distillation ($L_{CEPred}$): Encourages the compressed model to match the specific class prediction of the original model, even in cases where the original model was incorrect.
To optimize these objectives, we evaluated three weighting strategies: UNIFORM (equal weights), LEARNABLE (parameters optimized during training), and SOFTADAPT (dynamic weighting based on relative loss performance).
Key Results and Insights
Our experiments across datasets like CIFAR-10/100, ImageNet, and Brain MRI segmentation revealed several critical insights:
- Logit Pairing is the MVP: The simplest addition, Logit Pairing ($L_{MSE}$), proved most effective. By mimicking the teacher's logit distribution, the compressed model naturally preserved fairness and interpretability.
- Massive Mismatch Reduction: On CIFAR-100, our method achieved a 4.1× reduction in total prediction mismatches (CIEs) and a 5.7× reduction in "unforced errors" (CIE-U), where the compressed model fails on an image the original model handled correctly.
- Holistic Alignment: By enforcing functional equivalence, we observed that secondary metrics like fairness and saliency map consistency (measured via Intersection over Union of attributions) improved as a byproduct. The compressed model "looks" at the same parts of an image as the original model to reach its conclusion.
Conclusion
This research demonstrates that functional alignment is a viable and necessary goal for model compression. By shifting focus from pure accuracy to label-preservation-aware loss functions, we can deploy edge models that are not only smaller but also more reliable, fair, and interpretable. Future work will explore the interaction of these loss functions with quantization and their impact on privacy-preserving model features.
Summary updated