Going Beyond Classification Accuracy Metrics in Model Compression

V. Roshan Joseph, Shoaib Ahmed Siddiqui, Aditya Bhaskara, Ganesh Gopalakrishnan, Saurav Muralidharan, Michael Garland, Sheraz Ahmed, Andreas Dengel

TL;DR

This paper introduces a novel multi-part loss function for model compression that goes beyond just maintaining classification accuracy. By incorporating knowledge distillation techniques, their method significantly reduces prediction mismatches between compressed and original models, leading to improved fairness and attribution alignment without sacrificing accuracy.

What Problem Does This Paper Address?

As edge devices become more prevalent, there's a growing need for efficient, lightweight machine learning models. Model compression techniques aim to shrink model size (e.g., through pruning or quantization) while preserving performance, typically measured by top-1 accuracy. However, these methods often create a "misalignment" with the original, uncompressed model. This misalignment can manifest in various ways, such as disparate impacts on different classes (leading to unfairness) or changes in how the model arrives at its predictions (affecting explainability). The authors aim to address this by developing compression methods that not only maintain accuracy but also ensure semantic alignment with the original model across these crucial metrics.

Methods

The core of the proposed solution is a novel multi-part loss function inspired by knowledge distillation. Instead of solely relying on the standard cross-entropy loss to fine-tune a compressed model, they introduce terms that encourage the compressed model's predictions to align with those of the original, uncompressed "teacher" model. Specifically, they explore three distillation-based loss components:

  • KL Divergence (LKD): Encourages the probability distributions of the compressed and teacher models to be similar.
  • Mean Squared Error (LMSE): Directly minimizes the difference between the logit outputs of the two models.
  • Cross-Entropy with Hard Labels (LCEPred): Uses the teacher model's predicted class as the target for the compressed model's cross-entropy loss.

These loss terms can be combined linearly, and the authors investigate different strategies for weighting them: UNIFORM (equal weights), LEARNABLE (weights optimized via gradient descent), and SOFTADAPT (an adaptive method). They also introduce new metrics to quantify this misalignment, including "Compression Impacted Exemplars" (CIEs) and "Compression Impacted Pixels" (CIPs) for segmentation, as well as metrics for fairness and attribution similarity.

Datasets

The paper evaluates their approach on several standard datasets:

  • CIFAR-10 and CIFAR-100: Used for image classification tasks.
  • ImageNet: Used for image classification tasks.
  • Brain MRI Segmentation dataset: Used for semantic segmentation tasks.

They do not introduce new benchmarks but rather apply their methods to existing ones to demonstrate their general applicability.

Key Insights and Ideas

  • Beyond Top-1 Accuracy: The paper strongly advocates for moving beyond simple accuracy metrics in model compression. They show that models with similar top-1 accuracy can still exhibit significant differences in other critical aspects like fairness and interpretability.
  • Knowledge Distillation as a Unifier: By framing model compression as a knowledge distillation problem, they demonstrate that aligning the compressed model's outputs with the teacher model's outputs naturally improves other semantic metrics.
  • Logit Pairing is Powerful: The LMSE (logit pairing) loss term, in particular, appears to be highly effective in preserving fairness and attribution alignment, often outperforming other combinations.
  • Simple Combinations Work Well: Surprisingly, a simple combination of cross-entropy and MSE loss with uniform weights (UNIFORM (CE, MSE)) often yields the best results across various tasks and architectures.
  • Interconnectedness of Metrics: The authors observe correlations between metrics: reducing prediction mismatches (CIEs) often leads to improved fairness, and better attribution alignment also correlates with reduced CIEs.

Key Results

The proposed label-preservation-aware loss functions achieve significant improvements across various metrics:

  • Reduction in Prediction Mismatches: Up to 4.1x reduction in Compression Impacted Exemplars (CIEs) and up to 5.7x reduction in cases where the reference model was correct (CIE-Us).
  • Preservation of Accuracy: The methods largely retain the top-1 accuracy of the original models, with negligible drops (less than 0.1% in some cases).
  • Improved Fairness: Several of their proposed loss combinations, especially those including logit pairing, significantly preserve fairness across classes compared to baseline methods.
  • Enhanced Attribution Alignment: The methods also show improved Intersection over Union (IoU) scores for attribution maps, indicating that compressed models use similar features for predictions as the original models.
  • Generalizability: The approach demonstrates effectiveness across diverse architectures (ResNet, ResNeXt, DenseNet, U-Net), datasets (CIFAR, ImageNet, Brain MRI), and compression algorithms (Group-Sparsity, REWIND).

Limitations and Future Work

  • Metric Exhaustiveness: The authors acknowledge that the list of metrics they consider is not exhaustive, and other concerns like privacy might require different metrics.
  • Hyperparameter Tuning: While they propose methods for tuning loss weights (LEARNABLE, SOFTADAPT), the optimal strategy can still depend on the specific task and architecture.
  • Future Work: The paper suggests that their framework can serve as a simple "plug-and-play" component for existing compression algorithms. Future work could explore more sophisticated ways to combine loss functions and investigate the impact of their methods on other semantic metrics beyond fairness and attribution. They also hint at the broader environmental benefits of power-efficient ML deployment.

Summary updated

Abstract

With the rise in edge-computing devices, there has been an increasing demand to deploy energy and resource-efficient models. A large body of research has been devoted to developing methods that can reduce the size of the model considerably without affecting the standard metrics such as top-1 accuracy. However, these pruning approaches tend to result in a significant mismatch in other metrics such as fairness across classes and explainability. To combat such misalignment, we propose a novel multi-part loss function inspired by the knowledge-distillation literature. Through extensive experiments, we demonstrate the effectiveness of our approach across different compression algorithms, architectures, tasks as well as datasets. In particular, we obtain up to $4.1\times$ reduction in the number of prediction mismatches between the compressed and reference models, and up to $5.7\times$ in cases where the reference model makes the correct prediction; all while making no changes to the compression algorithm, and minor modifications to the loss function. Furthermore, we demonstrate how inducing simple alignment between the predictions of the models naturally improves the alignment on other metrics including fairness and attributions. Our framework can thus serve as a simple plug-and-play component for compression algorithms in the future.
BibTeX
@misc{joseph2020going,
  author = {V. Roshan Joseph and Shoaib Ahmed Siddiqui and Aditya Bhaskara and Ganesh Gopalakrishnan and Saurav Muralidharan and Michael Garland and Sheraz Ahmed and Andreas Dengel},
  title = {{Going Beyond Classification Accuracy Metrics in Model Compression}},
  year = {2020},
  eprint = {2012.01604},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.01604},
}
Figure