Robust Model Compression Using Deep Hypotheses

Omri Armstrong, Ran Gilad-Bachrach

Abstract

Large-scale machine learning models, while accurate, often lack interpretability and require substantial computational resources. This paper introduces CREMBO (Compact Robust Estimated Median Belief Optimization), a novel, model-agnostic framework for robust model compression. By utilizing "predicate depth" from robust statistics, CREMBO identifies a "median" student hypothesis that maximizes agreement with the teacher model's predictions. Our evaluation across diverse datasets shows that CREMBO consistently outperforms standard Knowledge Distillation (KD) in both accuracy and resilience to data perturbations.

Introduction

The deployment of large Deep Neural Networks (DNNs) and ensemble models is often limited by high inference costs and a lack of transparency. Model compression aims to distill the knowledge of a large "teacher" model into a smaller, more efficient "student" model. However, traditional methods like Knowledge Distillation (KD) can be sensitive to noise and are often specific to certain architectures. This research addresses the need for a robust, flexible compression scheme that can bridge different model classes—for instance, compressing a Random Forest into a shallow Decision Tree.

Methodology: The Median Hypothesis

The core innovation of this work is the application of Robust Statistics to the hypothesis space, pivoting away from standard loss minimization.

Predicate Depth and the Median

In traditional statistics, "depth" indicates how central a point is within a dataset. The authors extend this to functions: a "deep" hypothesis is one that aligns with the majority of other potential hypotheses consistent with the teacher's output. The median hypothesis is defined as the deepest one, offering a theoretically grounded "breakdown point"—a measure of how much input perturbation is required to change the output. By maximizing depth, the framework inherently maximizes robustness.

The CREMBO Framework

The authors propose two primary algorithms:

  1. MEMO (Multiclass Empirical Median Optimization): A core algorithm that uses binary search to find the maximum depth a student can achieve while remaining consistent with the teacher's "multi-label" constraints.
  2. CREMBO: An extension of MEMO that introduces $\delta$-insensitive depth. This allows the model to ignore a small percentage of outliers, acting as a "trimmed estimator" to improve generalization on noisy real-world data.

Experimental Evaluation

The framework was evaluated in two distinct scenarios:

  • Interpretable Compression: Compressing Random Forests and DNNs into depth-4 Decision Trees using five UCI datasets (Dermatology, Heart, Arrhythmia, Breast Cancer, and Iris).
  • DNN-to-DNN Compression: Compressing ResNet18 and VGG16 architectures into LeNet-5 and MobileNetV2 using the CIFAR-10 image dataset.

Key Results

Accuracy and Interpretability

In the UCI experiments, the median tree (MED) produced by CREMBO significantly outperformed standard student models. The performance gain is particularly notable in complex datasets like Arrhythmia, where the MED accuracy reached 54.79% compared to the baseline student tree's 7.7%.

Accuracy and win rate for Random Forest to Decision Tree compression across UCI datasets.

Comparison with Knowledge Distillation

On the CIFAR-10 image classification task, CREMBO consistently surpassed standard KD. For example, when compressing a VGG16 teacher into a LeNet-5 student, CREMBO achieved an accuracy of 72.43%, whereas KD reached only 70.49%.

CIFAR-10 compression results comparing CREMBO with standard Knowledge Distillation (KD).

Robustness through Agreement

Beyond raw accuracy, the authors measured "agreement"—the consistency of predictions across models trained on different data subsets. CREMBO models demonstrated 10-40% higher agreement than benchmarks, validating the theoretical link between predicate depth and robustness.

Conclusion and Future Work

CREMBO provides a robust, model-agnostic alternative to traditional model compression. By framing compression as a search for the median hypothesis, it achieves high accuracy and stability. Future research may focus on improving the calibration of teacher models through techniques like temperature scaling to further refine the "depth" calculation and extending the framework to more complex student model classes.

Summary updated

Abstract

Machine Learning models should ideally be compact and robust. Compactness provides efficiency and comprehensibility whereas robustness provides resilience. Both topics have been studied in recent years but in isolation. Here we present a robust model compression scheme which is independent of model types: it can compress ensembles, neural networks and other types of models into diverse types of small models. The main building block is the notion of depth derived from robust statistics. Originally, depth was introduced as a measure of the centrality of a point in a sample such that the median is the deepest point. This concept was extended to classification functions which makes it possible to define the depth of a hypothesis and the median hypothesis. Algorithms have been suggested to approximate the median but they have been limited to binary classification. In this study, we present a new algorithm, the Multiclass Empirical Median Optimization (MEMO) algorithm that finds a deep hypothesis in multi-class tasks, and prove its correctness. This leads to our Compact Robust Estimated Median Belief Optimization (CREMBO) algorithm for robust model compression. We demonstrate the success of this algorithm empirically by compressing neural networks and random forests into small decision trees, which are interpretable models, and show that they are more accurate and robust than other comparable methods. In addition, our empirical study shows that our method outperforms Knowledge Distillation on DNN to DNN compression.
BibTeX
@inproceedings{armstrong2021robust,
  author = {Omri Armstrong and Ran Gilad-Bachrach},
  title = {{Robust Model Compression Using Deep Hypotheses}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i8.16827},
}
Figure