Abstract
Large-scale machine learning models, while accurate, often lack interpretability and require substantial computational resources. This paper introduces CREMBO (Compact Robust Estimated Median Belief Optimization), a novel, model-agnostic framework for robust model compression. By utilizing "predicate depth" from robust statistics, CREMBO identifies a "median" student hypothesis that maximizes agreement with the teacher model's predictions. Our evaluation across diverse datasets shows that CREMBO consistently outperforms standard Knowledge Distillation (KD) in both accuracy and resilience to data perturbations.
Introduction
The deployment of large Deep Neural Networks (DNNs) and ensemble models is often limited by high inference costs and a lack of transparency. Model compression aims to distill the knowledge of a large "teacher" model into a smaller, more efficient "student" model. However, traditional methods like Knowledge Distillation (KD) can be sensitive to noise and are often specific to certain architectures. This research addresses the need for a robust, flexible compression scheme that can bridge different model classes—for instance, compressing a Random Forest into a shallow Decision Tree.
Methodology: The Median Hypothesis
The core innovation of this work is the application of Robust Statistics to the hypothesis space, pivoting away from standard loss minimization.
Predicate Depth and the Median
In traditional statistics, "depth" indicates how central a point is within a dataset. The authors extend this to functions: a "deep" hypothesis is one that aligns with the majority of other potential hypotheses consistent with the teacher's output. The median hypothesis is defined as the deepest one, offering a theoretically grounded "breakdown point"—a measure of how much input perturbation is required to change the output. By maximizing depth, the framework inherently maximizes robustness.
The CREMBO Framework
The authors propose two primary algorithms:
- MEMO (Multiclass Empirical Median Optimization): A core algorithm that uses binary search to find the maximum depth a student can achieve while remaining consistent with the teacher's "multi-label" constraints.
- CREMBO: An extension of MEMO that introduces $\delta$-insensitive depth. This allows the model to ignore a small percentage of outliers, acting as a "trimmed estimator" to improve generalization on noisy real-world data.
Experimental Evaluation
The framework was evaluated in two distinct scenarios:
- Interpretable Compression: Compressing Random Forests and DNNs into depth-4 Decision Trees using five UCI datasets (Dermatology, Heart, Arrhythmia, Breast Cancer, and Iris).
- DNN-to-DNN Compression: Compressing ResNet18 and VGG16 architectures into LeNet-5 and MobileNetV2 using the CIFAR-10 image dataset.
Key Results
Accuracy and Interpretability
In the UCI experiments, the median tree (MED) produced by CREMBO significantly outperformed standard student models. The performance gain is particularly notable in complex datasets like Arrhythmia, where the MED accuracy reached 54.79% compared to the baseline student tree's 7.7%.
Comparison with Knowledge Distillation
On the CIFAR-10 image classification task, CREMBO consistently surpassed standard KD. For example, when compressing a VGG16 teacher into a LeNet-5 student, CREMBO achieved an accuracy of 72.43%, whereas KD reached only 70.49%.
Robustness through Agreement
Beyond raw accuracy, the authors measured "agreement"—the consistency of predictions across models trained on different data subsets. CREMBO models demonstrated 10-40% higher agreement than benchmarks, validating the theoretical link between predicate depth and robustness.
Conclusion and Future Work
CREMBO provides a robust, model-agnostic alternative to traditional model compression. By framing compression as a search for the median hypothesis, it achieves high accuracy and stability. Future research may focus on improving the calibration of teacher models through techniques like temperature scaling to further refine the "depth" calculation and extending the framework to more complex student model classes.
Summary updated