TL;DR
This paper introduces a novel "Diversity-Penalizing Ensemble Training" (DPET) method for deep learning. Instead of training independent models and then combining them, DPET trains an ensemble of models simultaneously, penalizing differences in their outputs. This allows individual models to achieve near-ensemble performance while being deployable as single models, offering significant performance gains without increased testing costs.
What Problem Does This Paper Address?
Deep learning models, especially in areas like speech recognition, often benefit from ensemble methods where multiple models are trained and their predictions are averaged. This significantly improves accuracy but comes at a steep computational cost during inference (testing), as all models in the ensemble must be run. While model compression techniques exist to distill ensemble knowledge into a single model, this paper proposes a more integrated approach: achieving model compression during the ensemble training process itself. The goal is to train individual models within an ensemble such that each one performs as well as the full ensemble, thereby eliminating the need for multiple models at test time.
Methods
The core innovation is the "Diversity-Penalizing Ensemble Training" (DPET) procedure. The authors initialize an ensemble of $N$ neural networks with different random parameters. Instead of minimizing the standard cross-entropy loss for each model, DPET adds a penalty term to the objective function. This penalty is the Kullback-Leibler (KL) divergence between an individual model's output distribution and the average output distribution of the entire ensemble.
The new objective function for each model $i$ is: $F_i = \text{CrossEnt}(q, p^i) + \lambda D(p^i || \bar{p})$ where $q$ is the target distribution, $p^i$ is the output distribution of model $i$, $\bar{p}$ is the average output distribution of the ensemble, and $\lambda$ is a hyperparameter controlling the strength of the diversity penalty.
By minimizing this combined objective, each model is encouraged to not only learn the correct labels but also to converge towards the average prediction of the ensemble. This implicitly drives the models to become similar, reducing the variance introduced by random initialization and effectively acting as a regularizer. The authors also detail how the derivatives for this new objective can be efficiently computed, allowing for standard mini-batch stochastic gradient descent (SGD) training. They also propose a schedule for $\lambda$, starting with a small value and increasing it during training, to prioritize learning from data early on and then focusing on ensemble convergence.
Datasets
- IARPA Babel: Used for experiments on limited resource Automatic Speech Recognition (ASR) tasks. This dataset includes limited language packs (10 hours of transcribed speech) for languages like Tamil, Assamese, Bengali, and Zulu.
- TED-LIUM: Used for experiments to demonstrate the generalization performance of DPET across various conditions. This dataset consists of a standard 118-hour training set, a 1.6-hour development set, and a 2.6-hour test set.
Key Insights and Ideas
- Implicit Model Compression: DPET achieves model compression as a byproduct of ensemble training, rather than as a separate post-training step. This is a more efficient approach.
- Regularization Effect: The diversity penalty acts as a powerful regularizer, reducing the variance caused by model initialization and improving generalization, particularly when models have a large number of parameters.
- Ensemble Performance in Single Models: The core idea is that by forcing individual models to emulate the ensemble average, each single model can achieve performance close to that of the full ensemble.
- Reduced Performance Gap: DPET significantly shrinks the performance gap between the ensemble and individual models, making single models much more competitive.
- No Test-Time Overhead: The primary advantage is that after training, any single model from the DPET ensemble can be used for inference, incurring no additional computational cost compared to a standard single-model setup.
Key Results
- IARPA Babel: DPET achieved absolute improvements in Word Error Rate (WER) ranging from 0.9% to 1.6% across four languages, demonstrating its effectiveness on low-resource ASR tasks where overfitting is a concern.
- TED-LIUM:
- DPET consistently improved performance as the number of parameters in the individual DNNs increased, especially when baseline models started to degrade due to overfitting.
- For models with 2.8 million parameters, DPET achieved a 0.6% absolute improvement (3.3% relative) on the test set compared to the baseline.
- Crucially, DPET-trained individual models achieved performance very close to, and sometimes better than, the baseline ensemble. For example, three out of four DPET individual models performed as well as or better than the baseline ensemble (17.4% WER).
- The performance gap between the ensemble and individual models was reduced by DPET (0.1%-0.3%) compared to the baseline (0.4%-0.5%).
- Validation frame accuracy plots showed that DPET helped prevent overfitting, with DPET models achieving higher accuracy during later training iterations.
Limitations and Future Work
- Hyperparameter Tuning: The parameter $\lambda$ and its schedule require empirical tuning. While the paper suggests a linear schedule and typical ranges for $\lambda_{init}$ and $\lambda_{final}$, optimal values might vary.
- Ensemble Size: While larger ensembles generally lead to more significant DPET effects, the paper notes diminishing returns beyond an ensemble size of 4-8, suggesting a practical trade-off between performance and computational cost.
- Future Work: The authors plan to investigate more efficient ways to train such ensembles.
Summary updated