Abstract
Ensemble learning combines several individual models to obtain better generalization performance. Currently, deep learning architectures are showing better performance compared to the shallow or traditional models. Deep ensemble learning models combine the advantages of both the deep learning models as well as the ensemble learning such that the final model has better generalization performance. This paper reviews the state-of-art deep ensemble models and hence serves as an extensive summary for the researchers. The ensemble models are broadly categorised into bagging, boosting, stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous/heterogeneous ensemble, decision fusion strategies based deep ensemble models. Applications of deep ensemble models in different domains are also briefly discussed. Finally, we conclude this paper with some potential future research directions.
Introduction
Deep learning (DL) has revolutionized artificial intelligence, yet individual models often remain "brittle." They can be overly sensitive to initial weight settings, prone to overfitting on small datasets, and may get stuck in local optima during training. Traditional ensemble learning solved similar issues for "shallow" models by combining multiple learners to reduce variance and bias. The challenge in the modern era is effectively and efficiently adapting these strategies for deep learning. Ensemble Deep Learning (EDL) addresses this by marrying the high-performance feature extraction of deep neural networks with robust generalization through model combination.
Taxonomy of Ensemble Deep Learning
The field of EDL can be categorized into several distinct strategies based on how models are trained and combined. These range from theoretical foundations to specific fusion strategies and diverse application domains.
Ensemble Strategies
- Classical Strategies: These involve adapting traditional methods like Bagging (training on different data subsets), Boosting (sequential training to correct previous errors), and Stacking (using a meta-learner to combine outputs).
- Implicit vs. Explicit Ensembles: Explicit ensembles train separate models, which can be computationally expensive. Implicit ensembles, such as Dropout, DropConnect, and Stochastic Depth, simulate an ensemble of exponential size within a single training run by randomly deactivating parts of the network.
- General Methods: This includes techniques like Negative Correlation Learning, which explicitly encourages diversity among learners, and the distinction between Homogeneous (same architecture) and Heterogeneous (different architectures) ensembles.
Fusion Strategies
Beyond simple averaging, sophisticated methods are used to combine model outputs. These include Bayes Optimal Classifiers, Majority Voting, Stacked Generalization, and Super Learners, which use cross-validation to determine optimal weights for each base model.
Theoretical Grounding and Diversity
The success of EDL is rooted in the Bias-Variance Decomposition and the Condorcet Jury Theorem, which mathematically proves that if each individual "voter" is better than random, adding more independent voters increases the probability of a correct majority decision. A core requirement is Diversity; an ensemble only works if the models make different mistakes. Diversity can be induced through data subsampling, varying architectures, or employing different loss functions.
Applications and Performance
EDL has demonstrated state-of-the-art results across various benchmarks:
- Healthcare: This is the leading domain for EDL (27% of surveyed literature), including Alzheimer’s diagnosis and COVID-19 detection in medical imaging.
- Computer Vision: Dominant in image classification (22.5% of papers) and facial expression recognition.
- Time-Series & Speech: Effective in electricity load forecasting, stock market prediction, and spoken language understanding.
Meta-analysis shows that Decision Fusion (29.5%) and Boosting (18.2%) are the most common strategies, while Implicit Ensembles are gaining ground due to their computational efficiency. Across nearly every domain, ensemble versions of architectures consistently outperform single-model counterparts.
Challenges and Future Directions
Despite its success, EDL faces several hurdles:
- Computational Overhead: Training multiple deep models is resource-intensive. Future research must focus on "parameter-efficient" ensembles.
- Model Selection: There is no universal standard for deciding which models should be included in an ensemble.
- Unsupervised Learning: Most research focuses on supervised tasks; unsupervised ensembles like Consensus Clustering remain an untapped frontier.
- Interpretability: Improving the explainability of ensemble decisions is vital for high-stakes fields like medicine, where an ensemble can act as a "black box inside a locked room."
Summary updated