Discussion of Ensemble Learning under the Era of Deep Learning

Yongquan Yang, Haijun LvarXiv 2021 · DOI · arXiv · OpenReview

Abstract

Due to the dominant position of deep learning (mostly deep neural networks) in various artificial intelligence applications, recently, ensemble learning based on deep neural networks (ensemble deep learning) has shown significant performances in improving the generalization of learning system. However, since modern deep neural networks usually have millions to billions of parameters, the time and space overheads for training multiple base deep learners and testing with the ensemble deep learner are far greater than that of traditional ensemble learning. Though several algorithms of fast ensemble deep learning have been proposed to promote the deployment of ensemble deep learning in some applications, further advances still need to be made for many applications in specific fields, where the developing time and computing resources are usually restricted or the data to be processed is of large dimensionality. An urgent problem needs to be solved is how to take the significant advantages of ensemble deep learning while reduce the required expenses so that many more applications in specific fields can benefit from it. For the alleviation of this problem, it is essential to know about how ensemble learning has developed under the era of deep learning. Thus, in this article, we present fundamental discussions focusing on data analyses of published works, methodologies, recent advances and unattainability of traditional ensemble learning and ensemble deep learning. We hope this article will be helpful to realize the intrinsic problems and technical challenges faced by future developments of ensemble learning under the era of deep learning.

The Problem: The Computational Barrier

Ensemble learning (EL) has been a cornerstone of machine learning since the 1990s, based on a simple premise: a committee of experts is usually smarter than a single individual. However, as the field has transitioned from "Traditional Ensemble Learning" (TEL) using shallow models to "Ensemble Deep Learning" (EDL) using Deep Neural Networks (DNNs), a massive computational wall has emerged.

Modern DNNs often contain millions or billions of parameters. While training a single model is resource-intensive, training an ensemble of ten or more is often prohibitively expensive for researchers without massive computing budgets. This creates a significant "efficiency gap" in the research landscape. While publications in general ensemble learning have grown exponentially, EDL research lags behind due to these astronomical overheads in time and hardware.

Comparison of publication trends between traditional ensemble learning and ensemble deep learning, highlighting the research lag in EDL.

Methodological Evolution

The evolution of ensemble methods can be categorized into three distinct stages, moving from manual feature engineering to automated, efficient deep learning architectures.

1. Traditional Ensemble Learning (TEL)

TEL relies on hand-crafted features and shallow learners. Key techniques include Bagging (training on different data subsets), Boosting (sequentially focusing on difficult samples), and Stacking (using a meta-model to combine predictions).

2. Usual Ensemble Deep Learning (UEDL)

This represents a "brute force" integration where deep learning replaces traditional components. The paper identifies three primary patterns:

  • Pattern A: DL is used solely for feature extraction.
  • Pattern B: Shallow base learners are replaced with DNNs.
  • Pattern C: A DNN acts as the "meta-learner" to integrate outputs from other models.
General framework for generating and integrating base learners in ensemble systems, showing both heterogeneous and homogeneous approaches.

3. Fast Ensemble Deep Learning (FEDL)

FEDL is the modern frontier, aiming to exploit the geometry of the "loss landscape" to reduce costs. Instead of training multiple models from scratch, FEDL methods like Snapshot Ensembling use cyclic learning rates to capture multiple local minima during a single training run. Stochastic Weight Averaging (SWA) takes this further by averaging model weights rather than outputs, resulting in a single high-performing model with zero extra inference cost.

  • The "Hand-Crafted" Bottleneck: TEL has hit a plateau because hand-crafted features lack the expressive power of the deep features automatically learned by DNNs.
  • Model Space vs. Parameter Space: A critical distinction in the paper is ensembling in model space (averaging outputs of multiple models) versus parameter space (combining weights). Parameter space ensembling is the key to making EDL practical for real-time applications like robotics.
  • Pattern Dominance: Meta-analysis shows that "Pattern B" (replacing traditional learners with deep ones) is currently the most common approach in recent literature, appearing in the majority of reviewed studies.

Future Directions

The authors conclude that the definition of FEDL requires more rigorous, unified framing. Future research must move beyond simply porting 1990s ensemble tricks into the deep learning era. Instead, we must focus on "intrinsically" reducing expenses by better understanding the inherent characteristics of deep learning, such as loss surface geometry and sub-parameter optimization, to find optimal sets of diverse learners without redundant computation.

Summary updated

Abstract

Due to the dominant position of deep learning (mostly deep neural networks) in various artificial intelligence applications, recently, ensemble learning based on deep neural networks (ensemble deep learning) has shown significant performances in improving the generalization of learning system. However, since modern deep neural networks usually have millions to billions of parameters, the time and space overheads for training multiple base deep learners and testing with the ensemble deep learner are far greater than that of traditional ensemble learning. Though several algorithms of fast ensemble deep learning have been proposed to promote the deployment of ensemble deep learning in some applications, further advances still need to be made for many applications in specific fields, where the developing time and computing resources are usually restricted or the data to be processed is of large dimensionality. An urgent problem needs to be solved is how to take the significant advantages of ensemble deep learning while reduce the required expenses so that many more applications in specific fields can benefit from it. For the alleviation of this problem, it is essential to know about how ensemble learning has developed under the era of deep learning. Thus, in this article, we present fundamental discussions focusing on data analyses of published works, methodologies, recent advances and unattainability of traditional ensemble learning and ensemble deep learning. We hope this article will be helpful to realize the intrinsic problems and technical challenges faced by future developments of ensemble learning under the era of deep learning.
BibTeX
@misc{yang2021discussion,
  author = {Yongquan Yang and Haijun Lv},
  title = {{Discussion of Ensemble Learning under the Era of Deep Learning}},
  year = {2021},
  eprint = {2101.08387},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.08387},
}
Figure