To Ensemble or Not Ensemble: When Does End-to-End Training Fail?

Andrew M. Webb, Charles Reynolds, Wenlin Chen, Henry W. J. Reeve, Dan-Andrei Iliescu, Mikel Luján, Gavin Brown

TL;DR

This paper investigates when End-to-End (E2E) training of neural network ensembles fails, a crucial question as E2E becomes more prevalent. They introduce a novel "Joint Training" loss that smoothly interpolates between independent ensemble training and full E2E training, revealing that over-parameterized networks are prone to E2E failure, while simpler networks benefit from it.

What Problem Does This Paper Address?

Deep learning architectures are increasingly trained using End-to-End (E2E) methods, where all components are optimized simultaneously using a single loss function. This trend extends to ensembles of neural networks, which are known for their strong performance. However, there's a debate: does E2E training of ensembles hinder or help performance? Some studies suggest it can be detrimental to generalization accuracy, while others show benefits. This paper aims to resolve this by systematically studying the failure cases of E2E ensemble training and understanding the underlying dynamics, particularly concerning the capacity of individual networks within the ensemble.

Methods

The core of the paper's methodology is the introduction of a Joint Training (JT) loss function. This loss is a convex combination of two extremes:

  1. Independent Training: Each network in the ensemble is trained with its own loss function, and their outputs are combined later (e.g., averaging logits).
  2. End-to-End (E2E) Training: All networks are treated as parts of a single, larger network and trained with a shared loss.

The JT loss, parameterized by $\lambda$, smoothly interpolates between these two. When $\lambda=0$, it's independent training; when $\lambda=1$, it's E2E training. By varying $\lambda$, the authors can observe how the training dynamics and performance change. They also analyze the Hessian of the loss function to understand the optimization landscape.

Datasets

  • Fashion-MNIST: Used for experiments with MLPs and smaller CNNs.
  • CIFAR-100: Used for experiments with DenseNets and larger CNNs.

Key Insights and Ideas

  • Over-parameterization is Key: The paper's most significant finding is that E2E ensemble training tends to fail when individual network members are significantly over-parameterized. In such cases, independent training or a hybrid approach performs better.
  • "Model Dominance" Effect: For over-parameterized ensembles trained E2E, a phenomenon called "model dominance" emerges. One network in the ensemble starts to perform exceptionally well, effectively "taking over" and leading to the poor performance of other ensemble members. This is visualized by strong filters in only a few network modules.
  • The "Sweet Spot" of Joint Training: The optimal performance often lies between pure independent training and pure E2E training. A slight deviation from E2E ($\lambda \approx 0.99$) can yield high accuracy while significantly improving robustness.
  • Robustness vs. Brittle Ensembles: Independent training naturally leads to robust ensembles where removing some members doesn't drastically hurt performance. E2E training, while potentially achieving higher accuracy, creates "brittle" ensembles that are highly sensitive to the removal of any member.
  • Dropout Connection: The robustness observed in slightly deviating from E2E training shares conceptual similarities with the motivations behind Dropout, suggesting a potential link between ensemble diversity and regularization techniques.

Key Results

  • MLPs (Fixed Parameter Budget): For a fixed number of parameters, a single large MLP (1x1024H) outperformed an ensemble of small MLPs (256x4H) trained independently. However, E2E training of the small ensemble came close to the large network's performance, with Joint Training ($\lambda=0.95$) achieving the best result among the ensemble configurations.
  • DenseNets (High Capacity): For over-parameterized DenseNets, E2E training ($\lambda=1$) was consistently sub-optimal. Independent training ($\lambda=0$) achieved the lowest test error in most configurations, indicating that for high-capacity models, E2E is not beneficial.
  • Small ConvNets (Intermediate Capacity): As network complexity increased, the optimal $\lambda$ value shifted away from E2E training. For the most complex networks (1.2M parameters), the optimal point lay clearly between independent and E2E training.
  • Robustness: Independent training ( $\lambda=0$) yielded highly robust ensembles. Joint Training with $\lambda \approx 0.99$ achieved comparable accuracy to E2E ($\lambda=1$) but maintained accuracy even when over 50% of ensemble members were dropped.

Limitations and Future Work

  • Focus on Classification: While the framework is generalizable, the experiments were primarily conducted on classification tasks.
  • Ill-conditioning: The paper notes that as $\lambda \to 1$, the condition number of the Hessian tends to infinity, suggesting potential optimization issues for E2E training, especially with first-order methods. This warrants further investigation with second-order methods.
  • Multi-Branch Architectures: The findings raise questions about whether all multi-branch architectures should be trained E2E, or if individual branch training might be better, especially for over-parameterized branches.
  • Dropout Connection: The observed similarity between ensemble robustness and Dropout's motivations suggests a promising avenue for future research.
  • Generalization of "Model Dominance": Further work is needed to fully understand the conditions under which "model dominance" occurs and its implications beyond classification.

Summary updated

Abstract

End-to-End training (E2E) is becoming more and more popular to train complex Deep Network architectures. An interesting question is whether this trend will continue—are there any clear failure cases for E2E training? We study this question in depth, for the specific case of E2E training an ensemble of networks. Our strategy is to blend the gradient smoothly in between two extremes: from independent training of the networks, up to to full E2E training. We find clear failure cases, where overparameterized models cannot be trained E2E. A surprising result is that the optimum can sometimes lie in between the two, neither an ensemble or an E2E system. The work also uncovers links to Dropout, and raises questions around the nature of ensemble diversity and multi-branch networks.
BibTeX
@inproceedings{webb2020ensemble,
  author = {Andrew M. Webb and Charles Reynolds and Wenlin Chen and Henry W. J. Reeve and Dan-Andrei Iliescu and Mikel Luján and Gavin Brown},
  title = {{To Ensemble or Not Ensemble: When Does End-to-End Training Fail?}},
  booktitle = {ECML/PKDD},
  year = {2020},
  url = {https://openreview.net/forum?id=AX17vlpQLju},
}

Loading related content...

Figure