TL;DR
This paper investigates when End-to-End (E2E) training of neural network ensembles fails, a crucial question as E2E becomes more prevalent. They introduce a novel "Joint Training" loss that smoothly interpolates between independent ensemble training and full E2E training, revealing that over-parameterized networks are prone to E2E failure, while simpler networks benefit from it.
What Problem Does This Paper Address?
Deep learning architectures are increasingly trained using End-to-End (E2E) methods, where all components are optimized simultaneously using a single loss function. This trend extends to ensembles of neural networks, which are known for their strong performance. However, there's a debate: does E2E training of ensembles hinder or help performance? Some studies suggest it can be detrimental to generalization accuracy, while others show benefits. This paper aims to resolve this by systematically studying the failure cases of E2E ensemble training and understanding the underlying dynamics, particularly concerning the capacity of individual networks within the ensemble.
Methods
The core of the paper's methodology is the introduction of a Joint Training (JT) loss function. This loss is a convex combination of two extremes:
- Independent Training: Each network in the ensemble is trained with its own loss function, and their outputs are combined later (e.g., averaging logits).
- End-to-End (E2E) Training: All networks are treated as parts of a single, larger network and trained with a shared loss.
The JT loss, parameterized by $\lambda$, smoothly interpolates between these two. When $\lambda=0$, it's independent training; when $\lambda=1$, it's E2E training. By varying $\lambda$, the authors can observe how the training dynamics and performance change. They also analyze the Hessian of the loss function to understand the optimization landscape.
Datasets
- Fashion-MNIST: Used for experiments with MLPs and smaller CNNs.
- CIFAR-100: Used for experiments with DenseNets and larger CNNs.
Key Insights and Ideas
- Over-parameterization is Key: The paper's most significant finding is that E2E ensemble training tends to fail when individual network members are significantly over-parameterized. In such cases, independent training or a hybrid approach performs better.
- "Model Dominance" Effect: For over-parameterized ensembles trained E2E, a phenomenon called "model dominance" emerges. One network in the ensemble starts to perform exceptionally well, effectively "taking over" and leading to the poor performance of other ensemble members. This is visualized by strong filters in only a few network modules.
- The "Sweet Spot" of Joint Training: The optimal performance often lies between pure independent training and pure E2E training. A slight deviation from E2E ($\lambda \approx 0.99$) can yield high accuracy while significantly improving robustness.
- Robustness vs. Brittle Ensembles: Independent training naturally leads to robust ensembles where removing some members doesn't drastically hurt performance. E2E training, while potentially achieving higher accuracy, creates "brittle" ensembles that are highly sensitive to the removal of any member.
- Dropout Connection: The robustness observed in slightly deviating from E2E training shares conceptual similarities with the motivations behind Dropout, suggesting a potential link between ensemble diversity and regularization techniques.
Key Results
- MLPs (Fixed Parameter Budget): For a fixed number of parameters, a single large MLP (1x1024H) outperformed an ensemble of small MLPs (256x4H) trained independently. However, E2E training of the small ensemble came close to the large network's performance, with Joint Training ($\lambda=0.95$) achieving the best result among the ensemble configurations.
- DenseNets (High Capacity): For over-parameterized DenseNets, E2E training ($\lambda=1$) was consistently sub-optimal. Independent training ($\lambda=0$) achieved the lowest test error in most configurations, indicating that for high-capacity models, E2E is not beneficial.
- Small ConvNets (Intermediate Capacity): As network complexity increased, the optimal $\lambda$ value shifted away from E2E training. For the most complex networks (1.2M parameters), the optimal point lay clearly between independent and E2E training.
- Robustness: Independent training ( $\lambda=0$) yielded highly robust ensembles. Joint Training with $\lambda \approx 0.99$ achieved comparable accuracy to E2E ($\lambda=1$) but maintained accuracy even when over 50% of ensemble members were dropped.
Limitations and Future Work
- Focus on Classification: While the framework is generalizable, the experiments were primarily conducted on classification tasks.
- Ill-conditioning: The paper notes that as $\lambda \to 1$, the condition number of the Hessian tends to infinity, suggesting potential optimization issues for E2E training, especially with first-order methods. This warrants further investigation with second-order methods.
- Multi-Branch Architectures: The findings raise questions about whether all multi-branch architectures should be trained E2E, or if individual branch training might be better, especially for over-parameterized branches.
- Dropout Connection: The observed similarity between ensemble robustness and Dropout's motivations suggests a promising avenue for future research.
- Generalization of "Model Dominance": Further work is needed to fully understand the conditions under which "model dominance" occurs and its implications beyond classification.
Summary updated