Ensembles—combining multiple models to make a single prediction—are a cornerstone of reliable machine learning. The current "gold standard" is Deep Ensembles, which simply trains several models from different random starting points. While effective, this method relies entirely on luck to create diverse models.
Researchers have attempted to be more systematic using "particle-based" Bayesian methods, which apply a mathematical "repulsive force" to push models apart during training. However, they face a dilemma:
- Weight Space: Pushing models apart based on their raw parameters (weights) is inefficient. Because neural networks are over-parameterized, two models can have very different weights but behave identically, wasting the diversity effort.
- Function Space: Pushing models apart based on their final outputs (logits) often leads to "underfitting." In many datasets, there is only one "correct" output, so forcing models to have different outputs just makes them wrong.
To solve this, researchers introduce a "Goldilocks" solution: performing inference in the feature space (the activations of the layer just before the final output). This method, called feature-WGD (Wasserstein Gradient Descent), builds more robust neural network ensembles by promoting diversity in intermediate feature representations.
The feature-WGD Approach
The core of feature-WGD relies on three technical innovations:
- Shared Classifier Head: All ensemble members share the exact same final classification layer. This forces their individual "feature extractors" to speak the same semantic language, making it meaningful to compare and diversify their activations.
- Feature-Space Repulsion: During training, a repulsive force is applied to the feature vectors. This encourages each model to capture distinct features of the same input.
- Low-Dimensional Projection: Since feature spaces can be huge (thousands of dimensions), calculating repulsion is computationally expensive and unstable. The authors project these features into a low-dimensional subspace where the data likelihood changes most significantly, focusing the diversity effort where it matters most.
Exploiting "Multi-View" Structure
Real-world objects often have multiple identifying features. For example, a lion can be identified by its mane or its facial structure. Feature-WGD encourages different ensemble members to "specialize" in these different views.
Because members look at different features, the ensemble is less likely to fail if one feature is obscured or corrupted. If a "mane-specialist" is confused by a blurry image, the "face-specialist" can still provide a correct prediction. Furthermore, standard Deep Ensembles often stop improving after a few models are added. Feature-WGD continues to improve even as the ensemble size grows, suggesting that explicit diversity is more "particle-efficient."
Robustness and Performance Gains
The method was evaluated on standard image classification benchmarks, including CIFAR-10/100, ImageNet, and their corrupted variants (e.g., ImageNet-C) which add visual noise like blur or snow to test robustness to real-world distribution shifts.
Feature-WGD consistently outperformed Deep Ensembles across various metrics:
- Accuracy: On CIFAR-100, it achieved 82.9% accuracy compared to 82.3% for Deep Ensembles.
- Robustness: The gains were even more pronounced on corrupted data. On ImageNet-C, feature-WGD improved accuracy by 1.5% over the baseline.
- Calibration: It produced more "honest" uncertainty. The Expected Calibration Error (ECE) was lower, meaning the model's confidence levels better matched its actual accuracy.
- Efficiency: Despite the extra math, the computational overhead is minimal—only about 6% slower per training epoch than standard ensembles.
Limitations and Future Directions
While highly effective, feature-WGD has a few limitations. The method works best with "wide" networks; if a network is too narrow, there isn't enough "room" in the feature space for models to find diverse, useful representations. Additionally, this study only shared the very last layer. Exploring diversity in shallower parts of the network could be a promising future direction. Finally, scaling this to massive, distributed systems would require further optimization of the communication needed for the repulsive force.
Summary updated