Abstract
Ensembles of deep neural networks demonstrate improved performance over single models. For enhancing the diversity of ensemble members while keeping their performance, particle-based inference methods offer a promising approach from a Bayesian perspective. However, the best way to apply these methods to neural networks is still unclear: seeking samples from the weight-space posterior suffers from inefficiency due to over-parameterization, while seeking samples directly from the function-space posterior often results in serious underfitting. In this study, we propose optimizing particles in the feature space where the activation of a specific intermediate layer lies. Our method encourages each member to capture distinct features, improving ensemble prediction robustness. Extensive evaluation on real-world datasets shows that our model significantly outperforms the gold-standard Deep Ensembles on various metrics, including accuracy, calibration, and robustness.
Introduction
Ensembles are the industry standard for boosting model performance and estimating uncertainty. However, the standard approach—Deep Ensembles—relies on different random initializations to achieve diversity. This often leads to redundant ensemble members that fail to capture the full breadth of the data distribution. While Bayesian methods attempt to enforce diversity, they typically struggle with the high dimensionality of weight space or the restrictive nature of function space (logits).
This paper introduces Feature Space Particle Inference, a novel framework that promotes diversity in intermediate feature representations. By targeting the "Goldilocks" zone between raw weights and final outputs, our method ensures that ensemble members learn distinct, complementary features of the data.
Methodology: Feature-WGD
We propose Feature-WGD (Wasserstein Gradient Descent), which treats ensemble members as interacting particles. The method is built on several key technical pillars:
1. Shared Classifier and Independent Extractors
To ensure that features across different models remain semantically comparable, each ensemble member utilizes its own feature extractor but feeds into a single shared linear classifier. This architecture forces the diversity to manifest in how the data is processed, rather than how it is finally categorized.
2. The Repulsive Force
During training, we apply a "repulsive force" to the feature vectors. If two models produce similar feature representations for the same input, this force pushes them apart in the feature space. This prevents the models from converging on the same local minima.
3. Subspace Projection and Heavy-Tailed Priors
Calculating diversity in high-dimensional feature spaces is computationally expensive. We address this by projecting the repulsive force onto a low-dimensional subspace identified by the gradients of the log-likelihood. Furthermore, we utilize a Cauchy prior—a heavy-tailed distribution—which better aligns with the natural activation patterns of neural networks compared to standard Gaussian priors.
The Multi-View Hypothesis
The core intuition behind our approach is the Multi-View Hypothesis. Real-world objects possess multiple identifying characteristics (e.g., a lion can be identified by its mane, its face, or its paws). If Model A focuses on the mane and Model B focuses on the face, the ensemble becomes significantly more robust. If one feature is obscured—such as a lion standing behind tall grass—the ensemble can still rely on the member that learned the alternative feature.
Experimental Results
We evaluated Feature-WGD against standard Deep Ensembles across several benchmarks, including CIFAR-10/100 and ImageNet, as well as their "corrupted" counterparts (CIFAR-C and ImageNet-C).
Robustness and Accuracy
On ImageNet-C, which tests models against real-world distribution shifts like blur and weather effects, Feature-WGD improved accuracy by +1.5% over Deep Ensembles. On CIFAR-100, our method outperformed the baseline across all primary metrics:
- Accuracy: 82.9% (vs. 82.3%)
- Negative Log-Likelihood: 0.624 (vs. 0.632)
Calibration
Feature-WGD consistently achieved lower Expected Calibration Error (ECE). This indicates that the model's confidence levels are better aligned with its actual probability of being correct, a critical requirement for safety-critical applications.
Scalability
Unlike standard ensembles that often see diminishing returns as more models are added, Feature-WGD continues to improve as the ensemble size grows. By explicitly carving out new "views" for each additional member, the method maximizes the utility of the added computational capacity.
Conclusion
Feature Space Particle Inference offers a powerful alternative to traditional ensemble methods. By encouraging diversity in the feature space, we achieve models that are not only more accurate but significantly more robust to data corruption and better calibrated. Future work will explore the impact of network width on feature capacity and the potential for extending this inference to shallower layers of the network.
Summary updated