Feature Space Particle Inference for Neural Network Ensembles

Shingo Yashima, Teppei Suzuki, Kohta Ishikawa, Ikuro Sato, Rei Kawakami

Abstract

Ensembles of deep neural networks demonstrate improved performance over single models. For enhancing the diversity of ensemble members while keeping their performance, particle-based inference methods offer a promising approach from a Bayesian perspective. However, the best way to apply these methods to neural networks is still unclear: seeking samples from the weight-space posterior suffers from inefficiency due to over-parameterization, while seeking samples directly from the function-space posterior often results in serious underfitting. In this study, we propose optimizing particles in the feature space where the activation of a specific intermediate layer lies. Our method encourages each member to capture distinct features, improving ensemble prediction robustness. Extensive evaluation on real-world datasets shows that our model significantly outperforms the gold-standard Deep Ensembles on various metrics, including accuracy, calibration, and robustness.

Introduction

Ensembles are the industry standard for boosting model performance and estimating uncertainty. However, the standard approach—Deep Ensembles—relies on different random initializations to achieve diversity. This often leads to redundant ensemble members that fail to capture the full breadth of the data distribution. While Bayesian methods attempt to enforce diversity, they typically struggle with the high dimensionality of weight space or the restrictive nature of function space (logits).

This paper introduces Feature Space Particle Inference, a novel framework that promotes diversity in intermediate feature representations. By targeting the "Goldilocks" zone between raw weights and final outputs, our method ensures that ensemble members learn distinct, complementary features of the data.

Methodology: Feature-WGD

We propose Feature-WGD (Wasserstein Gradient Descent), which treats ensemble members as interacting particles. The method is built on several key technical pillars:

1. Shared Classifier and Independent Extractors

To ensure that features across different models remain semantically comparable, each ensemble member utilizes its own feature extractor but feeds into a single shared linear classifier. This architecture forces the diversity to manifest in how the data is processed, rather than how it is finally categorized.

2. The Repulsive Force

During training, we apply a "repulsive force" to the feature vectors. If two models produce similar feature representations for the same input, this force pushes them apart in the feature space. This prevents the models from converging on the same local minima.

3. Subspace Projection and Heavy-Tailed Priors

Calculating diversity in high-dimensional feature spaces is computationally expensive. We address this by projecting the repulsive force onto a low-dimensional subspace identified by the gradients of the log-likelihood. Furthermore, we utilize a Cauchy prior—a heavy-tailed distribution—which better aligns with the natural activation patterns of neural networks compared to standard Gaussian priors.

The Multi-View Hypothesis

The core intuition behind our approach is the Multi-View Hypothesis. Real-world objects possess multiple identifying characteristics (e.g., a lion can be identified by its mane, its face, or its paws). If Model A focuses on the mane and Model B focuses on the face, the ensemble becomes significantly more robust. If one feature is obscured—such as a lion standing behind tall grass—the ensemble can still rely on the member that learned the alternative feature.

Experimental Results

We evaluated Feature-WGD against standard Deep Ensembles across several benchmarks, including CIFAR-10/100 and ImageNet, as well as their "corrupted" counterparts (CIFAR-C and ImageNet-C).

Robustness and Accuracy

On ImageNet-C, which tests models against real-world distribution shifts like blur and weather effects, Feature-WGD improved accuracy by +1.5% over Deep Ensembles. On CIFAR-100, our method outperformed the baseline across all primary metrics:

  • Accuracy: 82.9% (vs. 82.3%)
  • Negative Log-Likelihood: 0.624 (vs. 0.632)

Calibration

Feature-WGD consistently achieved lower Expected Calibration Error (ECE). This indicates that the model's confidence levels are better aligned with its actual probability of being correct, a critical requirement for safety-critical applications.

Scalability

Unlike standard ensembles that often see diminishing returns as more models are added, Feature-WGD continues to improve as the ensemble size grows. By explicitly carving out new "views" for each additional member, the method maximizes the utility of the added computational capacity.

Conclusion

Feature Space Particle Inference offers a powerful alternative to traditional ensemble methods. By encouraging diversity in the feature space, we achieve models that are not only more accurate but significantly more robust to data corruption and better calibrated. Future work will explore the impact of network width on feature capacity and the potential for extending this inference to shallower layers of the network.

Summary updated

Abstract

Ensembles of deep neural networks demonstrate improved performance over single models. For enhancing the diversity of ensemble members while keeping their performance, particle-based inference methods offer a promising approach from a Bayesian perspective. However, the best way to apply these methods to neural networks is still unclear: seeking samples from the weight-space posterior suffers from inefficiency due to the over-parameterization issues, while seeking samples directly from the function-space posterior often results in serious underfitting. In this study, we propose optimizing particles in the feature space where the activation of a specific intermediate layer lies to address the above-mentioned difficulties. Our method encourages each member to capture distinct features, which is expected to improve ensemble prediction robustness. Extensive evaluation on real-world datasets shows that our model significantly outperforms the gold-standard Deep Ensembles on various metrics, including accuracy, calibration, and robustness. Code is available at https://github.com/DensoITLab/featurePI .
BibTeX
@inproceedings{yashima2022feature,
  author = {Shingo Yashima and Teppei Suzuki and Kohta Ishikawa and Ikuro Sato and Rei Kawakami},
  title = {{Feature Space Particle Inference for Neural Network Ensembles}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=OyuzyzztF1},
}
Figure