Deep Combinatorial Aggregation

Yuesong Shen, Daniel Cremers

Abstract

Standard deep ensembles are effective for uncertainty estimation but are computationally expensive and do not easily allow for a single, summarized "average" model. Deep Combinatorial Aggregation (DCA) addresses these limitations by maintaining multiple instances of individual network components (such as layers or blocks) rather than full models. By combining these components in an exponential number of ways, DCA achieves state-of-the-art uncertainty estimation and introduces Deep Combinatorial Weight Averaging (DCWA), a simpler and more robust alternative to Stochastic Weight Averaging (SWA).

Introduction

While deep learning has revolutionized various fields, standard neural networks are often "overconfident," providing incorrect predictions with high certainty. This lack of reliable uncertainty estimation is a significant barrier for safety-critical applications like autonomous driving or medical diagnosis. The current gold standard, Deep Ensembling, involves training multiple independent models and averaging their predictions. However, deep ensembles are computationally expensive to scale and do not naturally produce a single, high-performing averaged model.

Methods: Combinatorial Explosion

The core innovation of Deep Combinatorial Aggregation (DCA) is to move beyond ensembling full models by looking at the hierarchical structure of the networks. Instead of one version of a network, DCA maintains $n$ different instances for every component.

The DCA process: components from different model instances are combinatorially aggregated to form an exponential number of model proposals.

Key Components of DCA:

  • Combinatorial Aggregation: If a network has $L$ layers and $n$ versions of each layer, one can form $n^L$ different model proposals. For example, a 20-layer network with just 2 versions per layer yields over a million possible models.
  • Joint Training: During training, each forward pass randomly selects one instance for each layer to form a temporary model proposal. All instances are updated via backpropagation.
  • Granularity: The method can be applied at different levels, from individual channels or layers (fine-grain) to residual blocks or the entire model (coarse-grain).
  • Consistency Enforcing Loss (CEL): To prevent random combinations from diverging, a KL divergence-based loss term encourages different model proposals to agree on their predictions.
  • DCWA: For fine-grain models, averaging the weights of the $n$ instances of a layer creates a single superior "average" model that works as a plug-and-play improvement over standard training.

Experimental Results

The authors validated DCA across image classification (CIFAR-10, SVHN), distributional shift (CIFAR-10-C), and out-of-distribution (OOD) tasks.

Performance of DCA compared to baselines across varying levels of corruption severity on CIFAR-10-C. DCA (Trunkwise and Modelwise) consistently maintains higher accuracy and lower Negative Log-Likelihood.

Key Findings:

  • Superior Uncertainty: On CIFAR-10, "Modelwise DCA" achieved a Negative Log-Likelihood (NLL) of 0.1550, significantly outperforming standard Deep Ensembles (0.1746) and SWAG (0.2016).
  • Robustness to Shift: As shown in the CIFAR-10-C results, DCA maintains higher accuracy and lower uncertainty across all levels of corruption severity compared to standard methods and MC Dropout.
  • Outlier Detection: In OOD tasks, DCA reduced the False Positive Rate (FPR@95) to 10.8%, compared to 14.0% for Deep Ensembles.

Discussion and Insights

There is a notable trade-off regarding granularity: Fine-grain DCA (layer-wise) is excellent for producing a single high-performance averaged model (DCWA), while Coarse-grain DCA (block-wise or model-wise) is significantly better for uncertainty estimation and OOD detection. Unlike SWA, which requires custom learning rate schedules and BatchNorm adjustments, DCWA is "plug-and-play" and works with standard training protocols.

Limitations

Despite its benefits, DCA introduces training overhead, requiring multiple passes per minibatch to update all component instances. Additionally, storing multiple instances increases VRAM requirements, though this remains manageable on modern hardware. Future work may explore hybrid approaches that combine the weight-averaging benefits of fine-grain DCA with the uncertainty estimation strengths of coarse-grain DCA.

Summary updated

Abstract

Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue. This includes deep ensemble, a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks. In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA). DCA creates multiple instances of network components and aggregates their combinations to produce diversified model proposals and predictions. DCA components can be defined at different levels of granularity. And we discovered that coarse-grain DCAs can outperform deep ensemble for uncertainty-aware learning both in terms of predictive performance and uncertainty estimation. For fine-grain DCAs, we discover that an average parameterization approach named deep combinatorial weight averaging (DCWA) can improve the baseline training. It is on par with stochastic weight averaging (SWA) but does not require any custom training schedule or adaptation of BatchNorm layers. Furthermore, we propose a consistency enforcing loss that helps the training of DCWA and modelwise DCA. We experiment on in-domain, distributional shift, and out-of-distribution image classification tasks, and empirically confirm the effectiveness of DCWA and DCA approaches.
BibTeX
@inproceedings{shen2022deep,
  author = {Yuesong Shen and Daniel Cremers},
  title = {{Deep Combinatorial Aggregation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=byMcacS8GYZ},
}
Figure