Abstract
Standard deep ensembles are effective for uncertainty estimation but are computationally expensive and do not easily allow for a single, summarized "average" model. Deep Combinatorial Aggregation (DCA) addresses these limitations by maintaining multiple instances of individual network components (such as layers or blocks) rather than full models. By combining these components in an exponential number of ways, DCA achieves state-of-the-art uncertainty estimation and introduces Deep Combinatorial Weight Averaging (DCWA), a simpler and more robust alternative to Stochastic Weight Averaging (SWA).
Introduction
While deep learning has revolutionized various fields, standard neural networks are often "overconfident," providing incorrect predictions with high certainty. This lack of reliable uncertainty estimation is a significant barrier for safety-critical applications like autonomous driving or medical diagnosis. The current gold standard, Deep Ensembling, involves training multiple independent models and averaging their predictions. However, deep ensembles are computationally expensive to scale and do not naturally produce a single, high-performing averaged model.
Methods: Combinatorial Explosion
The core innovation of Deep Combinatorial Aggregation (DCA) is to move beyond ensembling full models by looking at the hierarchical structure of the networks. Instead of one version of a network, DCA maintains $n$ different instances for every component.
Key Components of DCA:
- Combinatorial Aggregation: If a network has $L$ layers and $n$ versions of each layer, one can form $n^L$ different model proposals. For example, a 20-layer network with just 2 versions per layer yields over a million possible models.
- Joint Training: During training, each forward pass randomly selects one instance for each layer to form a temporary model proposal. All instances are updated via backpropagation.
- Granularity: The method can be applied at different levels, from individual channels or layers (fine-grain) to residual blocks or the entire model (coarse-grain).
- Consistency Enforcing Loss (CEL): To prevent random combinations from diverging, a KL divergence-based loss term encourages different model proposals to agree on their predictions.
- DCWA: For fine-grain models, averaging the weights of the $n$ instances of a layer creates a single superior "average" model that works as a plug-and-play improvement over standard training.
Experimental Results
The authors validated DCA across image classification (CIFAR-10, SVHN), distributional shift (CIFAR-10-C), and out-of-distribution (OOD) tasks.
Key Findings:
- Superior Uncertainty: On CIFAR-10, "Modelwise DCA" achieved a Negative Log-Likelihood (NLL) of 0.1550, significantly outperforming standard Deep Ensembles (0.1746) and SWAG (0.2016).
- Robustness to Shift: As shown in the CIFAR-10-C results, DCA maintains higher accuracy and lower uncertainty across all levels of corruption severity compared to standard methods and MC Dropout.
- Outlier Detection: In OOD tasks, DCA reduced the False Positive Rate (FPR@95) to 10.8%, compared to 14.0% for Deep Ensembles.
Discussion and Insights
There is a notable trade-off regarding granularity: Fine-grain DCA (layer-wise) is excellent for producing a single high-performance averaged model (DCWA), while Coarse-grain DCA (block-wise or model-wise) is significantly better for uncertainty estimation and OOD detection. Unlike SWA, which requires custom learning rate schedules and BatchNorm adjustments, DCWA is "plug-and-play" and works with standard training protocols.
Limitations
Despite its benefits, DCA introduces training overhead, requiring multiple passes per minibatch to update all component instances. Additionally, storing multiple instances increases VRAM requirements, though this remains manageable on modern hardware. Future work may explore hybrid approaches that combine the weight-averaging benefits of fine-grain DCA with the uncertainty estimation strengths of coarse-grain DCA.
Summary updated