Ensembling is a cornerstone of robust machine learning. Whether combining multiple models, fusing different data modalities, or aggregating predictions across test-time augmentations, the standard approach is often simple uniform averaging.
However, uniform averaging suffers from a fundamental flaw: it treats all predictions as equally informative, regardless of how confused a model might be about a specific instance. Imagine an image classifier evaluating an image under five different geometric rotations. For three rotations, the object remains clear, and the model outputs a sharp, confident prediction for "cat." For the other two rotations, the object is heavily obscured, causing the model to become completely confused and distribute its probability evenly across "cat," "dog," "car," and "house."
If we simply average these five probability distributions, the high-entropy, confused predictions dilute the clear, low-entropy signals. The ensemble's overall confidence drops, and the correct class can easily be overtaken by spurious noise.
Entropy as an Unsupervised Proxy for Expertise
To solve this dilution problem, we need a way to dynamically assess how reliable a prediction is at inference time, without access to ground-truth labels. The key insight across a wide body of research is to use the Shannon entropy of a model's output probability distribution as an unsupervised proxy for confidence.
Mathematically, for a discrete probability distribution $P(y)$, the entropy $H(P)$ is defined as:
$$H(P) = -\sum_{i} P(y_i) \log P(y_i)$$
A sharp, highly confident distribution yields low entropy, while a flat, uncertain distribution yields high entropy. By weighting each prediction proportionally to its certainty—often using the reciprocal of entropy ($1/H$), exponentiated negative entropy ($\exp(-H)$), or inverse perplexity—the system can dynamically prioritize the most reliable signals.
As Oh et al. demonstrate in DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation, prediction entropy serves as an effective indicator of model expertise. By comparing these values, systems can interpolate weights to favor the most certain predictor. In fact, the authors show that weight interpolation guided by entropy can yield a merged model with lower prediction entropy than either constituent model alone, proving that confidence-guided fusion does not merely average performance but actively resolves ambiguity.
Filtering Noise in Test-Time Augmentation and Adaptation
This mechanism is particularly powerful in Test-Time Augmentation (TTA) and Test-Time Adaptation, where a model must handle out-of-distribution or corrupted data on the fly.
In TTA, a single input is transformed into multiple augmented views. Instead of treating every transformation equally, Chun et al. in Cyclic Test Time Augmentation with Entropy Weight Method propose using the reciprocal of softmax entropy to weight individual views. This automatically filters out transformations that destroy class-discriminative features, ensuring that transformations inducing model confusion are ignored. Similarly, Seth et al. utilize uncertainty-aware weighting in Uncertainty-Aware Test-Time Augmented Ensemble of BERTs for Classification of Common Mental Illnesses on Social Media Posts to mitigate worst-case overconfidence and reduce maximum calibration errors.
When adapting models continuously during inference, standard entropy minimization risks making the model overconfident in its own mistakes. To counter this, the authors of Test-time low rank adaptation via confidence maximization for zero-shot generalization of vision-language models implement a weighted consensus across multiple augmented views, ensuring updates are driven by consistent, low-entropy patterns rather than spurious artifacts. Furthermore, Guan et al. show in Statistics Caching Test-Time Adaptation for Vision-Language Models how instance-level entropy can balance a zero-shot model's base output with an adapted cache prediction, dynamically falling back to the base model when the cache becomes highly uncertain.
Balancing Multi-Modal, Hierarchical, and Multi-Expert Systems
The utility of entropy-based weighting extends far beyond TTA into any architecture where heterogeneous or conflicting predictive signals must be reconciled.
Multi-View and Multi-Modal Fusion
In multi-view classification, different data streams (such as audio, video, and text) carry varying levels of noise depending on the context. Liang et al. demonstrate in View-Association-Guided Dynamic Multi-View Classification that assigning importance based on the entropy of individual view distributions suppresses uncertain modalities. For instance, in voice disorder analysis, Koudounas et al. use entropy measures to dynamically prioritize whether a sustained vowel or a spoken sentence provides the more certain pathological signal. For complex visual tasks, Li demonstrates in Discriminant Mutual Subspace Learning for Indoor and Outdoor Face Recognition that an adaptive fusion strategy based on classifier entropy significantly improves performance by discounting ambiguous feature spaces.
Hierarchical and Subnetwork Fusion
When processing data at multiple semantic levels, conflict is common. Ma et al. show in Reasoning Like Experts: Leveraging Multimodal Large Language Models for Drawing-based Psychoanalysis that weighting hierarchical levels by their prediction entropy allows the system to prioritize the most information-dense signals. Similarly, Kim et al. use negative Shannon entropy in Subnetwork representation learning for discovering network biomarkers in predicting lymph node metastasis in early oral cancer to integrate predictions from multiple subnetwork experts, ensuring that modules with reliable signals dominate the final output.
Human-Machine and Model-Retrieval Hybridization
Entropy can also bridge entirely different paradigms of knowledge. Zheng et al. combine deep learning predictions with human expert diagnoses by dynamically adjusting weights based on the model's output entropy and human self-reported confidence. In retrieval-augmented systems, Zhou et al. use an entropy-based confidence network in Syntax-Aware Retrieval Augmentation for Neural Symbolic Regression to prevent noisy retrieved examples from overriding a confident neural model, a strategy mirrored by Lee et al. in RA-TTA: Retrieval-Augmented Test-Time Adaptation for Vision-Language Models.
Broad Applicability Across Domains
The core principle remains invariant across diverse domains:
- Language Modeling: Ormazabal et al. use prediction entropy in CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models to dynamically shift weight toward a small, domain-specific expert when it exhibits low entropy compared to a massive black-box language model. For text classification ensembles, Mobin et al. employ inverse perplexity as a confidence metric to prioritize certain models.
- Graph Networks: Zhang et al. use entropy-based weighting to fuse conflicting predictive signals in the presence of noisy or incomplete data.
- Preference Learning: Even in reinforcement learning from human preferences, An et al. apply a related weighting scheme based on the divergence of reward model distributions from a prior, treating higher divergence as a signal of confidence.
Why It Works
Entropy-based confidence weighting works because it transforms ensembling from a static, democratic vote into a dynamic, meritocratic consultation. Instead of allowing uncertain or hallucinated predictions to pollute the consensus, the system uses the shape of the probability distribution itself to measure reliability. By letting the sharpest, most certain distributions dictate the final output, systems naturally filter out noise, preserve class-discriminative features, and achieve superior calibration without requiring explicit supervision.
Key Foundational Work
Spectral entropy feature in full-combination multi-stream for robust ASR (2005) — contributed 1 concept elements to this direction.
Bayesian Fusion of Color and Texture Segmentations (1999) — contributed 1 concept elements to this direction.