Probabilistic Aggregation in Ensemble Learning

185 concepts161 papers4 proposals

When combining multiple machine learning models into an ensemble, the simplest approach is a majority vote. If two out of three models classify an image as a "cat," the ensemble outputs "cat." This strategy is known as hard voting. While intuitive, hard voting suffers from a fundamental flaw: it treats all predictions as equally certain, completely discarding the internal confidence levels of the constituent models. A model that is 99% confident is given the exact same weight as a model that is 50.1% confident.

To overcome this limitation, modern ensemble systems rely on soft voting (or probabilistic aggregation). Instead of compressing a model's rich internal prediction down to a single discrete label, soft voting retains the entire probability distribution across all possible classes. By averaging or weighting these continuous probabilities, the ensemble makes a significantly more nuanced and robust decision.

The Intuition: Why Continuous Probabilities Matter

Why does preserving the full probability distribution improve performance? The core reason lies in how soft voting leverages model confidence to resolve ambiguity.

Consider a multi-class classification problem. As explained by Liming Fu et al. in A Machine Learning Based Ensemble Method for Automatic Multiclass Classification of Decisions, a soft voting ensemble aggregates the probability scores assigned to each class across all base learners, selecting the category with the highest aggregate probability. This allows a single highly confident model to override multiple uncertain models. For instance, if Model A and Model B weakly vote for Class 1 (with 51% confidence each), but Model C strongly votes for Class 2 (with 99% confidence), a hard vote would incorrectly choose Class 1. Soft voting correctly allows Model C's high-confidence signal to carry the day.

Furthermore, discrete labels discard the valuable information hidden within a model's secondary choices. In Ensemble Methods for Unsupervised WSD, Samuel Brody et al. demonstrate that probability-mixture methods outperform simple majority voting because they capture these secondary preferences. If multiple models agree that a certain incorrect class is their runner-up choice, that shared distribution contains structural information that helps guide the collective signal toward the true global consensus, resolving ambiguities that might otherwise derail individual models.

Smoothing Decision Boundaries and Reducing Variance

Every machine learning architecture has its own structural biases and erratic decision boundaries. When models are deployed on complex or noisy data, these individual quirks can lead to high variance and classification errors.

Averaging continuous probabilities acts as an elegant regularizer. By calculating a weighted average of probability outputs, soft voting smooths out the sharp, jagged decision boundaries of individual models. As shown by Geng Tian et al. in A deep ensemble learning-based automated detection of COVID-19 using lung CT images and Vision Transformer and ConvNeXt, this aggregation balances the strengths of entirely different architectural paradigms (such as Vision Transformers and Convolutional Networks), reducing individual variance and lowering the overall classification error.

Similarly, Jianying Shi et al. emphasize in Recognition System for Diagnosing Pneumonia and Bronchitis Using Children's Breathing Sounds Based on Transfer Learning that averaging class probability vectors improves classification stability. It effectively cancels out the unique biases inherent to different model initializations and feature extraction methods, leading to more robust predictions than any single model could provide.

Advanced Aggregation: Weighting and Softmax Scaling

While simple averaging works well, performance can be further optimized by dynamically weighting each model's contribution or applying non-linear scaling to their outputs.

One approach is to learn explicit weights for each constituent model based on its historical performance or domain expertise. For example, Yuhang Zhu utilizes a learned weight model within a soft voting framework in RDproj at SemEval-2024 Task 4: An Ensemble Learning Approach for Multilingual Detection of Persuasion Techniques in Memes to dynamically balance predictions from multiple base learners, enhancing robustness against data imbalances. Similarly, Steven Xu et al. employ learned weights in UNIMELB at SemEval-2016 Tasks 4A and 4B: An Ensemble of Neural Networks and a Word2Vec Based Model for Sentiment Classification to adjust how each model's probability distribution contributes to the final output based on the specific context of the data.

Alternatively, the ensemble can use non-linear scaling, such as softmax-weighted aggregation, to amplify high-confidence predictions while dampening noisy, uncertain ones. In Node Copying for Protection Against Graph Neural Network Topology Attacks, Florence Regol et al. demonstrate that aggregating predictions across multiple surrogate structures via softmax averaging naturally weights high-confidence outputs more heavily than low-confidence ones. This self-scaling property provides a robust defense against adversarial perturbations without requiring explicit meta-classifiers.

In a similar vein, Aakriti Agrawal et al. introduce a product-of-experts approach in Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems to aggregate normalized log-likelihoods across an ensemble of large language models. Their work demonstrates that leveraging internal confidence levels consistently outperforms traditional majority voting on complex reasoning tasks, proving that the mathematical benefits of probabilistic aggregation scale from traditional classifiers up to the largest modern generative models.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_soft_voting_ensemble_aa8f8bbf,
  title = {soft voting ensemble},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/soft-voting-ensemble/rdr_aa8f8bbff9c6495c9ed3720775489b54},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_soft_voting_ensemble_aa8f8bbf_v1,
  title = {soft voting ensemble},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/soft-voting-ensemble/rdr_aa8f8bbff9c6495c9ed3720775489b54/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake