When building an ensemble of machine learning models, the most common approach is to average their outputs. However, this simple arithmetic mean relies on a dangerous assumption: that the raw scores produced by different models are numerically comparable. In practice, especially when combining heterogeneous architectures, this assumption frequently fails, leading to ensembles where a single model's scale dominates the others.
Researchers across domains—from recommendation systems to medical survival analysis—have converged on a robust solution: Rank-Based Aggregation. Instead of averaging raw scores, these methods transform scores into ordinal ranks before combining them.
The Problem of Numerical Dominance
Imagine you are ensembling two models to predict the risk of a disease. Model A outputs a probability between 0 and 1. Model B, a Support Vector Machine, outputs a distance from a hyperplane that might range from -100 to +100.
If you simply average these scores, Model B will almost entirely determine the final ranking. A small change in Model B’s output (e.g., from 50 to 51) would completely overwhelm the entire range of Model A. This "scale mismatch" is a fundamental hurdle in heterogeneous ensembling.
As noted by Sebastian Pölsterl and Pankaj Gupta in their work on predicting prostate cancer survival, raw score averaging is problematic because base learners often optimize different loss functions and operate on arbitrary scales. Their research demonstrated that transforming risk scores into ranks ensures that each constituent model contributes equally to the final consensus, regardless of its specific output distribution.
Ranking as a Universal Normalizer
Rank-based aggregation works by treating each model as a "voter" that provides an ordered list of preferences rather than a precise measurement. By converting scores to ranks (e.g., the item with the highest score becomes rank 1, the second highest rank 2, and so on), we map every model's output onto the same uniform distribution.
This approach offers several intuitive advantages:
- Scale Invariance: It doesn't matter if a model outputs values in $[0, 1]$ or $[-\infty, \infty]$.
- Outlier Robustness: Extreme raw scores that might skew a mean are tamed into predictable rank positions.
- Consensus Focus: The ensemble prioritizes items that multiple models agree are "near the top," even if the models disagree on the absolute magnitude of the scores.
In the context of personalized tag recommendation, Lars Schmidt-Thieme and Steffen Rendle utilized this in their winning approach for BibSonomy. They found that aggregating predicted rank positions was significantly more effective than averaging raw scores, specifically because it prevented models with high-variance output ranges from disproportionately influencing the final recommendation list.
Stability Across Architectures
The utility of rank aggregation extends to complex embedding spaces. In Knowledge Graph Embedding (KGE), different architectures (like TransE or RotatE) produce scores based on vastly different geometric interpretations.
Ling Yue, quanming yao, and their colleagues addressed this in RelEns-DSC. They argued that rankings provide a "more stable and comparable signal" across heterogeneous architectures. By focusing on the relative ordering of entities rather than the raw distance metrics, their ensemble framework achieved state-of-the-art results on competitive benchmarks like the Open Graph Benchmark (OGB).
Summary of the Intuition
Rank-based aggregation shifts the ensemble's goal from averaging magnitudes to finding a consensus on order. By discarding the potentially noisy and uncalibrated "confidence" of raw scores, the ensemble gains a more reliable signal of what the models actually agree on. Whether the task is identifying the most relevant tag or the patient at highest risk, the relative position is often a more robust feature than the raw numerical output.