Bayesian Analysis of Rank Data with Covariates and Heterogeneous Rankers

Xinran Li, Dingdong Yi, Jun S. LiuStatistical 2016 · arXiv

TL;DR

This paper introduces a unified Bayesian framework (BARC) for analyzing rank data, extending traditional methods to handle covariates associated with ranked items, rankers of varying quality, and incomplete rankings. They propose a novel parameter-expanded Gibbs sampler for efficient inference, enabling more robust rank aggregation and uncertainty quantification.

What Problem Does This Paper Address?

Combining rankings from multiple sources is common, but real-world ranking data presents challenges:

  • Covariates: Ranked entities often have associated features (e.g., player stats, medical case features) that influence rankings.
  • Heterogeneous Rankers: Rankers might have different levels of expertise (quality) or differing opinions on how to rank items.
  • Incomplete Rankings: Not all rankers may provide a complete ranking of all items, or rankings might be specific to subgroups.

Existing methods often struggle to simultaneously address these complexities, leading to less accurate aggregated rankings and an inability to understand the sources of disagreement or variation.

Methods

The authors build upon the Thurstone-Mosteller-Daniels (TMD) model, a classic framework for rank data. Their key contributions include:

  • Bayesian Analysis of Rank data with Covariates (BARC): A unified Bayesian model that incorporates covariates of ranked entities into the TMD framework.
  • Extensions for Heterogeneity:
    • BARCW (Bayesian Analysis of Rank data with Covariates and rankers' Weights): Models ranker quality by allowing their noise precision (variance) to vary, effectively weighting more reliable rankers higher.
  • BARCM (Bayesian Analysis of Rank data with Covariates and Mixture of rankers): Uses a Dirichlet process mixture model to cluster rankers into groups with potentially different opinions, allowing for heterogeneous ranking mechanisms.
  • Parameter-Expanded Gibbs Sampler: A novel MCMC algorithm designed to improve computational efficiency and convergence speed, particularly when dealing with the complex dependencies introduced by covariates and latent variables. This method addresses the slow mixing often seen in standard Gibbs samplers for such models.

Datasets

The paper evaluates their methods using two real-world datasets:

  • NFL Quarterback Rankings: Weekly rankings from multiple sports websites, along with player statistics (covariates).
  • Orthodontics Treatment Evaluation Rankings: Rankings of medical cases by orthodontists, with associated objective measurements (covariates).

They also conduct extensive simulation studies to compare their methods against established rank aggregation techniques under various scenarios.

Key Insights and Ideas

  • Covariates Matter: Incorporating covariates significantly improves rank aggregation accuracy, especially when they are genuinely informative about the underlying item quality.
  • Ranker Quality is Quantifiable: By modeling ranker reliability through variance parameters, BARCW can effectively identify and down-weight less reliable rankers.
  • Opinions Can Differ: BARCM reveals that rankers can indeed have distinct "opinions" or underlying ranking strategies, which can be clustered and analyzed.
  • Uncertainty is Crucial: Bayesian inference naturally provides credible intervals for aggregated ranks, highlighting areas where rankers disagree and where the aggregated ranking is less certain.
  • Efficient Sampling is Key: The parameter-expanded Gibbs sampler is vital for making these complex Bayesian models computationally tractable.

Key Results

  • Improved Accuracy: BARC and its extensions (BARCW, BARCM) consistently outperform traditional methods like Borda Count and Markov Chain approaches in simulation studies, particularly when covariates are relevant. For instance, in Scenario 1 with relevant covariates, BARC models achieved Kendall tau distances that were up to 35% better than Borda Count.
  • Robustness: BARC models show robustness to partial ranking lists and are less susceptible to missing information compared to methods that don't leverage covariates.
  • Identification of Ranker Characteristics: BARCW successfully identified rankers with different qualities, and BARCM clustered rankers with different opinions, as demonstrated in both simulations and real-data analyses. For example, in the NFL data, BARCW identified a subset of rankers with significantly higher quality.
  • Computational Efficiency: The parameter-expanded Gibbs sampler significantly reduces auto-correlation in MCMC samples compared to regular Gibbs samplers, leading to faster convergence without a substantial increase in computational time per step.

Limitations and Future Work

  • Covariates of Rankers: The current framework primarily focuses on covariates of ranked entities. The authors suggest that incorporating covariates of rankers themselves would be a valuable future extension, potentially aiding in detecting ranker qualities and clustering opinions.
  • Heavy-Tailed Errors: The paper primarily uses Gaussian error distributions. Extending the Thurstone model to accommodate more heavy-tailed error distributions could improve robustness to significant heterogeneity across ranking lists.
  • Computational Cost: While the parameter-expanded sampler improves efficiency, the computational cost still increases significantly with the number of entities and rankers, suggesting avenues for further parallelization or algorithmic improvements.

Summary updated

Abstract

Data in the form of ranking lists are frequently encountered, and combining ranking results from different sources can potentially generate a better ranking list and help understand behaviors of the rankers. Of interest here are the rank data under the following settings: (i) covariate information available for the ranked entities; (ii) rankers of varying qualities or having different opinions; and (iii) incomplete ranking lists for non-overlapping subgroups. We review some key ideas built around the Thurstone model family by researchers in the past few decades and provide a unifying approach for Bayesian Analysis of Rank data with Covariates (BARC) and its extensions in handling heterogeneous rankers. With this Bayesian framework, we can study rankers' varying quality, cluster rankers' heterogeneous opinions, and measure the corresponding uncertainties. To enable an efficient Bayesian inference, we advocate a parameter-expanded Gibbs sampler to sample from the target posterior distribution. The posterior samples also result in a Bayesian aggregated ranking list, with credible intervals quantifying its uncertainty. We investigate and compare performances of the proposed methods and other rank aggregation methods in both simulation studies and two real-data examples.
BibTeX
@inproceedings{li2016bayesian,
  author = {Xinran Li and Dingdong Yi and Jun S. Liu},
  title = {{Bayesian Analysis of Rank Data with Covariates and Heterogeneous Rankers}},
  booktitle = {Statistical},
  year = {2016},
}
Figure