Score Aggregation via Spectral Method

Mingyu Xiao, Yuqing WangIJCAI 2017 · DOI

TL;DR

This paper introduces a novel "spectral method" for aggregating scores from multiple agents, moving beyond simple averaging to find an "optimal" consensus score. By framing score aggregation as a noise-reduction problem, their method leverages eigenvector decomposition to identify the most informative "direction" for combining individual scores, leading to more robust and consistent aggregate rankings.

What Problem Does This Paper Address?

In many real-world scenarios, we need to combine individual assessments or scores from multiple sources (agents) to arrive at a single, representative score for a set of items (candidates). Think of calculating a student's GPA from various course grades, or aggregating user ratings for a product. The most common approach is to simply sum or average these scores. However, this "sum-up" method might not always yield the most accurate or representative outcome, especially when agents have different criteria or biases, effectively introducing "noise" into the data. This paper addresses the challenge of finding a more sophisticated aggregation method that can effectively "filter out" this noise and identify a more "ground truth"-aligned aggregate score.

Methods

The authors propose a "spectral method" for score aggregation. The core idea is to view the individual scores as noisy estimates of an underlying "true" score. They mathematically model this by considering the score matrix (where rows are candidates and columns are agents) as data points in a multi-dimensional space, with each dimension representing an agent.

  1. Geometric Interpretation: They frame the problem as projecting these multi-dimensional "candidate points" onto a single-dimensional "object vector space." The sum-up/average methods implicitly choose a fixed projection direction (all agents equally weighted).
  2. Noise Minimization: The goal is to find a projection direction that minimizes the "noise information," defined as the total squared distance between the original candidate points and their projected points. This is analogous to finding the best-fit line in regression.
  3. Spectral Approach: Instead of trying many projection directions, they propose a spectral method. This involves computing a "judgement matrix" (Z = XᵀX, where X is the score matrix). The key insight is that the eigenvector corresponding to the largest eigenvalue of this judgement matrix provides the optimal direction for projection. This eigenvector effectively captures the most significant underlying structure in the score data, representing a weighted combination of agent scores that minimizes noise.
  4. Consistency Indicator (SNR): They also introduce a Signal-to-Noise Ratio (SNR) based on the eigenvalues of the judgement matrix. A high SNR indicates that the aggregate score is well-supported by the input data, suggesting agent consistency, while a low SNR points to significant noise and potential inconsistency among agents.

Datasets

The paper uses two main examples to illustrate their method:

  • MovieLens Dataset (ml-latest-small): A subset of user ratings for movies, used to demonstrate the method on a small, manageable scale.
  • CWTS Leiden Ranking Data: Data on university publications, specifically focusing on the "top 10% P" indicator, used to rank universities. This is a larger, real-world dataset example.

No new benchmarks were introduced; rather, existing datasets were used for evaluation and comparison with the sum-up method.

Key Insights and Ideas

  • Score Aggregation as Noise Reduction: Reframing score aggregation as a problem of minimizing noise information provides a powerful new perspective.
  • Eigenvectors as Optimal Weights: The eigenvector associated with the largest eigenvalue of the judgement matrix naturally emerges as the optimal "direction" (or implicitly, the optimal weights for agents) for aggregation. This is a mathematically elegant solution that avoids arbitrary weight assignments.
  • Spectral Method Simplicity: The spectral method is surprisingly simple to implement and analyze, relying on standard linear algebra operations (eigenvalue decomposition).
  • SNR as a Measure of Trustworthiness: The proposed SNR offers a quantitative way to assess how reliable the aggregated scores are, indicating the consistency of the input data.

Key Results

  • The spectral method produces aggregate scores that are claimed to be more robust and better aligned with the "ground truth" by minimizing noise, compared to simple sum-up or average methods.
  • In the movie example, their method yielded scores with a higher SNR (21.7) compared to the average method (18.8), suggesting better data extraction.
  • In the university ranking example, their method ranked universities differently than the sum-up method, often favoring those with strong performance in a few key fields over those with moderate performance across many. This is presented as a desirable outcome, encouraging specialization.

Limitations and Future Work

  • Assumption of Equal Commensurability: The spectral method, in its basic form, assumes agents' scores are already on a comparable scale. The authors acknowledge that normalization might be needed if this isn't the case.
  • Projection onto Vectors Through Origin: The current method projects candidate points onto vectors originating from the origin. The authors suggest exploring projections onto other types of vectors, though they note potential interpretability issues in extreme cases.
  • Threshold for SNR: Determining an appropriate threshold for the SNR to definitively declare inconsistency requires further study and may depend on the specific application.
  • Further Exploration of Vector Spaces: Investigating more general projection spaces beyond simple vectors could be a fruitful area for future research.

Summary updated

BibTeX
@inproceedings{xiao2017score,
  author = {Mingyu Xiao and Yuqing Wang},
  title = {{Score Aggregation via Spectral Method}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/64},
}

Loading related content...

Figure