Towards an objective ranking in online reputation systems: the effect of the rating projection

Hao Liao, An Zeng, Yi-Cheng ZhangarXiv 2014 · OpenReview

TL;DR

This paper tackles the problem of accurately ranking online items by addressing the assumption that user ratings are linearly representative of their true preferences. They propose a "rating projection" method to transform discrete, linearly separated ratings into more nuanced values that better capture non-linear user preferences, leading to improved ranking accuracy across various algorithms.

What Problem Does This Paper Address?

Online platforms like Amazon and Netflix rely heavily on reputation systems to help users discover high-quality items. These systems typically collect user ratings (e.g., 1-5 stars) and then employ algorithms to rank items based on these ratings. A fundamental challenge is that while rating scales are discrete and linearly spaced (e.g., a 4-star rating is numerically one step above a 3-star rating), a user's actual preference difference between these ratings might not be linear. For instance, the difference in preference between a 4-star and 5-star rating might be much smaller than the difference between a 1-star and 2-star rating. This discrepancy can lead to inaccurate rankings. The paper aims to bridge this gap by developing a method to adjust raw ratings to better reflect these non-linear user preferences.

Methods

The core contribution of this paper is the rating projection method. This method treats raw, discrete ratings (specifically 1-5 stars) and transforms them into more representative values. The authors preserve the "worst" (1), "neutral" (3), and "best" (5) ratings, but adjust the intermediate ratings (2 and 4) using tunable parameters, $p_1$ and $p_2$. The formulas for this projection are:

$R_2 = 1 + p_1 \times 2$ $R_4 = 3 + p_2 \times 2$

By varying $p_1$ and $p_2$, the paper explores how these adjusted ratings can better capture non-linear preferences. This projection is presented as a data preprocessing step that can be fed into existing ranking algorithms. The paper then evaluates this approach using several established ranking algorithms: the simple "Mean" method, "Iterative Refinement" (IR), "Correlation-based Ranking" (CR), and "Reputation Redistribution Ranking" (RR).

Datasets

The paper uses two real-world datasets:

  • MovieLens: A subset of the GroupLens project data, containing 1 million ratings from 6040 users for 3706 items.
  • Netflix: A subset of the Netflix Prize dataset, with 1.07 million ratings from 5000 users for 16195 movies.

In addition to these real datasets, the authors also generate artificial networks to better understand the underlying mechanisms of their rating projection method. These artificial networks are constructed to mimic user behavior and incorporate different scenarios of user rating confusion.

Key Insights and Ideas

  • Non-linearity of User Preferences: The central idea is that user preferences are not linearly correlated with rating scales. A 4-star rating doesn't necessarily mean a user likes an item twice as much as a 2-star rating.
  • Rating Projection as a Preprocessing Step: The proposed method isn't a new ranking algorithm itself but a way to improve existing ones by providing them with more informative input data.
  • Importance of Intermediate Ratings: The study highlights that the adjustment of intermediate ratings (2 and 4) is crucial, with the parameter $p_2$ (affecting the adjustment of rating 4) showing a more significant impact than $p_1$ (affecting rating 2).
  • System-Specific Optimal Parameters: The optimal values for $p_1$ and $p_2$ are not universal and depend on the specific characteristics of the online system (e.g., Movielens vs. Netflix).
  • Robustness to Spammers: The rating projection method, particularly when used with robust ranking algorithms like RR, shows promise in mitigating the negative impact of spamming behavior on ranking accuracy.

Key Results

  • Improved Ranking Accuracy: Across both real and artificial datasets, using projected ratings consistently led to improved ranking accuracy (measured by the Ranking Score, RS) compared to using original ratings.
  • Significant Improvements: For real datasets, improvements in RS ranged from 2% (for Mean and CR methods) to 6% (for RR method) in Movielens, and 4-5% in Netflix.
  • Optimal Parameters: For Movielens, optimal parameters were found to be $(p_1, p_2) = (0.75, 0.25)$, while for Netflix, they were $(0.85, 1)$. These values suggest that in Movielens, a rating of 4 is perceived closer to 3, whereas in Netflix, a rating of 4 is perceived closer to 5.
  • Spamming Impact: In artificial networks with injected spam, ranking accuracy significantly decreased. However, the rating projection method still provided substantial improvements even in the presence of spammers.
  • RR Algorithm Performance: The RR algorithm, in conjunction with rating projection, demonstrated the most significant improvements, suggesting its effectiveness in handling noisy data and uncovering true item quality.

Limitations and Future Work

  • Focus on 5-Star Systems: The current projection method is designed for 5-star rating systems. The authors suggest that extending this to other rating scales (e.g., 10-score systems, thumb-up/down systems) is a promising area for future research.
  • Optimal Parameter Determination: While the paper identifies optimal parameters for specific datasets, developing a more generalizable method for determining these parameters across different real-world systems remains an open challenge.
  • Broader Applications: The authors believe the concept of "rating projection" could be extended to other areas involving weighted networks, such as link prediction and network analysis, to adjust link weights and potentially achieve similar improvements.

Summary updated

Abstract

Online reputation systems are commonly used by e-commerce providers nowadays. In order to generate an objective ranking of online items' quality according to users' ratings, many sophisticated algorithms have been proposed in the literature. In this paper, instead of proposing new algorithms we focus on a more fundamental problem: the rating projection. The basic idea is that even though the rating values given by users are linearly separated, the real preference of users to items between different values gave is nonlinear. We thus design an approach to project the original ratings of users to more representative values. This approach can be regarded as a data pretreatment method. Simulation in both artificial and real networks shows that the performance of the ranking algorithms can be improved when the projected ratings are used.
BibTeX
@misc{liao2014towards,
  author = {Hao Liao and An Zeng and Yi-Cheng Zhang},
  title = {{Towards an objective ranking in online reputation systems: the effect of the rating projection}},
  year = {2014},
  eprint = {1411.4972},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1411.4972},
}
Figure