A methodology for learning, analyzing, and mitigating social influence bias in recommender systems

Sanjay Krishnan, Jay Patel, Michael J. Franklin, Ken GoldbergRecSys 2014 · DOI

TL;DR

This paper introduces a novel methodology to learn, analyze, and mitigate "Social Influence Bias" (SIB) in recommender systems. By leveraging a unique dataset from the California Report Card, they demonstrate that users' ratings are significantly influenced by the community's average rating, and propose a data-driven approach to correct for this bias, leading to more accurate and representative user preferences.

What Problem Does This Paper Address?

Recommender systems, ubiquitous in online platforms, often display aggregate ratings (like average star ratings) to help users make decisions. However, this practice can introduce "Social Influence Bias" (SIB), where users' true opinions are swayed by the perceived "norm" of the community. This bias can lead to ratings that are less diverse, less representative of individual preferences, and ultimately degrade the effectiveness of the recommender system. While the existence of SIB has been noted, a systematic methodology to quantify, analyze, and actively correct for it, especially in multi-valued rating systems, has been lacking. This paper aims to fill that gap.

Methods

The authors propose a three-phase methodology:

  • Learning Phase: They collect "triplets" of data: a user's initial rating, the median rating they see before potentially revising, and their final rating. This is enabled by a system that allows users to re-rate items after seeing aggregate information.
  • Analysis Phase: A novel non-parametric significance test, based on the Wilcoxon statistic, is introduced to rigorously determine if rating changes are statistically closer to the median. This test is robust to the multi-modal and discrete nature of recommender system ratings, unlike traditional parametric tests. They also introduce a parameter, $\Delta$, to quantify the degree of concentration around the median.
  • Mitigation Phase:
    • Correction Model: Polynomial regression, guided by the Bayesian Information Criterion (BIC) for optimal model selection, is used to learn a function that predicts a user's "unbiased" rating based on their final rating and the observed median. This model can be applied to historical data to correct existing biases.
  • Prediction Model: This model aims to understand the nature of SIB by predicting the change in rating based on the initial rating and the median. This helps in understanding whether positive and negative influences are symmetric.

Datasets

  • California Report Card (CRC): A newly introduced dataset collected from a prototype web/mobile application where participants rate six political issues on a 13-point letter grade scale (A+ to F). This dataset is unique because it allows users to revise their ratings after seeing the median, providing the crucial "triplet" data.
  • Independent Reference Survey: A SurveyMonkey survey administered to a random sample of participants to collect ratings for the same issues without the median being revealed, serving as a baseline to compare against the CRC data.

Key Insights and Ideas

  • SIB is pervasive and quantifiable: The study provides statistically significant evidence of SIB in the CRC dataset, showing that users' revised ratings tend to converge towards the median.
  • Non-parametric approach is crucial: Traditional statistical tests struggle with the discrete and often multi-modal nature of recommender system ratings. The Wilcoxon-based test offers a more robust solution.
  • The "correction" is key: The ability to learn a model that predicts unbiased ratings from biased ones is a significant contribution, offering a practical way to improve recommender system accuracy.
  • Asymmetric influence: While many changes move towards the median, the prediction model suggests that the magnitude of influence might differ for ratings above versus below the median, particularly for certain issues.

Key Results

  • Significant SIB detected: In the CRC, 862 out of 9390 ratings were changed after users saw the median. These changed ratings were, on average, 19.3% closer to the median than those that were not changed.
  • CRC vs. Reference Survey: Final ratings in the CRC change group were 12.0% closer to the median compared to the reference survey, indicating that the SIB observed in the CRC is not solely due to natural rating tendencies.
  • Correction Model Performance: The correction model achieved a normalized RMSE of 12.8% in predicting rating changes. When applied to mitigate bias, it reduced the bias by an average of 76.3%.
  • Classification Accuracy: A logistic regression model to classify changed vs. unchanged ratings achieved an average AUC score of 0.8670, with an average precision of 84.7% and recall of 70.0% at a 0.5 probability threshold.

Limitations and Future Work

  • Generalizability: The authors acknowledge that extending this methodology to other recommender systems with different interfaces and item inventories is a key area for future work.
  • Large Inventories: Investigating how to handle large item inventories and determining the required amount of training data are open questions.
  • Alternative Regression: Exploring alternative regression methods like Gaussian Process Regression and LOESS for bias mitigation is suggested.
  • User Studies: Conducting user studies with manipulated medians (as in Asch's experiments) and exploring methods to classify participants as conformers or non-conformists are future directions.
  • Textual Data: Quantifying the role of social influence on textual data (e.g., review content) is another avenue for research.

Summary updated

BibTeX
@inproceedings{krishnan2014methodology,
  author = {Sanjay Krishnan and Jay Patel and Michael J. Franklin and Ken Goldberg},
  title = {{A methodology for learning, analyzing, and mitigating social influence bias in recommender systems}},
  booktitle = {RecSys},
  year = {2014},
  doi = {10.1145/2645710.2645740},
}
Figure