Measuring Quality, Reputation and Trust in Online Communities

Hao Liao, Giulio Cimini, Matús Medo

TL;DR

This paper introduces QTR, a novel ranking algorithm for online communities that simultaneously assesses the quality of content (objects) and the reputation of users. It uniquely incorporates social trust relationships as a crucial factor, demonstrating that this leads to more robust and less biased rankings compared to traditional methods.

What Problem Does This Paper Address?

In today's digital world, online communities are awash with information and user-generated content. This creates a significant challenge: how do we effectively identify high-quality content and trustworthy users amidst the noise? Traditional ranking systems, like Google's PageRank, focus on link structures. While effective for web pages, they often fall short in complex online communities where users interact with both content (objects) and each other. These interactions create a social fabric, including friendships and trust, which can provide valuable signals for ranking. The authors aim to bridge this gap by developing a unified ranking framework that leverages both content interactions and social connections.

Methods

The core of the paper is the QTR (Quality, Trust, and Reputation) algorithm. It's designed for bipartite networks, where one set of nodes represents users and the other represents objects (e.g., papers, songs). QTR iteratively calculates two scores:

  • Quality (Q) for objects: This score is derived from how users interact with an object and the reputation of those users.
  • Reputation (R) for users: This score is based on the quality of objects a user interacts with and the reputation of other users who interact with those same objects.

What makes QTR novel is its integration of social trust relationships. The algorithm can incorporate a "trust" score between users, which influences both their own reputation and how they contribute to the quality scores of objects they interact with. This is a generalization of algorithms like HITS, allowing for tunable parameters that control the influence of different factors (e.g., object degree, user degree, trust). The authors also propose a time-decay mechanism to account for the evolving nature of online communities.

Datasets

The authors evaluated their QTR algorithm on two real-world datasets:

  • EconoPhysics Forum: This dataset consists of user interactions (uploading, downloading, viewing) with papers on an online forum for physicists and social scientists. It's a bipartite network of users and papers.
  • Last.fm: This dataset includes user listening records for artists and user-to-user friendship relationships on the music streaming platform. It's a bipartite network of users and artists, embedded within a social network of friends.

Key Insights and Ideas

  • Trust is a powerful ranking signal: The most significant finding is that incorporating social trust relationships demonstrably improves the quality and robustness of rankings. It helps to mitigate the issue of isolated, extremely high scores that plague simpler ranking methods.
  • Unified ranking is possible: QTR provides a flexible framework to simultaneously rank both users and objects, recognizing their interdependence in online communities.
  • Parameter tuning matters: The algorithm's performance is sensitive to its parameters. Different configurations can lead to vastly different ranking outcomes, highlighting the need for careful tuning based on the specific community and goals.
  • Bias mitigation: The authors show that poorly chosen algorithms or datasets can lead to heavily biased results. QTR, particularly with appropriate parameter settings, can help avoid these biases.
  • Time matters: Online communities are dynamic. The proposed time-decay mechanism is a crucial step towards creating ranking systems that adapt to the changing relevance of content and users over time.

Key Results

  • Improved Distribution: When trust is incorporated, the probability distributions of quality and reputation scores become more evenly spread, avoiding extreme outliers. This is particularly evident in the Last.fm dataset.
  • Better User/Object Representation: In the EconoPhysics forum, the configuration labeled "0110" (which penalizes high-degree users) yielded the most balanced results, with users having a reasonable number of contributions and papers showing better average citations.
  • Trust Enhances Correlation: For Last.fm, incorporating trust significantly improved the correlation between user reputation and their number of friends (CRf), indicating that highly connected users are indeed more reputed.
  • Comparison to HITS: QTR reduces to standard HITS when certain parameters are zeroed out. The paper demonstrates that the added parameters and trust component offer significant advantages over basic HITS.

Limitations and Future Work

  • Parameter Sensitivity: The authors acknowledge that the number of parameters in QTR can make it challenging to assess reliability and tune effectively.
  • Weighting Schemes: The choice of weights for different user actions (upload, download, view) in the EconoPhysics forum was a specific, albeit reasonable, choice. Future work could explore more sophisticated or data-driven weighting systems.
  • Time Decay Function: While a time-decay mechanism is proposed, its specific form and parameters require further investigation and testing.
  • Projected Trust: The concept of "projected trust" (inferred from user actions rather than explicit declarations) shows promise but requires more extensive study.
  • Dataset Limitations: The EconoPhysics dataset lacks explicit social/trust information, requiring the authors to focus on the bipartite network alone for some experiments. The Last.fm dataset has a peculiar user degree distribution.

The authors suggest that future work should focus on exploring different weighting schemes, refining the time-decay function, and conducting more extensive tests with projected trust to further enhance the algorithm's capabilities.

Summary updated

Abstract

In the Internet era the information overload and the challenge to detect quality content has raised the issue of how to rank both resources and users in online communities. In this paper we develop a general ranking method that can simultaneously evaluate users’ reputation and objects’ quality in an iterative procedure, and that exploits the trust relationships and social acquaintances of users as an additional source of information. We test our method on two real online communities, the EconoPhysics forum and the Last.fm music catalogue, and determine how different variants of the algorithm influence the resultant ranking. We show the benefits of considering trust relationships, and define the form of the algorithm better apt to common situations.
BibTeX
@inproceedings{liao2012measuring,
  author = {Hao Liao and Giulio Cimini and Matús Medo},
  title = {{Measuring Quality, Reputation and Trust in Online Communities}},
  booktitle = {ISMIS},
  year = {2012},
  url = {https://openreview.net/forum?id=MAE5lzppGs},
}
Figure