Measuring credibility of users in an e-learning environment

Wei Wei, Jimmy Ho-Man Lee, Irwin King

Introduction

In modern e-learning platforms, such as "Learning Villages," users interact dynamically by posting content and citing others. Traditional citation analysis, rooted in bibliometrics, often fails to capture the nuance of these social learning environments. Specifically, standard algorithms like PageRank or HITS suffer from "sentiment blindness," treating every link as a positive endorsement. In reality, a citation in a discussion forum may represent a rebuttal or disagreement—a "negative edge." Furthermore, these models often exhibit a "direct-link bias," ignoring the indirect influence of citations that occur multiple steps away in the network.

To address these challenges, this paper introduces the Credit Author Ranking (CAR) algorithm. CAR is a graph-based approach designed to measure user credibility by accounting for both positive ("support") and negative ("against") citations, as well as indirect influence across the network.

Methodology: The CAR Algorithm

The researchers propose a multi-layered graph modeling approach to capture the complexity of user interactions:

  1. Basic Article Citation Model (BACM): A weighted directed graph where nodes are articles and edges represent citations. Weights are assigned as +1 for support and -1 for opposition.
  2. $k$-Extended Article Citation Model ($k$-EACM): This layer extends the BACM by including indirect citations up to $k$ hops away. To maintain accuracy, the weight of these indirect links decays as the distance increases, ensuring that "friend-of-a-friend" support is valued less than a direct citation.
  3. $k$-User Citation Model ($k$-UCM): This final layer maps article-level relationships back to the authors, creating a comprehensive user-to-user graph.

The core of the system is the CAR algorithm, inspired by Kleinberg’s HITS. CAR maintains two distinct scores for every user: a Credible Attribute ($x$), representing positive reputation, and a Trouble Attribute ($y$), representing negative or controversial reputation. The algorithm iteratively updates these weights based on the support and opposition links in the $k$-UCM graph. A user’s final credibility score ($z$) is calculated as the difference between these two attributes ($z = x - y$).

Experimental Setup

Due to the difficulty of obtaining labeled real-world data for credibility, the authors utilized simulated datasets. The population consisted of 300 users categorized into three types: 50 "Good" (G), 50 "Bad" (B), and 200 "Average" (A).

The interactions were governed by a "GBA matrix," which defines the probability of one user type supporting another. For instance, G-type users have a high probability (0.9) of supporting other G-type users. The simulations were run for 100 cycles to generate a dense citation structure.

GBA matrices for three simulation scenarios defining interaction probabilities between user types.

Key Results and Analysis

The CAR algorithm was evaluated by its ability to correctly identify G-type (good) users within the top rankings across three different simulation scenarios. The results demonstrate high accuracy and robustness.

Percentage of 'Good' users identified in the top rankings across three simulation scenarios.

In the most favorable simulation (Simulation 1), the algorithm achieved 100% accuracy in identifying G-type users within the top 30 rankings. Even when looking at the top 50—the total number of G-type users available—the algorithm captured between 78% and 86% of the high-quality contributors. Crucially, the algorithm successfully discriminated against "bad" and "average" users, pushing them down the rankings even if they were highly active.

Discussion and Conclusion

The CAR algorithm offers a significant advancement in measuring credibility within noisy online communities. By splitting reputation into separate "credible" and "trouble" vectors, it effectively models controversial users who might have high engagement but low reliability. Furthermore, the use of indirect links through the $k$-EACM allows the system to surface quality content that may not yet have a high volume of direct citations.

However, several limitations remain. The reliance on simulated data means that real-world human behaviors—such as sarcasm, trolling, or shifting alliances—have yet to be fully tested. Future work should focus on real-world validation, optimal parameter tuning for the decay function and $k$ value, and addressing the computational scalability of $k$-extended graphs in very large communities.

Summary updated

Abstract

Learning Villages (LV) is an E-learning platform for people's online discussions and frequently citing postings of one another. In this paper, we propose a novel method to rank credit authors in the LV system. We first propose a k-EACM graph to describe the article citation structure in the LV system. And then we build a weighted graph model k-UCM graph to reveal the implicit relationship between authors hidden behind the citations among their articles. Furthermore, we design a graph-based ranking algorithm, the Credit Author Ranking (CAR) algorithm, which can be applied to rank nodes in a graph with negative edges. Finally, we perform experimental evaluations by simulations. The results of evaluations illustrate that the proposed method works pretty well on ranking the credibility of users in the LV system.
BibTeX
@inproceedings{wei2007measuring,
  author = {Wei Wei and Jimmy Ho-Man Lee and Irwin King},
  title = {{Measuring credibility of users in an e-learning environment}},
  booktitle = {The Web Conference},
  year = {2007},
  doi = {10.1145/1242572.1242807},
}
Figure