Analyzing Wikipedia Membership Dataset and PredictingUnconnected Nodes in the Signed Networks

Zhihao Wu, Taoran Li, Ray Roman

Abstract

In the age of digital interaction, person-to-person relationships existing on social media may be different from the very same interactions that exist offline. Examining potential or spurious relationships between members in a social network is a fertile area of research for computer scientists—here we examine how relationships can be predicted between two unconnected people in a social network by using area under Precision-Recall curve and ROC. Modeling the social network as a signed graph, we compare Triadic model, Latent Information model and Sentiment model and use them to predict peer-to-peer interactions, first using a plain signed network, and second using a signed network with comments as context. We see that our models are much better than random model and could complement each other in different cases.

Introduction

Most social network analysis focuses on "unsigned" networks—simply whether a connection exists or not. However, real-world interactions are "signed": they can be supportive (positive) or antagonistic (negative). Predicting these signs is crucial for understanding community health, targeted advertising, and conflict resolution.

The challenge lies in the complexity of human interaction. Is a "friend of an enemy" likely to be an enemy (Balance Theory)? Or does a user’s hidden personality—perhaps they are just naturally grumpy or overly supportive—matter more? This research investigates whether simple sociological models can compete with data-rich Natural Language Processing (NLP) when predicting these interactions in a large, noisy dataset like Wikipedia.

Methodology

The study utilizes the Wikipedia Requests for Adminship (RfA) dataset from the Stanford SNAP Lab, consisting of approximately 10,000 nodes and 160,000 edges. Each edge represents a vote (support or oppose) and includes a written comment. To address the significant "sign imbalance" (an abundance of positive votes), the authors sampled the data to ensure a balanced training set.

Three primary modeling frameworks were compared:

  1. PSL - Triadic Model: Built on Probabilistic Soft Logic (PSL), this model uses rules derived from social science "Balance Theory." PSL allows for "truthiness" on a spectrum from 0 to 1, providing more flexibility than rigid Boolean logic.
  2. PSL - Latent Information Model: This approach attempts to uncover hidden node properties, such as whether a user is inherently "Active" or "Favorable," using Expectation-Maximization (EM) and Probabilistic Principal Component Analysis (PPCA).
  3. Sentiment Model: A classic NLP pipeline treating comments as a binary classification problem using Bag-of-Words, TF-IDF weighting, and Logistic Regression.
Logical rules for the Triadic model based on social balance theory.

Key Results

The Sentiment model emerged as the clear winner, demonstrating that the explicit text of an interaction contains significantly more predictive power than network structure alone. It required relatively less data to achieve high accuracy.

The Triadic model, while less accurate than the Sentiment model, achieved an AUC/negPR score of approximately 0.4 to 0.5. As shown in the performance metrics, this is significantly better than a random guess and provides a valuable baseline for structural prediction when text data is unavailable.

Performance metrics for the Triadic model across different evidence ratios.

The Latent Information model struggled due to "data sparsity." Most Wikipedia users did not have enough interactions for the model to accurately learn their hidden personality traits. Regarding parameter tuning for PSL models, Maximum Likelihood Estimation (MLE) was found to be the most effective method compared to other estimation techniques.

Discussion and Future Work

The primary limitations encountered were data sparsity and imbalance. The Wikipedia RfA process is overwhelmingly positive, making it difficult to train models to recognize rare negative interactions.

Future research should explore a hybrid approach: combining Sentiment Analysis with PSL. Instead of treating them as separate models, the authors propose creating new PSL rules that incorporate the similarity of users' communication content. This would allow a model to reason about both the content of speech and the topology of the social network simultaneously, potentially offering the best of both structural and linguistic analysis.

Summary updated

Abstract

In the age of digital interaction, person-to-person relationships existing on social media may be different from the very same interactions that exist offline. Examining potential or spurious relationships between members in a social network is a fertile area of research for computer scientists -- here we examine how relationships can be predicted between two unconnected people in a social network by using area under Precison-Recall curve and ROC. Modeling the social network as a signed graph, we compare Triadic model,Latent Information model and Sentiment model and use them to predict peer to peer interactions, first using a plain signed network, and second using a signed network with comments as context. We see that our models are much better than random model and could complement each other in different cases.
BibTeX
@misc{wu2021analyzing,
  author = {Zhihao Wu and Taoran Li and Ray Roman},
  title = {{Analyzing Wikipedia Membership Dataset and PredictingUnconnected Nodes in the Signed Networks}},
  year = {2021},
  eprint = {2110.09111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.09111},
}
Figure