Improving (Dis)agreement Detection with Inductive Social Relation Information From Comment-Reply Interactions

Yun Luo, Zihan Liu, Stan Z. Li, Yue Zhang

TL;DR

Detecting whether a reply agrees or disagrees with a comment is notoriously difficult using text alone. This paper introduces a method to boost detection accuracy by building an "inductive social relation graph" based on historical interactions between users, allowing models to leverage social context (e.g., "this user usually agrees with that user") alongside state-of-the-art language models.

What Problem Does This Paper Address?

Identifying agreement, disagreement, or neutrality in online discussions—a task known as (dis)agreement detection—is a cornerstone of understanding social polarization and the spread of ideas. However, textual information is often insufficient. Sarcasm, lack of context, or domain-specific jargon can make a disagreeing reply look neutral or even agreeable to a machine.

Furthermore, models often struggle with cross-domain settings. A model trained to detect disagreement in political subreddits might fail when applied to climate change discussions because the linguistic "cues" for conflict change. The authors argue that while the topic changes, the social structure of how people interact remains a powerful, underutilized signal.

Methods

The authors propose a framework that combines natural language processing (NLP) with graph representation learning:

  1. Inductive Social Relation Graph: Instead of relying on platform-specific metadata (like "follows" or "retweets"), they construct a graph purely from comment-reply interactions. They define four relation types: Supporter, Opponent, Acquaintance, and Interaction.
  2. Graph Autoencoder (GAE): To handle these different relationship types, they use a Relational Graph Convolutional Network (RGCN) as an encoder. This allows the model to learn "author embeddings"—mathematical representations of users based on who they interact with. They use a DistMult decoder to pre-train these embeddings by predicting missing links in the social graph.
  3. Information Fusion: For a given comment-reply pair, the model extracts the text features using a Pre-trained Language Model (like BERT or RoBERTa). Simultaneously, it extracts social features from the graph for the two authors involved. These two vectors are concatenated and fed into a final classifier.
  4. Inductive Reasoning: The system is "inductive," meaning it can handle new authors or interactions not seen during the initial graph construction by looking at the local neighborhood of the authors in the social network.

Datasets

The study utilizes the DEBAGREEMENT dataset, a large-scale collection of 42,804 comment-reply pairs from Reddit. The dataset spans five diverse and often contentious subreddits:

  • r/Brexit
  • r/Climate
  • r/BlackLivesMatter
  • r/Republican
  • r/Democrats

This variety allows the authors to test both "in-domain" performance (training and testing on the same topics) and "cross-domain" performance (training on four topics and testing on the fifth).

Key Insights and Ideas

  • Social Context as a Tie-Breaker: In cases where text is ambiguous, the social graph acts as a "global" memory. If the model knows Author A and Author B have a history of opposing each other, it is more likely to correctly label a vague reply as "disagree."
  • The "Friend of my Enemy" Logic: By using GNNs (Graph Neural Networks), the model can capture implicit relations. Even if two users haven't interacted directly, the model can infer their likely stance based on their mutual "enemies" or "allies."
  • Long-Form Text Benefit: Interestingly, the social features provided the biggest performance boost for long-token comment-reply pairs. In long comments, the actual stance is often buried under layers of explanation; the social signal provides a clear "anchor" for the model.
  • Domain Invariance: Social relations are more stable than linguistic patterns across different topics. This makes social information particularly effective for cross-domain generalization.

Key Results

  • State-of-the-Art Performance: The proposed model (RoBERTa-rel) achieved the best results on the DEBAGREEMENT benchmark, outperforming text-only models like BERT-joint and RoBERTa-joint.
  • In-Domain: RoBERTa-rel achieved a Macro-F1 score of 66.91%, a significant improvement over the standard RoBERTa-joint (65.67%).
  • Cross-Domain: The model showed even more resilience here. For example, when testing on the "Democrats" subreddit after training on others, the social-aware model significantly outperformed text-only baselines, proving that social ties translate across topics better than keywords do.
  • Ablation Success: The authors proved that both the pre-training of the graph and the specific "reconstruction loss" used in their autoencoder were vital to the model's success.

Limitations and Future Work

  • Graph Sparsity: The authors noted that social interactions on platforms like Reddit are often "sparse"—many users only interact once and then disappear. The model's effectiveness is currently limited by how much historical data is available for a specific user.
  • Temporal Dynamics: While the paper touches on time intervals, the authors suggest that social relations evolve. Future work could involve more complex modeling of how a "supporter" might turn into an "opponent" over time.
  • Dataset Density: The authors call for more comprehensive datasets with denser interaction histories to further refine how "inductive" these social signals can truly be in real-time applications.

Summary updated

Abstract

(Dis)agreement detection aims to identify the authors’ attitudes or positions (agree, disagree, neutral) towards a specific text. It is limited for existing methods merely using textual information for identifying (dis)agreements, especially for cross-domain settings. Social relation information can play an assistant role in the (dis)agreement task besides textual information. We propose a novel method to extract such relation information from (dis)agreement data into an inductive social relation graph, merely using the comment-reply pairs without any additional platform-specific information. The inductive social relation globally considers the historical discussion and the relation between authors. Textual information based on a pre-trained language model and social relation information encoded by pre-trained RGCN are jointly considered for (dis)agreement detection. Experimental results show that our model achieves state-of-the-art performance for both the in-domain and cross-domain tasks on the benchmark – DEBAGREEMENT. We find social relations can boost the performance of the (dis)agreement detection model, especially for the long-token comment-reply pairs, demonstrating the effectiveness of the social relation graph. We also explore the effect of the knowledge graph embedding methods, the information fusing method, and the time interval in constructing the social relation graph, which shows the effectiveness of our model.
BibTeX
@inproceedings{luo2023improving,
  author = {Yun Luo and Zihan Liu and Stan Z. Li and Yue Zhang},
  title = {{Improving (Dis)agreement Detection with Inductive Social Relation Information From Comment-Reply Interactions}},
  booktitle = {The Web Conference},
  year = {2023},
  doi = {10.1145/3543507.3583314},
}

Loading related content...

Figure