TL;DR
Detecting whether a reply agrees or disagrees with a comment is notoriously difficult using text alone. This paper introduces a method to boost detection accuracy by building an "inductive social relation graph" based on historical interactions between users, allowing models to leverage social context (e.g., "this user usually agrees with that user") alongside state-of-the-art language models.
What Problem Does This Paper Address?
Identifying agreement, disagreement, or neutrality in online discussions—a task known as (dis)agreement detection—is a cornerstone of understanding social polarization and the spread of ideas. However, textual information is often insufficient. Sarcasm, lack of context, or domain-specific jargon can make a disagreeing reply look neutral or even agreeable to a machine.
Furthermore, models often struggle with cross-domain settings. A model trained to detect disagreement in political subreddits might fail when applied to climate change discussions because the linguistic "cues" for conflict change. The authors argue that while the topic changes, the social structure of how people interact remains a powerful, underutilized signal.
Methods
The authors propose a framework that combines natural language processing (NLP) with graph representation learning:
- Inductive Social Relation Graph: Instead of relying on platform-specific metadata (like "follows" or "retweets"), they construct a graph purely from comment-reply interactions. They define four relation types: Supporter, Opponent, Acquaintance, and Interaction.
- Graph Autoencoder (GAE): To handle these different relationship types, they use a Relational Graph Convolutional Network (RGCN) as an encoder. This allows the model to learn "author embeddings"—mathematical representations of users based on who they interact with. They use a DistMult decoder to pre-train these embeddings by predicting missing links in the social graph.
- Information Fusion: For a given comment-reply pair, the model extracts the text features using a Pre-trained Language Model (like BERT or RoBERTa). Simultaneously, it extracts social features from the graph for the two authors involved. These two vectors are concatenated and fed into a final classifier.
- Inductive Reasoning: The system is "inductive," meaning it can handle new authors or interactions not seen during the initial graph construction by looking at the local neighborhood of the authors in the social network.
Datasets
The study utilizes the DEBAGREEMENT dataset, a large-scale collection of 42,804 comment-reply pairs from Reddit. The dataset spans five diverse and often contentious subreddits:
- r/Brexit
- r/Climate
- r/BlackLivesMatter
- r/Republican
- r/Democrats
This variety allows the authors to test both "in-domain" performance (training and testing on the same topics) and "cross-domain" performance (training on four topics and testing on the fifth).
Key Insights and Ideas
- Social Context as a Tie-Breaker: In cases where text is ambiguous, the social graph acts as a "global" memory. If the model knows Author A and Author B have a history of opposing each other, it is more likely to correctly label a vague reply as "disagree."
- The "Friend of my Enemy" Logic: By using GNNs (Graph Neural Networks), the model can capture implicit relations. Even if two users haven't interacted directly, the model can infer their likely stance based on their mutual "enemies" or "allies."
- Long-Form Text Benefit: Interestingly, the social features provided the biggest performance boost for long-token comment-reply pairs. In long comments, the actual stance is often buried under layers of explanation; the social signal provides a clear "anchor" for the model.
- Domain Invariance: Social relations are more stable than linguistic patterns across different topics. This makes social information particularly effective for cross-domain generalization.
Key Results
- State-of-the-Art Performance: The proposed model (RoBERTa-rel) achieved the best results on the DEBAGREEMENT benchmark, outperforming text-only models like BERT-joint and RoBERTa-joint.
- In-Domain: RoBERTa-rel achieved a Macro-F1 score of 66.91%, a significant improvement over the standard RoBERTa-joint (65.67%).
- Cross-Domain: The model showed even more resilience here. For example, when testing on the "Democrats" subreddit after training on others, the social-aware model significantly outperformed text-only baselines, proving that social ties translate across topics better than keywords do.
- Ablation Success: The authors proved that both the pre-training of the graph and the specific "reconstruction loss" used in their autoencoder were vital to the model's success.
Limitations and Future Work
- Graph Sparsity: The authors noted that social interactions on platforms like Reddit are often "sparse"—many users only interact once and then disappear. The model's effectiveness is currently limited by how much historical data is available for a specific user.
- Temporal Dynamics: While the paper touches on time intervals, the authors suggest that social relations evolve. Future work could involve more complex modeling of how a "supporter" might turn into an "opponent" over time.
- Dataset Density: The authors call for more comprehensive datasets with denser interaction histories to further refine how "inductive" these social signals can truly be in real-time applications.
Summary updated