Abstract
(Dis)agreement detection aims to identify authors' attitudes—agree, disagree, or neutral—within textual interactions. Traditional methods rely heavily on textual features, which often struggle with context-dependent nuances and cross-domain generalization. This paper proposes a novel method that extracts inductive social relation information from comment-reply pairs to supplement textual analysis. By constructing an inductive social relation graph that considers historical discussions and author relationships, we leverage both a pre-trained language model for text and a Relational Graph Convolutional Network (RGCN) for social context. Experimental results on the DEBAGREEMENT benchmark demonstrate that our model achieves state-of-the-art performance in both in-domain and cross-domain tasks. Notably, social relations significantly boost detection accuracy for long-token interactions, highlighting the effectiveness of integrating social structure into linguistic models.
Introduction
Identifying whether a reply agrees or disagrees with a parent comment is a fundamental task in understanding social polarization and the dynamics of online discourse. However, textual information is frequently insufficient for accurate detection. Sarcasm, domain-specific jargon, and the lack of explicit context can make a disagreeing reply appear neutral or even agreeable to a machine learning model.
Furthermore, text-only models often fail in cross-domain settings. For instance, a model trained on political discussions may struggle with climate change debates because the linguistic cues for conflict differ between topics. This research addresses these limitations by introducing an "inductive social relation graph" built from historical user interactions, providing a stable, topic-invariant signal that complements linguistic data.
Methodology
The proposed framework integrates natural language processing (NLP) with graph representation learning through a multi-step process:
1. Inductive Social Relation Graph Construction
Unlike methods that require platform-specific metadata (such as "follows" or "likes"), this model constructs a graph purely from comment-reply interactions. It defines four distinct relation types based on interaction history: Supporter, Opponent, Acquaintance, and Interaction. This allows the model to capture the social "vibe" between users regardless of the specific topic being discussed.
2. Graph Representation Learning
To process these multi-relational structures, the authors utilize a Relational Graph Convolutional Network (RGCN) as an encoder. This component learns "author embeddings"—mathematical representations of users based on their interaction neighborhoods. A DistMult decoder is used to pre-train these embeddings by predicting missing links in the social graph, ensuring the model understands the underlying social fabric.
3. Information Fusion and Classification
For any given comment-reply pair, the system extracts:
- Textual Features: Encoded using a Pre-trained Language Model (PLM) such as BERT or RoBERTa.
- Social Features: Extracted from the inductive graph for both the commenter and the replier.
These vectors are concatenated and passed through a final classifier. Because the system is "inductive," it can generate embeddings for new authors or interactions not seen during training by analyzing their local neighborhood in the social network.
Experimental Setup
The model was evaluated using the DEBAGREEMENT dataset, which contains 42,804 comment-reply pairs from five contentious Reddit subreddits: r/Brexit, r/Climate, r/BlackLivesMatter, r/Republican, and r/Democrats. This dataset allows for rigorous testing of:
- In-domain performance: Training and testing on the same subreddit.
- Cross-domain performance: Training on four subreddits and testing on the fifth, unseen topic.
Key Insights and Results
State-of-the-Art Accuracy
The proposed model, RoBERTa-rel, outperformed text-only baselines across the board. In in-domain tests, it achieved a Macro-F1 score of 66.91%, compared to 65.67% for a standard RoBERTa-joint model.
Social Context as a Tie-Breaker
The social graph acts as a "global memory." If the text is ambiguous but the model knows Author A and Author B have a history of disagreement, it can use that social signal to correctly identify a "disagree" label. This was particularly evident in long-token pairs, where the core stance is often buried under complex explanations; here, the social anchor provided the most significant performance gains.
Cross-Domain Robustness
Social relations proved to be more stable than linguistic patterns across different domains. When testing on the "Democrats" subreddit after training on other topics, the social-aware model significantly outperformed text-only models. This suggests that while the words people use to argue change between politics and science, the way they interact with allies and opponents remains consistent.
Conclusion
By incorporating inductive social relations, this research demonstrates that (dis)agreement detection can be significantly improved without requiring additional platform metadata. The integration of RGCN-based social embeddings with PLM-based textual features provides a robust solution for both ambiguous text and cross-domain applications. Future work will look to address graph sparsity and the temporal evolution of social relations, exploring how "supporters" might transition into "opponents" over time.
Summary updated