Few-Shot (Dis)Agreement Identification in Online Discussions with Regularized and Augmented Meta-Learning

Yuanyuan Lei, Ruihong HuangEMNLP 2022 · DOI

Introduction

Online discussions on emergent and controversial topics—ranging from policy shifts to global health crises—generate vast amounts of data reflecting public sentiment. Identifying agreement and disagreement within these exchanges is crucial for understanding social dynamics. However, traditional supervised learning models require extensive labeled datasets for every new topic, creating a "cold-start" problem when novel issues arise.

In their EMNLP 2022 paper, Yuanyuan Lei and Ruihong Huang address this challenge by introducing a meta-learning framework designed for few-shot (dis)agreement identification. Their approach enables models to generalize to unseen topics using only a handful of labeled examples.

The Challenge: Domain Shift and Data Scarcity

The primary obstacle in opinion mining for emergent topics is the combination of domain shift and data scarcity. Models trained on historical data (e.g., old political forums) often fail on new topics because they lack exposure to topic-specific jargon. Furthermore, standard fine-tuning on small datasets (e.g., 5 or 10 examples) frequently leads to overfitting, where the model memorizes specific labels rather than learning generalizable patterns of agreement.

Methodology: A Dual-Pronged Meta-Learning Approach

The authors propose a metric-based meta-learning architecture that learns "how to learn" agreement across diverse domains.

1. Base Architecture and Episodic Training

The system utilizes a RoBERTa backbone coupled with a Bi-LSTM to generate embeddings for (Quote, Response) pairs. The model is trained using an episodic strategy: in each episode, it is provided with a small "support set" to define the classes (Agreement, Disagreement, Neutral) for a specific task, and then evaluated on a "query set."

The proposed meta-learning architecture, incorporating attentive class embeddings and lexicon-based regularization.

2. Attentive Class Embedding

To represent each class effectively, the model employs an attention mechanism. Rather than simple averaging, this mechanism weights support examples based on their relevance to the current query instance, allowing for more nuanced class representations.

3. Lexicon-Based Regularization

Regardless of the topic, certain "anchor" words consistently signal agreement or disagreement. The authors compiled a lexicon of domain-independent indicators across four categories: cue words, agreement, denial, and hedges.

Categories and examples of domain-independent indicators used for lexicon-based regularization.

To leverage these universal cues, a regularization loss is introduced. This loss forces the model to prioritize these lexicon words by calculating the gradient of the output with respect to word embeddings, ensuring the model focuses on linguistically significant "saliency" markers during training.

4. Domain-Aware Task Augmentation

To help the model handle topic-specific expressions, the framework uses domain-aware task augmentation. During training, support and query sets within a single episode are drawn from the same sub-topic. This ensures the model learns to distinguish agreement patterns within a consistent semantic context.

Experimental Setup and Datasets

The model was trained on established benchmarks like the Internet Argument Corpus (IAC) and Agreement by Create Debaters (ABCD). It was evaluated on the Wikipedia Talk Pages (AWTP) dataset and two new Reddit-based benchmarks introduced by the authors: SubReddit-MaskMandate and SubReddit-CovidVaccine. These represent "emergent" topics that did not exist when the training datasets were compiled.

Key Findings

The proposed model (Meta + aug + reg) consistently outperformed standard fine-tuning and prompt-based baselines:

  • Performance Gains: In 2-way classification, the model achieved Macro F1 scores in the high 70s, a 4-6 point improvement over standard RoBERTa.
  • Robustness in 3-Way Tasks: For the more complex Agreement vs. Disagreement vs. Neutral task, the model showed gains of up to 4.4% over the best baseline.
  • Ablation Results: Removing either lexicon regularization or task augmentation resulted in significant performance drops, confirming that both universal linguistic cues and domain-specific awareness are vital for few-shot success.

Conclusion

By combining the "physics" of agreement (universal lexicon cues) with domain-specific awareness (task augmentation), this meta-learning framework provides a robust solution for identifying opinions in the wild. While the computational cost of second-order gradients remains a challenge, the ability to rapidly adapt to emergent topics marks a significant step forward for real-time social media analysis.

Summary updated

BibTeX
@inproceedings{lei2022few,
  author = {Yuanyuan Lei and Ruihong Huang},
  title = {{Few-Shot (Dis)Agreement Identification in Online Discussions with Regularized and Augmented Meta-Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.409},
}
Figure