TL;DR
This paper tackles the issue of stance detection models being brittle to minor text variations. They propose using Contrastive Learning (CL) to train more robust sentence representations, showing that this approach significantly improves resilience against adversarial attacks and performs well on imbalanced datasets.
What Problem Does This Paper Address?
Stance detection, the task of identifying an author's position (e.g., support, against) on a given topic, is crucial for understanding public opinion and discourse. However, existing models are often fragile. Even small changes to input text, such as spelling errors, word substitutions, or grammatical variations, can drastically alter a model's prediction, leading to unreliable results. This lack of robustness is a significant barrier to deploying these models in real-world scenarios where text is often noisy and imperfect. The paper aims to make stance detection systems more resilient to these adversarial perturbations by learning more robust representations.
Methods
The core of the proposed methodology is Contrastive Learning (CL). CL works by learning representations where similar examples are pulled closer together in an embedding space, while dissimilar examples are pushed apart. In this context, "similarity" is defined by the stance label. The authors explore various strategies for constructing these "triplets" (anchor, positive, negative) to feed into the CL framework.
They also incorporate Masked Language Modeling (MLM) as a token-level objective, which is a standard technique for pre-training transformer models. The combined loss from CL and MLM is used to train a transformer model (specifically, DistilRoBERTa). This pre-trained model is then fine-tuned on stance detection datasets.
To measure robustness, they employ three types of adversarial attacks:
- Spelling Errors: Introducing misspellings by altering or swapping letters in words.
- Adding Tautology: Appending phrases like "False is not true and" to sentences.
- Synonym Replacement: Replacing frequent words with their synonyms.
They also introduce the concept of Partial Perturbation (PP), where only instances correctly classified by the model on the original test set are perturbed. This allows for a more focused evaluation of the model's resilience to specific types of errors.
Datasets
The paper uses seven publicly available stance detection datasets:
- DebateForum (DF)
- SemEval2016 (SE)
- ARC
- Perspectrum
- FNC-1
- KSD-Biden
- KSD-Trump
These datasets cover various domains and exhibit different characteristics, including varying sizes and class imbalances. The FNC-1 dataset is noted as being particularly imbalanced. The authors focus on binary stance labels (support/against, agree/disagree, favor/against, etc.) and filter datasets accordingly.
Key Insights and Ideas
- Contrastive Learning is Key to Robustness: The study strongly suggests that CL is a powerful tool for improving the robustness of stance detection models against adversarial attacks. By explicitly learning to group similar-stance sentences and separate dissimilar ones, the models become less sensitive to superficial text variations.
- Harder Examples Lead to Better Resilience: Strategies that incorporate "hard" examples (e.g., ModelHard, ModelH&E) in the contrastive learning process tend to yield more robust models, especially when dealing with mixed topics or imbalanced datasets. This implies that forcing the model to learn from challenging distinctions is beneficial.
- Effectiveness on Imbalanced Datasets: The proposed CL-based models show a notable advantage on imbalanced datasets like FNC-1, outperforming the baseline significantly in terms of both F1-score and resilience. This is a critical finding for real-world applications where data imbalance is common.
- Partial Perturbation Reveals Deeper Resilience: The Partial Perturbation setup highlights the true resilience of the models by focusing on correctly classified instances. This method revealed significant resilience gains for the CL-based models.
Key Results
The paper demonstrates that their proposed CL-based models consistently outperform a baseline DistilRoBERTa model fine-tuned directly on stance datasets.
- Resilience: Across various setups (Mixed Topics, Individual Topics, and their Partial Perturbation variants), the CL-based models (especially ModelHard and ModelH&E) exhibit significantly higher average resilience scores compared to the baseline. For instance, in the Mixed Topics + PP setup, the average resilience was 93.77 ± 4.06 for ModelH&E, compared to 90.87 ± 9.2 for the baseline.
- F1-Score: While the primary focus is robustness, the CL models also achieve comparable or superior F1-scores on original, non-perturbed datasets. In the Mixed Topics setup, ModelH&E achieved the best average resilience while maintaining a comparable average F1-score to the baseline.
- Imbalanced Datasets: The models show particular strength on imbalanced datasets like FNC-1, where they consistently outperform the baseline. For example, on FNC-1 in the Mixed Topics setup, ModelH&E achieved a resilience of 97.95 (compared to the baseline's 93.77) and an F1-score of 52.2 (compared to the baseline's 48.86).
Limitations and Future Work
- Binary Stance Focus: The current methodology is primarily designed for binary stance detection (e.g., for/against). The authors acknowledge that real-world social media discussions often involve more than two stances (e.g., for/against/neither, or support/refute/comment). Extending the framework to handle multi-class stance detection is a natural next step.
- Limited Adversarial Attacks: While three types of adversarial attacks were used, the paper could explore a wider range of perturbations to further test the model's robustness.
- Dataset Specificity: The performance gains might vary across different types of datasets and domains. Further investigation into how the CL strategies generalize to diverse text types and topics would be valuable.
Summary updated