TL;DR
This paper introduces LKI-BART, a novel approach that leverages Large Language Models (LLMs) to extract contextual knowledge for stance detection. By injecting this LLM-generated knowledge into a BART model and employing a prototypical contrastive learning scheme, LKI-BART achieves state-of-the-art performance on challenging zero-shot and cross-target stance detection tasks.
What Problem Does This Paper Address?
Stance detection, the task of identifying an author's attitude towards a specific target in a text, is crucial for various applications like analyzing public opinion and detecting misinformation. Traditional methods often struggle with unseen targets (zero-shot) or related but different targets (cross-target) because they primarily rely on target-specific background information. This can lead to a disconnect between the text and the target, potentially misguiding the model. While recent work has explored using LLMs for stance detection, there's a need for a more systematic way to integrate LLM-derived knowledge that explicitly captures the relationship between the text and the target, going beyond just target background.
Methods
The core innovation of LKI-BART lies in its two-pronged approach:
- LLM-Driven Knowledge Injection: The authors use a carefully crafted prompt to instruct LLMs (specifically GPT-3.5-turbo) to extract relevant information. This prompt encourages the LLM to identify keywords, analyze implied emotions and rhetorical devices, and provide a brief stance analysis (without a definitive conclusion to mitigate hallucination). This "LLM-driven knowledge" is designed to be more specific and context-rich than traditional background knowledge.
- Prototypical Contrastive Learning: To effectively bridge the gap between the LLM-generated knowledge and stance labels, the authors adapt the BART (Bidirectional and Auto-Regressive Transformer) model. The stance detection task is reframed as a conditional stance label generation problem. Crucially, they decouple the stance embedding from the BART decoder and project it into a lower-dimensional space. This projected embedding is then regularized using a novel prototypical contrastive loss. This loss encourages intra-class similarity (embeddings of the same stance label clustering together) and discourages inter-class similarity, effectively creating a well-structured feature space for stance representations.
Datasets
The paper evaluates LKI-BART on two benchmark datasets:
- VAST (Zero-Shot Stance Detection): This dataset, derived from the New York Times "Room for Debate" section, contains comments with various targets and stance labels (Favor, Against, Neutral). It's used for zero-shot evaluation, meaning targets seen during training are not present in the test set.
- P-Stance (Cross-Target Stance Detection): This dataset consists of political tweets annotated with stances ("Favor" or "Against") towards three specific targets: "Donald Trump," "Joe Biden," and "Bernie Sanders." It's used to evaluate the model's ability to generalize to related but unseen targets.
Key Insights and Ideas
- LLMs as Knowledge Extractors: The paper demonstrates that LLMs, when prompted effectively, can go beyond simple fact retrieval to extract nuanced relationships between text and targets, which is vital for complex tasks like stance detection.
- The Power of Explicit Text-Target Relationships: The study highlights that focusing solely on target background knowledge is insufficient. Explicitly capturing how the text relates to the target, as facilitated by LLM-driven knowledge, significantly improves performance.
- Prototypical Contrastive Learning for Robustness: The proposed prototypical contrastive learning scheme is more robust than traditional supervised contrastive learning. By using class-wise prototypes, it helps to regularize the feature space and makes the model less susceptible to in-batch outliers.
- Generative Models for Stance Detection: Reframing stance detection as a conditional generation task using BART proves effective, allowing the model to leverage the rich contextual information from LLM-driven knowledge more holistically.
Key Results
- State-of-the-Art Performance: LKI-BART achieves state-of-the-art results on the VAST dataset for zero-shot stance detection, outperforming previous methods.
- Strong Cross-Target Performance: On the P-Stance dataset, LKI-BART significantly outperforms existing models, showing improvements of 10-15 F1-points.
- Ablation Studies Confirm Effectiveness: Ablation studies confirm that each component of LKI-BART (LLM-driven knowledge, BART backbone, and prototypical contrastive learning) contributes to the overall performance gain. The "Analyse stance" part of the LLM prompt was particularly impactful.
- Feature Visualization: t-SNE visualizations show that LKI-BART produces more clustered and well-separated stance embeddings compared to a vanilla BART model, indicating better feature representation.
Limitations and Future Work
- LLM Dependency: The performance of LKI-BART is inherently tied to the capabilities of the LLM used. The authors used GPT-3.5-turbo and acknowledge that exploring other LLMs and prompt formats could yield further improvements.
- Budget and Time Constraints: The experiments were conducted with a specific LLM and within certain time and budget limitations. Future work could involve comparing a wider range of LLMs.
- Potential for Bias: The authors acknowledge that LLMs, including ChatGPT, can produce biased information. While they did not apply additional processing to mitigate this, they emphasize that their training data comes from publicly available sources.
- Reproducibility: The authors have made their code publicly available to facilitate future research and reproduction.
Summary updated