Abstract
Stance detection aims to identify whether the author of a text is in favor of, against, or neutral to a given target. The main challenge of this task comes two-fold: few-shot learning resulting from the varying targets and the lack of contextual information of the targets. Existing works mainly focus on solving the second issue by designing attention-based models or introducing noisy external knowledge, while the first issue remains under-explored. In this paper, inspired by the potential capability of pre-trained language models (PLMs) serving as knowledge bases and few-shot learners, we propose to introduce prompt-based fine-tuning for stance detection. PLMs can provide essential contextual information for the targets and enable few-shot learning via prompts. Considering the crucial role of the target in stance detection task, we design target-aware prompts and propose a novel verbalizer. Instead of mapping each label to a concrete word, our verbalizer maps each label to a vector and picks the label that best captures the correlation between the stance and the target. Moreover, to alleviate the possible defect of dealing with varying targets with a single hand-crafted prompt, we propose to distill the information learned from multiple prompts. Experimental results show the superior performance of our proposed model in both full-data and few-shot scenarios.
Introduction
Stance detection is a nuanced task often conflated with sentiment analysis. However, while sentiment analysis identifies general emotional polarity, stance detection requires determining an author's position toward a specific target (e.g., "Climate Change" or "Atheism"). This presents two major hurdles: the few-shot problem, where new targets constantly emerge without sufficient labeled data, and contextual gaps, where the target itself may not explicitly appear in the text.
Traditional methods often rely on complex attention mechanisms or external knowledge bases that struggle to generalize. To address these issues, we introduce TAPD (Target-Aware Prompt Distillation), a framework that reformulates stance detection as a prompt-based learning task, leveraging the latent knowledge within Pre-trained Language Models (PLMs).
Methodology: The TAPD Framework
TAPD improves upon standard prompt-based learning through three core innovations designed to capture the interplay between text and target.
Target-Aware Prompts and Vector-Based Verbalizer
Instead of standard classification, TAPD treats stance detection as a cloze (fill-in-the-blank) task. We utilize three distinct prompt patterns to view the problem through different lenses: sentiment analysis, natural language inference, and the stance task itself.
Crucially, we move away from traditional verbalizers that map labels to fixed words (e.g., "Favor" $\rightarrow$ "yes"). Because a single word cannot capture the nuanced relationship between every possible target and stance, we propose a Vector-Based Verbalizer. This maps labels to trainable stance vectors which are concatenated with the target's representation. This allows the model to learn a unique, target-specific meaning for each stance label.
Multi-Prompt Distillation
Recognizing that no single prompt pattern is perfect, TAPD employs a sequential Knowledge Distillation process. A "student" model is trained on one prompt while simultaneously learning from a "teacher" model trained on a previous prompt. This distillation allows the final model to benefit from the collective knowledge of multiple perspectives, leading to more robust performance across varying targets.
Experimental Evaluation
Datasets
We evaluated TAPD on two benchmark datasets:
- SemEval-2016 Task 6: English tweets covering five targets (e.g., Atheism, Feminist Movement).
- UKP Dataset: Over 25,000 argument sentences across eight diverse topics like the Death Penalty and Gun Control.
Key Results
TAPD demonstrated superior performance in both full-data and few-shot scenarios:
- Full Data: On SemEval-2016, TAPD achieved an $F1_{avg}$ of 66.87, significantly outperforming standard BERT fine-tuning (59.09) and specialized models like Stancy (62.62).
- Few-Shot Scenarios: In extreme cases with only 2 labeled examples per target, TAPD maintained a significant lead. On the UKP dataset, it achieved a $MacF1_{avg}$ of 33.71, compared to the best baseline at 32.75.
- Cross-Target Transfer: The model showed strong capabilities in transferring knowledge between related targets (e.g., training on "Hillary Clinton" and testing on "Donald Trump").
Conclusion
TAPD demonstrates that PLMs can serve as effective few-shot learners for stance detection when guided by target-aware prompts and vector-based verbalizers. By distilling knowledge across multiple prompt patterns, the model captures complex target-stance relationships that traditional methods miss. Future work will explore automated prompt-tuning to remove the need for manual template design.
Summary updated