Chain of Stance: Stance Detection with Large Language Models

Junxia Ma, Changjiang Wang, Hanwen Xing, Dongming Zhao, Yazhou Zhang

TL;DR

Researchers have introduced Chain of Stance (CoS), a novel prompting method that transforms Large Language Models (LLMs) into expert stance detectors. By decomposing the complex task of identifying an author's attitude into a structured sequence of reasoning steps—covering context, intent, and emotion—CoS achieves state-of-the-art results on benchmark datasets, significantly outperforming traditional fine-tuning and "black-box" prompting methods.

What Problem Does This Paper Address?

Stance detection is the task of determining whether the author of a text is in favor of, against, or neutral toward a specific target (e.g., "Climate Change" or "Atheism"). While it sounds simple, it is notoriously difficult for machines because human language is rife with sarcasm, cultural nuances, and indirect expressions.

Current approaches generally fall into two camps:

  1. Fine-tuning: Training models on massive datasets, which is computationally expensive and time-consuming.
  2. Direct Prompting: Asking an LLM for a label (e.g., "Is this tweet pro-choice?"). This often fails to capture the underlying logic or "thought process" required for nuanced social media text.

The authors address this gap by leveraging the "encyclopedic" knowledge of LLMs through a structured reasoning chain, making the model's decision-making process both more accurate and more interpretable.

Methods

The core innovation is the Chain of Stance (CoS) framework. Inspired by "Chain of Thought" (CoT) prompting—which helps models solve math problems by thinking step-by-step—CoS applies a similar logic to social linguistics.

Instead of asking for a final judgment immediately, the framework guides the LLM through six specific steps:

  1. Contextual Understanding: Identifying the topic, author identity, and socio-cultural background.
  2. Main Idea Interpretation: Extracting the core viewpoints and intentions.
  3. Emotional Analysis: Analyzing tone (e.g., sarcastic, affirmative) and rhetorical devices.
  4. Stance Comparison: Explicitly comparing the text against the three possible labels (Favor, Against, None).
  5. Logical Inference: Checking the consistency and rationality of the gathered evidence.
  6. Final Decision: Synthesizing all previous steps into a final stance polarity.

Unlike standard CoT, which allows the model to "think freely," CoS enforces a specific structure that prioritizes the relationship between human emotion and stance.

Datasets

The researchers evaluated their method using the SemEval-2016 Task 6 dataset ("Stance Detection in Tweets"). This is a standard benchmark containing 4,163 tweets across five targets:

  • Hillary Clinton (HC)
  • Feminist Movement (FM)
  • Legalization of Abortion (LA)
  • Atheism (A)
  • Climate Change is a Real Concern (CC)

The models tested included Mistral-7B, Qwen1.5-7B, LLaMA 3-8B, and LLaMA 2-7B.

Key Insights and Ideas

  • Structure Beats Scale: Interestingly, the Mistral-7B model using CoS outperformed much larger models and complex fine-tuned baselines. This suggests that how we ask an LLM to think is often more important than the raw number of parameters.
  • Emotion is the Key: The authors found that explicitly forcing the model to analyze "emotional inclination" and "rhetorical devices" (like sarcasm) was the primary driver in improving accuracy for social media text.
  • Interpretability as a Feature: Because the model outputs its reasoning for each of the six steps, researchers can see why a model misclassified a tweet (e.g., it understood the sentiment but failed to grasp the cultural context).

Key Results

The CoS method achieved state-of-the-art (SOTA) performance:

  • Zero-Shot Setting: Achieved a leading F1 score of 76.43 (using Mistral-7B).
  • Few-Shot Setting: Achieved an F1 score of 79.84, surpassing existing specialized models like CoSD and MB-Cal.
  • Ablation Success: Removing the "Chain of Stance" steps (the "w/o CoS" test) led to a significant drop in performance across all models, proving that the sequential reasoning process is the "secret sauce" of the performance gains.

Limitations and Future Work

The authors identify one primary trade-off: Computational Efficiency. Because the CoS prompt is much longer than a standard prompt and requires the model to generate multiple intermediate "assertions" before reaching a conclusion, it increases the time and cost per inference.

Future work will focus on:

  • Optimizing the length of the prompts to improve speed without sacrificing accuracy.
  • Exploring how to further leverage the "expert" personas within the chain to handle even more complex, domain-specific knowledge.

Summary updated

Abstract

Stance detection is an active task in natural language processing (NLP) that aims to identify the author’s stance towards a particular target within a text. Given the remarkable language understanding capabilities and encyclopedic prior knowledge of large language models (LLMs), how to explore the potential of LLMs in stance detection has received significant attention. Unlike existing LLM-based approaches that focus solely on fine-tuning with large-scale datasets, we propose a new prompting method, called Chain of Stance (CoS). In particular, it positions LLMs as expert stance detectors by decomposing the stance detection process into a series of intermediate, stance-related assertions that culminate in the final judgment. This approach leads to significant improvements in classification performance. We conducted extensive experiments using four SOTA LLMs on the SemEval 2016 dataset, covering the zero-shot and few-shot learning setups. The results indicate that the proposed method achieves state-of-the-art results with an F1 score of 79.84 in the few-shot setting.
BibTeX
@inproceedings{ma2024chain,
  author = {Junxia Ma and Changjiang Wang and Hanwen Xing and Dongming Zhao and Yazhou Zhang},
  title = {{Chain of Stance: Stance Detection with Large Language Models}},
  booktitle = {NLPCC},
  year = {2024},
  url = {https://openreview.net/forum?id=371gKAqdO3},
}

Loading related content...

Figure