Stance detection asks whether a text is for, against, or neutral toward a specific target, such as a bill or a candidate. Teams increasingly train stance classifiers on comments written by a large language model (LLM) that was prompted to argue one side. The appeal is real: The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions shows that synthetic comments plus a quarter of the real labels can beat a model trained on all real labels. The weak point is that the prompt's requested stance is treated as ground truth. Limited Effectiveness of LLM-based Data Augmentation for COVID-19 Misinformation Stance Detection shows that generators often "flip" the task, writing a rebuttal when asked for support, or refuse outright. These errors are not random. They are plausible, coherent texts carrying the wrong label.
Our claim is that these flips are predictable from one property of the request: whether the requested stance and the natural emotional tone of the text point the same way. A Favor comment that is furious ("I can't believe they killed the climate bill") is sentiment-incongruent. We hypothesize that generators flip or soften exactly these cases most often. We also hypothesize that standard label-cleaning methods, which group examples by similarity in text-embedding space, make things worse. Those embeddings cluster by emotional tone, so the cleaner "corrects" the rare valid incongruent examples toward the sentiment-implied label. That reinforces the shortcut stance models are already known to take. We propose to measure this failure directly and to repair labels using neighbors found in a representation that has been made insensitive to sentiment. Our test is whether this repair recovers accuracy on the incongruent slice of real test data, where synthetic training should help most and currently helps least.
Core Claim
Synthetic stance data carries structured label noise: errors that depend on the content of each instance and fall on semantically nearby labels rather than random ones. In stance, the structure is organized by sentiment-stance congruence. Write $r_{t,s,c}$ for the probability that a sample generated for target $t$ with requested stance $s$ has a different true stance, where $c$ indicates whether $s$ is congruent with the dominant sentiment of the generated text. We predict
$$ r_{t,s,\text{incongruent}} \gg r_{t,s,\text{congruent}}, $$
with the gap largest for controversial or safety-sensitive targets. The motivation comes from two observations. Refusal and task flipping concentrate on counter-attitudinal requests (Limited Effectiveness…). Synthetic text is also emotionally restrained compared with real discourse (The Power of LLM-Generated Synthetic Data…), so the generator struggles most when the stance must be carried by emotion that points the "wrong" way.
The second half of the claim is about repair. If label repair finds each sample's neighbors in a space where sentiment has been removed and the target is conditioned on, neighbor voting reflects stance rather than tone. Repair then fixes flips without erasing valid incongruent examples. Neither the denoising work nor the stance-debiasing work in the file does this. SiDyP (Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement) repairs LLM-assigned labels using training dynamics and nearest neighbors in plain text-embedding space. It was evaluated only on single-text tasks such as TREC and 20News, where no secondary attribute masquerades as the label. FACTUAL (Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration) attacks the sentiment shortcut directly, but it assumes its own LLM-written counterfactuals are correctly labeled, and its authors flag this as open.
Research Program
The first component is an audit, which we treat as a contribution in its own right. We will generate synthetic comments for the targets of X-Stance, SemEval-2016 Task 6, P-Stance, and the COVID-19 misinformation benchmarks, using the generator families in the file: Mistral-7B, LLaMA-3.1-8B, and GPT-4o. Each request specifies a target, a stance, and an emotional register that is either congruent or incongruent with that stance. Human annotators will label the true stance and dominant sentiment of a stratified sample of roughly 3,000 outputs. This gives the first per-cell estimates of $r_{t,s,c}$. It also tells us whether incongruent requests that do succeed come out emotionally muted, which would be a form of style collapse rather than a label flip.
The second component is a diagnosis of existing cleaners. We will run nearest-neighbor label repair in the style of SiDyP and of NLAP (Neighbor-aware Label Refinement), which recovers missing true labels by querying neighbors under the assumption that similar instances share labels. We then count how often each method flips an audited-correct incongruent sample to the wrong label versus how often it fixes a genuine flip. Gap 2 in the file predicts the first rate is high. If that prediction fails, the proposal shrinks to an auditing paper.
The third component is the repair method, which uses three ingredients.
First, each synthetic sample receives a candidate label set rather than a single label, following partial label learning (training when each example comes with a set of possible labels, one of which is assumed correct). Low-risk congruent samples keep their requested label alone. High-risk samples, meaning incongruent requests or sensitive targets, get the set {requested stance, its opposite, neutral}. The risk cells are set by the audit, not by hand.
Second, neighbors are retrieved in a target-conditioned encoder trained to be invariant to sentiment. We train it on a small real seed set (a few hundred labels per target, matching the low-label regime of SQBC). Two constraints shape it. Pairs of texts with the same stance but rewritten sentiment, generated as in FACTUAL's non-causal counterfactuals, must map close together. An adversarial sentiment predictor on the representation must fail.
Third, the candidate set is narrowed by neighbor voting in that space, with the iterative pruning used by NLAP. Reduction-based alternatives such as RPLG (Reduction-based Pseudo-label Generation) are a natural fit because there are only three classes, but RPLG needs a clean validation set and its per-class auxiliary model adds cost, so we test it as a variant rather than the default.
The counterfactual pairs themselves pass through the same audit and candidate-set treatment. This closes FACTUAL's stated gap rather than inheriting it.
Baselines
The naive reference trains a target-conditioned BERT or DeBERTa classifier on synthetic data with prompt labels taken at face value, plus the same small real seed set, which is the assumption made in The Power of LLM-Generated Synthetic Data…. On X-Stance that paper reports average F1 0.727 for the full real-label model and 0.754 for SQBC with synthetic data and 25% of real labels. We will reproduce both, and we expect our method to exceed 0.754 at the same 25% budget. Real-only training and backtranslation are the cheap references. On the COVID-19 benchmarks, the Limited Effectiveness paper reports macro-F1 0.773 for unaugmented DeBERTa-v3, 0.789 with backtranslation, and 0.793 with GPT-4o augmentation. Beating 0.793 there, and doing so specifically on the incongruent slice, is the bar.
The filtering baseline discards suspected refusals and flips with a DistilRoBERTa rejection classifier, as in that paper. This tests whether recovering flipped samples is better than throwing them away. Co-teaching represents small-loss sample selection. SiDyP's authors report it underperforms on LLM-label noise, and we include it to confirm the result transfers to stance.
The strongest denoising competitor is SiDyP, run unmodified on the same synthetic set with the same encoder backbone, epochs, and tuning grid. Its paper reports average BERT accuracy gains of 7.21% in zero-shot settings and 7.30% in few-shot settings on its own benchmarks. We found no reported result on Lacuna for SiDyP on any stance dataset. The strongest stance-debiasing competitor is FACTUAL, which reports 76.16% average F1 on SemEval-2016 with GPT-3.5 (up from 61.02%), Bias-SSC reduced from 27.13 to 11.31, and 79.98% zero-shot F1 on VAST. Beating it means matching or exceeding its F1 with lower Bias-SSC, using a small trainable classifier instead of a calibrated LLM. Bias-SSC is FACTUAL's measure of how much predictions follow sentiment rather than stance.
All methods get the same synthetic pool, the same real seed labels, the same backbone, and an equal hyperparameter search budget, with three seeds each. Two ablations isolate the key ingredient. The first runs our full pipeline with neighbors drawn from the plain text encoder instead of the sentiment-invariant one. The second assigns the same candidate set to every sample instead of using the audit's congruence-based risk cells. If the first ablation matches the full method, the invariance claim is wrong.
Why not just…?
Why not just ask a stronger LLM to re-check each synthetic label? This is the most serious threat, and we include it as a baseline: GPT-4o relabels every synthetic sample, with and without "ignore sentiment" instructions. The evidence suggests it fails in exactly the region we care about. FACTUAL reports a Bias-SSC of 27.13 for GPT-3.5 and argues that the bias sits in the weights and resists prompting. A sentiment-biased verifier should therefore "correct" valid incongruent samples toward the sentiment-implied label. The audit measures this directly. If the LLM verifier's error rate on incongruent samples is close to its rate on congruent ones, this alternative wins, and we would report that.
Why not just run SiDyP or another existing cleaner? SiDyP finds candidates through nearest neighbors in text-embedding space. In stance, those neighborhoods are organized by emotional tone, so the cleaner may suppress the shortcut-breaking examples. The diagnosis component and the plain-encoder ablation test this directly. If SiDyP's slice-level results equal ours, the contribution reduces to the audit.
Why not just throw away suspect samples, or use backtranslation instead of LLM generation? Filtering discards precisely the incongruent samples that teach a model that anger can signal support. Backtranslation paraphrases text but cannot create an outraged Favor argument that did not already exist. Both are run as baselines. The test is incongruent-slice F1, where we expect both to trail the full method, even though backtranslation's 0.789 overall on COVID shows it is competitive on aggregate.
Why not just self-train on unlabeled real data? On P-Stance, self-training with 2.8 million unlabeled tweets lifts BERTweet from 80.53% to 82.42%, but it needs a large in-domain pool and an unbiased teacher. We compare against it on P-Stance at a matched seed budget, and in a new-target cold-start split where such a pool does not yet exist.
Proposed Experiments
The main evaluation reports macro-F1 on each benchmark. We also report F1 on a real-data incongruent slice, defined by human-adjudicated sentiment-stance disagreement on test items, along with FACTUAL's Bias-SSC. The audit reports per-cell flip rates with confidence intervals, broken down by generator and target.
Three checks decide the outcome. First, does the audit show a large congruent-versus-incongruent gap in flip rates? Without that gap, structured repair has no basis. Second, does plain-embedding repair damage valid incongruent samples more often than it fixes flips? Third, does the full method beat SiDyP, LLM re-verification, and FACTUAL on incongruent-slice F1 while holding overall F1? Each check can fail independently, and each outcome tells us something about when synthetic stance data helps and when it hurts.
References
- The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions
- Limited Effectiveness of LLM-based Data Augmentation for COVID-19 Misinformation Stance Detection
- Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement
- Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration
- Neighbor-aware Label Refinement: Enhancing Unreliable Instance-Dependent Partial Labels
- Reduction-based Pseudo-label Generation for Instance-dependent Partial Label Learning