Do Sentiment Chains of Thought Escape the Lexicon?

Old sentiment systems looked up each word's polarity in a dictionary and added up the scores. They fail in well-known places: negation ("not bad"), contrast ("brilliant actors, but a disastrous plot"), sarcasm ("oh great, another delay"), and factual complaints that contain no emotional words at all. Chain-of-thought prompting asks a language model to write out intermediate reasoning before giving its answer. It is often presented as the cure for these failures, because the model can supposedly reason its way past a misleading keyword.

Recent audits cast doubt on that story. Written reasoning is frequently decorative: in one medical audit, corrupting the chain left accuracy unchanged in most cases. Language models also show a strong bias toward reading negated sentences as affirmative. This note proposes two things. The first is a way to measure whether a sentiment chain of thought actually overrides the dictionary-style keyword prior, or only wraps that prior in a plausible story. The second is a training method meant to make the chain causally responsible for overriding it. The method works by requiring the chain to change in exactly the right place when a minimal edit to the sentence changes its sentiment.

Core Claim

We claim that current sentiment chains of thought still carry a measurable "lexicon residual." This means that, among sentences with the same true label, a chain-of-thought model's prediction still moves with a simple word-lexicon score. We also claim the residual is largest exactly where lexicons are known to fail.

We further claim that this residual can be reduced, and the chain made causally faithful, by training on minimally edited sentence pairs. In each pair, a single composition operator (a negator, a contrastive "but," an irrealis marker such as "would have been," or a sarcasm cue) is added or removed. The chain is written as a short sequence of typed, checkable claims: the anchor word, its dictionary polarity, the operator acting on it, the operator's scope, and the resulting polarity. Training requires that the chains for the two members of a pair differ only at the operator step, and that the final label follows that step.

The new ingredient is this paired, step-level consistency constraint. It is meant to make the operator step the thing the label actually depends on.

Relation to Prior Work

The closest work applies chain-of-thought to implicit sentiment. Reasoning Implicit Sentiment with Chain-of-Thought Prompting introduced THOR, which infers the aspect, then the latent opinion, then the polarity. It reports more than 6% F1 over prior top supervised models with Flan-T5 11B. It also concedes that supervised models often reach the right label through flawed intermediate steps. Application of Multiple Chain-of-Thought in Contrastive Reasoning for Implicit Sentiment Analysis argues for each candidate polarity in parallel (TRCR) and reports about 6% F1 over THOR on implicit subsets, at two to three times the generation cost. Emotion-o1 adapts reasoning length to difficulty.

All three optimize accuracy or efficiency with free-form chains. None of them models syntactic composition operators, and none tests whether the chain causes the label. Our proposal reuses their benchmarks and their models as baselines. It differs in what is supervised (typed operator steps, constrained across minimal pairs) and in what is measured (causal dependence and the lexicon residual).

Two lines of work motivate the gap. Right Diagnoses, Decorative Reasoning found a 72.9% chain-decoupling rate across 14 models, meaning chain edits mostly did not move answers. It also found that chain-of-thought cost Llama-3.1-8B more than 12 points versus direct answering. Analysing Chain of Thought Dynamics showed that instruction-tuned models change their answer after reasoning in only 25% of cases, often a sign of post-hoc rationalization.

Two further results constrain the design. This is not a Dataset showed zero-shot accuracy as low as 2–3% on negated sentences, and showed that fine-tuning reached above 90% only by learning surface cues such as "never means false." Stubborn Lexical Bias in Data and Models showed that rebalancing unigrams simply pushes shortcuts onto bigrams. Together these rule out evaluating on the same operators used in training, and they rule out treating data rebalancing as a fix.

The operator inventory comes from classical composition work. Interpreting Sentiment Composition with Latent Semantic Tree defines a grammar with labeled operators such as Negator, Priority Riser, and Irrealis Blocker. An Empirical Study on the Effect of Negation Words on Sentiment showed that negators shift polarity asymmetrically rather than simply flipping its sign. That result is why our chain records the operator and its scope instead of applying a fixed flip rule.

Research Program

Measuring the lexicon residual. The first component is the measurement. For each sentence $x$, let $L(x)$ be a lexicon score: the summed prior polarity of its words under a standard word list such as SentiWordNet, with no composition rules. Let $y$ be the gold label and $\hat{y}$ the model's prediction. We fit

$$\Pr(\hat{y} = + \mid x) = \sigma\big(\alpha_y + \beta \, L(x)\big),$$

where $\sigma$ is the logistic function and $\alpha_y$ is a separate intercept for each gold label. The coefficient $\beta$ is the lexicon residual: how much the prediction tracks keyword polarity once the true answer is held fixed. A model that genuinely composes meaning has $\beta$ near zero.

We report $\beta$ in two places. The first is the whole test set. The second is a "lexicon-conflict" subset, where the sign of $L(x)$ disagrees with $y$.

Building the minimal pairs. The second component is the data. From Stanford Sentiment Treebank parse trees, which carry phrase-level labels, we extract operator sites. We start from the 2,261 negator–argument pairs used in the negation study, then add contrastive conjunctions, intensifiers, and irrealis markers. For each site we create a minimal edit that inserts, deletes, or swaps the operator, and we label the edited sentence. Where the label is not determined by the treebank, we use human annotation.

Operators are split by type. Training covers "not," "but," and "very." Testing uses held-out operators such as "never," "hardly," "although," and "would have." This split directly tests the surface-cue failure documented in the negation benchmark.

Training the typed chains. The third component is the training method. Seed chains are typed steps written in natural language, and each step is checked against the treebank where possible: the anchor word and its prior polarity, the operator, its scope, and the composed polarity. These are generated by a large model and filtered.

The fine-tuning objective adds a paired term to the usual chain likelihood. For a minimal pair $(x, x')$, the model is penalized if the label does not follow the operator step. It is also penalized if the anchor and prior-polarity steps differ between the two chains, since those should not depend on the edited operator. Implicit-sentiment sentences, which have no anchor word, use THOR-style aspect and opinion steps in the same typed format. This lets one model handle both explicit composition and implicit sentiment.

Backbones. We use Flan-T5-xxl and Llama-3.1-8B. Flan-T5-xxl matches the THOR and TRCR papers. Llama-3.1-8B is the model on which chain-of-thought hurt in the medical audit.

Baselines

Direct answering. The naive reference is zero-shot and few-shot direct answering by the same backbones, plus the raw lexicon score $L(x)$ itself as a dictionary classifier. These fix the floor, and they show how much of the lexicon-conflict subset is lost simply by reading keywords.

Free-form chain-of-thought methods. The strongest existing chain-of-thought competitors are THOR and TRCR, both fine-tuned on the same training data and backbones with matched tuning budgets. On the SemEval-2014 implicit subsets, the file has no absolute reported number for either. THOR reports "over 6%" F1 gain over prior top models with Flan-T5 11B, and TRCR reports about 6% F1 over THOR. We therefore rerun both under identical conditions rather than copying figures. For TRCR, we report accuracy per generated token as well as raw accuracy, because it uses three chains per input.

Composition-specific systems. These are the strongest competitors on the composition question. The first is BERT with the latent-tree composition module, which reports 92.31% sentence accuracy on SST2 (Interpreting Sentiment Composition with Latent Semantic Tree). The second is PSTN, which reports a mean absolute error of 0.1062 on the negator–argument test pairs, versus 0.1480 for sign reversal (An Empirical Study on the Effect of Negation Words on Sentiment).

Ablations. Two ablations isolate the key ingredient. The first trains the same typed chains without the paired consistency term. The second trains direct-label fine-tuning on exactly the same minimal pairs, which is counterfactual augmentation with no chain at all.

What counts as beating the baselines. Our method must meet three conditions. It must exceed every baseline on held-out-operator accuracy. It must have a lexicon residual $\beta$ indistinguishable from zero on the conflict subset. It must also match the composition systems within one point on SST2 and stay at or below PSTN's mean absolute error.

Why not just…?

Why not just fine-tune directly on the counterfactual pairs, with no chain? This is the most serious threat, because the pairs alone may teach composition. The direct-label ablation settles it. If that ablation matches our method on held-out-operator accuracy and on $\beta$, then the chain contributes only auditability, and we will report that. Our prediction rests on the negation benchmark's finding that supervised models learn operator-specific surface cues (This is not a Dataset). We expect the typed operator step to transfer to unseen operators better than a label-only signal, but that is an empirical claim, not an assumption.

Why not just prompt with the operator schema and skip training? Prompting is cheap, so we include it as a baseline. The decorative-reasoning audits suggest that prompted chains in instruction-tuned models rarely drive the answer (Analysing Chain of Thought Dynamics; Right Diagnoses, Decorative Reasoning). The chain-edit test below measures whether a schema-prompted chain is followed.

Why not just use a rule-based composition engine or the latent-tree grammar? These systems are interpretable and strong on explicit composition. However, they need lexical anchors, so they cannot handle implicit or sarcastic sentences. The tree method also scales cubically with sentence length. We include them as baselines on SST, and we expect them to remain competitive on in-grammar negation. The case for our method rests on matching them there while also covering SemEval implicit sentiment and held-out operators.

Why not just sample several chains and vote, or argue each polarity as TRCR does? Voting over chains that share the same keyword prior may reinforce that prior rather than correct it. Comparing $\beta$ for TRCR against our method answers this directly, at reported token cost.

Decisive Experiments

Experiment 1: the lexicon residual of existing methods. We measure $\beta$ for direct answering, THOR, and TRCR on the SST conflict subset and the SemEval-2014 explicit and implicit subsets. If the free-form chains already show a residual near zero, the motivating premise fails, and the project reduces to an audit.

Experiment 2: causal tests on chains and inputs. First, we edit only the operator step of a generated chain, for example changing "negator scopes over 'good'" to "no operator." The chain-follow rate is the fraction of cases where the label changes accordingly. This is the mirror image of the chain-decoupling rate. Second, we apply minimal input edits and measure whether the chain updates at the operator step and nowhere else. Success means a chain-follow rate well above the free-form baselines, together with a held-out-operator accuracy gain over the direct counterfactual-augmentation ablation.

Experiment 3: transfer to an unrelated negation benchmark. We evaluate on This-is-not-a-Dataset without any training on it. The aim is to see whether improved operator handling carries over beyond sentiment, or whether it is again a sentiment-specific shortcut.

All experiments report generated-token counts. A gain bought with TRCR-level compute would be a weaker result than one achieved with a single short typed chain.