A news line such as "Quarterly profits fell by 15%" uses no emotional words, yet it hurts the company. A company post reading "We are deeply saddened by the passing of our founder" is full of sad words, yet it tends to earn sympathy. Work on online reputation monitoring has long held that emotional tone (sentiment) and effect on public standing (reputational polarity) can come apart. Separately, work on large language models shows that coherent but irrelevant text can sharply degrade their answers. This note connects the two. The emotional tone of a post may act as a distractor for a model judging reputation. Unlike the distractors studied so far, it sits inside the very sentence the model must read. No retrieval filter can remove it.
The proposal has two parts. First, we measure how much a language model's reputation judgments shift when tone changes but facts do not. We use pairs of rewritten posts and check, through the model's internal sentiment direction, whether that shift is caused by sentiment leaking into the reputation decision. Second, we test a training method built on two kinds of paired examples. The model should ignore changes in tone and still respond to changes in fact. The aim is that it stops being fooled by tone without becoming blind to genuine complaints.
Core Claim
The claim has three parts. Language models that classify reputational polarity concentrate their errors on posts where sentiment and reputation disagree. The cause is a known internal feature, the linear sentiment direction, being read out as if it were reputation. The remedy cannot be borrowed from retrieval-noise work, because that work removes whole irrelevant passages. Here the signal and the distractor share the same words, so the remedy has to work on the model's own behaviour or representations.
Where This Sits
Three lines of prior work motivate the problem.
The RepLab campaigns defined reputational polarity as a task distinct from sentiment and found it hard. Overview of RepLab 2012 reports about 41% accuracy on joint filtering and polarity. RepLab 2013 reports polarity peaking near 69%, even though filtering exceeds 90%. Detecting the Reputation Polarity of Microblog Posts shows that entity-centric word statistics, selected by log-likelihood ratio, beat sentiment lexicons. That result is indirect evidence that sentiment is a weak proxy for reputation. The NewsIR16 report (Report on NewsIR16) names "polar facts" as the clearest case of divergence.
None of this work studied modern language models. Our question is whether those models, which carry a strong internal sense of tone, inherit the confusion in a new form.
The second line, which also constrains the design, is the work on contextual distraction. Breaking Focus finds an average drop of about 45% under coherent distractor sentences. It also finds that "ignore irrelevant information" prompts do not help, and that preference training on distracted and clean pairs does. Improving Zero-shot Reader shows that distracted models remain highly confident. Two further methods prune or extract passages before or during answering: Superposition Prompting and Ext2Gen.
All of these assume the noise lives in separable chunks. We reuse their preference-pair idea but drop that assumption. In our setting the "distractor" is a property of the relevant sentence itself.
The third line supplies the probe. Language Models Linearly Represent Sentiment shows that sentiment lies along one direction in activation space and is gathered at neutral tokens such as commas and periods. That paper studies only movie-review-style valence. We use its direction as a measuring instrument for how much tone drives a reputation decision.
One prior result could seem to cut against us. From Sentiment to Reputation argues that the sentiment of replies does carry reputational information, so sentiment is not always noise. We take that seriously. It is exactly why the method must keep the model sensitive to evaluative content, such as a customer saying a service failed, while making it insensitive to how emotionally that content is phrased.
Research Program
Building the probe set. We start from the RepLab 2013 polarity data: English and Spanish tweets about 61 entities in automotive, banking, universities, and music. We split posts into two groups. The agreement subset holds posts where an off-the-shelf sentiment classifier's label matches the gold reputation label. The divergence subset holds posts where they disagree.
For a stratified sample of each subset we build two kinds of counterfactual rewrites, generated by a language model and checked by annotators.
- A tone swap changes emotional wording and intensity. It keeps fixed the events reported and any evaluative claim made about the entity. For example, a furious complaint becomes a flat report of the same failure. The gold reputation label should not change.
- A fact swap keeps the tone and changes the reputationally relevant event. For example, profits fell becomes profits rose. The gold label should change.
Annotators confirm each label on a held-out portion. Drawing the line between tone and evaluative content is the main annotation risk. We report agreement on that boundary separately rather than hiding it.
The context arm. A secondary arm tests the gap left open by Peetz et al. We append extra context to each post, in two varieties. One is emotionally charged but irrelevant to the entity's standing. The other states a polar fact neutrally. We ask which kind moves the judgment more.
Diagnosis. For each open model, starting with Llama-3.1-8B-Instruct as used by Ext2Gen, we estimate the sentiment direction following Hollinsworth et al. We then measure how strongly the reputation decision depends on that direction, by projecting the direction out at the tokens where sentiment is gathered. A random-direction projection serves as a control.
If divergence-subset accuracy rises and the tone-flip rate falls under the true projection but not the random one, we have causal evidence that tone is leaking into the decision. If projection also hurts the agreement subset, that shows sentiment is partly useful, which is the Peetz point. The method has to handle that trade-off.
The method. We fine-tune with preference pairs of two kinds, in the style of Breaking Focus. Invariance pairs prefer the same label across a tone swap. Sensitivity pairs prefer the changed label across a fact swap. The second kind is the ingredient we expect to matter. Without it, the cheapest way to resist tone is to discount all emotional language, which would misclassify genuine complaints.
Baselines
Naive reference points. These are the majority-class predictor and a pure sentiment classifier used directly as a reputation predictor. The second sets the floor that "tone equals reputation" achieves on each subset. By construction it is wrong on the whole divergence subset.
Strongest reported system. This is the Naive Bayes classifier over surface, sentiment, and log-likelihood-ratio features from Detecting the Reputation Polarity of Microblog Posts. It reports accuracy 0.8546 on RepLab 2013, against a competition baseline of 0.685. The RepLab overview's peak of about 69% suggests evaluation conditions may differ, so we will reimplement it on our exact splits rather than rely on the reported figure.
It is the strongest competitor because it uses same-entity supervision and entity-specific vocabulary, which may already sidestep tone. We will report its accuracy on the divergence subset and its flip rates on counterfactuals. For divergence-subset accuracy and counterfactual flip rates, no reported result was found on Lacuna for any system, LLR included. Beating it means matching its full-test accuracy while clearly exceeding it on the divergence subset and the counterfactual pairs.
Plain supervised fine-tuning. This is the same base model fine-tuned on RepLab 2013 training labels, with the same compute and the same tuning budget as our method. It is the most direct threat, and we address it in the next section.
Prompting and passage filtering. We test an "ignore emotional tone, judge only the effect on the entity" instruction. We also test Ext2Gen-style extraction on the context arm, since that is the only place it applies.
Ablations isolating the key ingredient. We compare training on invariance pairs only against invariance plus sensitivity pairs. Separately, we compare inference-time sentiment-direction projection against a random-direction projection.
Why not just…?
Why not just tell the model to ignore tone? Breaking Focus reports that "ignore irrelevant information" prompts fail against coherent distractors. Its distractors were added sentences, though, not tone embedded in the target. Our tone-flip rates under the instruction prompt will settle the question directly. If the prompt drives tone flips near zero without lowering fact-swap sensitivity, the training method is unnecessary, and we will say so.
Why not just fine-tune on RepLab labels? Same-entity supervision may teach the model reputation directly. The LLR result's strength hints at this. This is the most serious alternative. The test is whether a fine-tuned model still flips on tone swaps of divergence posts. If its tone-flip rate matches our method's, the contribution shrinks to the diagnosis.
Why not just augment the training data with tone-swapped examples? This is one-sided augmentation, and it may well reach the same divergence-subset accuracy. Our prediction is that it hurts the agreement subset, specifically genuine complaints, because it rewards ignoring emotional content. The invariance-only ablation is exactly this alternative. If it does not lose accuracy on complaints, the sensitivity pairs add nothing, and we will report that.
Why not just use retrieval-noise methods such as Ext2Gen or distraction-aware answer selection? These methods select or score passages. When the fact and the tone share a sentence, there is nothing to prune. We still run Ext2Gen on the context arm, where noise is separable. We expect it to help there and not on single-post tone swaps, and that contrast is itself part of the claim.
Decisive Checks
The first check is the hypothesis itself. Zero-shot model error should be markedly higher on the divergence subset than on the agreement subset, and tone-swap flip rates should be well above zero. If not, tone is not functioning as a distractor and the project stops at a negative result.
The second check is causal. Sentiment-direction projection should beat random projection on divergence accuracy and tone-flip rate.
The third check is the method. Two-sided training should lower tone-flip rate relative to plain fine-tuning while keeping fact-flip rate high. It should also beat invariance-only training on agreement-subset complaints. It must reach this at no loss in full-test accuracy relative to the reimplemented LLR classifier.
Results are reported per domain and per language. RepLab work stresses entity dependence, so a gain confined to one sector would be reported as exactly that.
References
- Overview of RepLab 2012: Evaluating Online Reputation Management Systems
- RepLab 2013: Comparative Evaluation of Online Reputation Monitoring Systems and Components
- Detecting the Reputation Polarity of Microblog Posts
- Report on the 1st International Workshop on Recent Trends in News Information Retrieval (NewsIR16)
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Language Models Linearly Represent Sentiment
- From Sentiment to Reputation