Style Is a Direction: Cheap Robustness to LLM-Rewritten Fake News

Fake news detectors learn how an article is written as much as what it claims. When a large language model (LLM) rewrites a fabricated story in a calm, professional newsroom voice, those detectors fail badly. Fake News in Sheep's Clothing reports F1 drops of up to 38% under such "style attacks." Its fix, SheepDog, asks GPT-3.5 to rewrite every training article in four styles and trains the detector to give all four versions the same verdict. That works, but it multiplies generation cost by the size of the training set, and every new attack style means another pass.

This note proposes a cheaper route built from a geometric claim we can test. A classifier's score is an inner product, which mixes the length of a representation with its direction. We hypothesize that most of what an LLM restyling does to a news article's embedding is either a change in length or a movement along a small number of shared directions. If that holds, two steps should give most of SheepDog's robustness with a few hundred rewrites in total rather than four per article. The first step is to classify by angle only, with a cosine classifier that scores by the angle between the article vector and each class vector. The second is to project out a low-dimensional "style subspace" estimated once from a small set of rewrite pairs. LLM hallucination enters in two places. It is a hazard, because rewrites can silently change facts and contaminate the style estimate. It is also a test, because a detector that has become style-invariant must still react when facts are swapped.

Core Claim

The claim has three parts, and each can be checked on its own. First, in a fine-tuned news encoder, the shift from an original article to its LLM-restyled version breaks into two parts. One is a change in length along the article's own vector, which we call the radial part. The other is a movement across the unit sphere, which we call the angular part, and it concentrates in a subspace of dimension $k$ much smaller than the embedding dimension. Second, a detector that scores

$$s_c(x) = \tau \cdot \cos\big(w_c,\; P\,h(x)\big), \qquad P = I - U U^\top,$$

recovers most of the robustness that SheepDog buys through per-article augmentation. Here $h(x)$ is the encoder output for article $x$, $w_c$ is the weight vector for class $c$ (real or fake), $\tau$ is a fixed temperature, and $U$ holds the top-$k$ principal directions of the normalized restyling differences, estimated from a small calibration set. Third, this invariance does not blind the detector to content. Entity and role swaps, the kind of factual edit LLMs make when they hallucinate (Milecki), must still move $s_c$.

The idea is new in two ways. Nobody in the file measures whether style attacks act on the norm or on the direction of detector features. And the one style-robust detector, SheepDog, uses an ordinary dot-product head and pays for invariance through generation, not through the geometry of the head.

Where This Sits

The method comes from the class-incremental and long-tailed learning literature. That work showed that nuisance factors show up as vector norms. In Learning a Unified Classifier Incrementally via Rebalancing, new classes inflate their weight norms, and cosine normalization removes the effect. Maintaining Discrimination and Fairness in Class Incremental Learning found the norm gap regular enough to fix after training with a single rescaling factor. Spherical Procrustes Alignment showed that imbalanced fine-tuning makes models inflate feature norms to gain confidence, and that a spherical head cuts calibration error from 28.51% to 4.44%. The closest analogue to our nuisance is few-shot entity tagging: Huang et al. normalized token prototypes so that word frequency, a surface property unrelated to meaning, stopped biasing predictions. We reuse this logic and apply it to writing style. None of these papers studies a nuisance that rotates features, and handling rotation is why we add the projection $P$.

The motivating evidence comes from fake news work. SheepDog shows the vulnerability and supplies the main baseline. Grover shows that detectors pick up distributional traces of how text was produced, not whether its assertions are true. That is exactly the shortcut a style attack exploits. MisSpans constrains our design: real misinformation mixes true and false content within a single sentence. So a style subspace that happened to absorb content edits would be disastrous, and our content-sensitivity test exists to catch that.

Two recent systems connect hallucination and fake news in a different way. PHPFND treats the regions where an LLM hallucinates as features pointing to hard, deceptive parts of an article. Is LLMs Hallucination Usable? generates deliberately wrong "negative reasoning" and distills it into a student, reaching 0.857 accuracy on PolitiFact. Both use LLM output as extra supervision, and both keep dot-product heads. Neither tests robustness to restyling. We use far less generation and change the classifier geometry instead. Finally, MARiA found that plain embedding cosine similarity beat a prompted Mixtral on SHROOM, the SemEval 2024 hallucination-detection task (70.3% vs 60.2%, model-aware track). In the model-agnostic track, an off-the-shelf bart-large-mnli entailment model was the best of its three methods. We take the entailment model as a filter for content-preserving rewrites, not as a detector.

Research Program

The work proceeds in three stages. The first is a diagnostic. We fine-tune RoBERTa and DeBERTa detectors, the same backbones SheepDog uses, on PolitiFact, GossipCop and LUN. For held-out articles we generate the four SheepDog styles and measure how the embedding shift and the logit change split between the radial and angular parts. We also measure how much of the angular part a rank-$k$ subspace captures, for $k$ from 1 to 64. We check whether the subspace estimated on one dataset carries over to another and to rewrites from a different LLM. Nobody has reported these numbers, and they decide whether the method can work.

The second stage builds the style subspace $U$. We draw a calibration set of a few hundred training articles and rewrite each into the four styles. Because LLM rewriting can hallucinate content, we keep only pairs that bart-large-mnli judges as entailing in both directions. The retained differences are normalized to unit length and decomposed with principal components analysis to give $U$. We train the cosine head with $P$ fixed, then re-estimate $U$ once from the fine-tuned encoder and fine-tune a second time. We also study a variant where $U$ is re-estimated per domain. That variant connects to the continual detection setting of Han et al., but it is secondary.

The third stage tests content sensitivity. We build a probe set by applying controlled factual edits to real articles, such as swapping entities and roles, flipping numbers, and turning hedged claims into certainties (following Liyan Tang et al.). For MisSpans articles, we also use their gold false spans. A good detector should be insensitive to style and sensitive to these edits.

Baselines

The naive reference point is a fine-tuned RoBERTa or DeBERTa with a standard dot-product head, trained on the original articles only. It is evaluated on clean test sets and on the style-attacked test sets generated with SheepDog's four-style protocol. Its drop under attack sets the size of the problem.

The strongest existing method is SheepDog itself (Wu and Hooi), with full four-style augmentation of every training article, the style-alignment loss, and the LLM-generated veracity attributions. The paper reports that SheepDog beats the strongest baseline by over 15% F1 on LUN under adversarial conditions. No reported absolute SheepDog F1 was found on Lacuna, so we will rerun it on identical splits, backbones and tuning budgets. To be fair, we give SheepDog and our method the same number of epochs, the same hyperparameter search size, and rewrites from the same LLM. Beating SheepDog outright is not the goal. We count the method a success if it comes within 2 F1 points of full SheepDog on attacked LUN, PolitiFact and GossipCop while using at least 20 times fewer generated rewrites.

The threatening baseline is SheepDog at a matched budget: SheepDog's augmentation and alignment loss applied to exactly the same few hundred calibration articles we use to estimate $U$. If budget-matched SheepDog matches us, the geometric idea adds nothing, and we will say so. To win, our method must beat it by a margin larger than the seed-to-seed spread over five seeds.

Three ablations isolate the ingredients. The first uses the cosine head without projection, which tests whether length removal alone helps. The second uses projection with a dot-product head, which amounts to projecting out a nuisance subspace without normalizing. The third removes the entailment filter, which tests whether hallucinated rewrites corrupt $U$. A zero-shot prompted LLM detector is included as a reference, since SheepDog's authors describe LLMs as mediocre zero-shot veracity predictors. For calibration we compare against the naive model with post-hoc temperature scaling.

Why not just…?

Why not just run SheepDog with fewer rewrites? This is the real threat. The file contains no evidence on how SheepDog's robustness scales with augmentation budget. It may be that a few hundred augmented articles already teach enough invariance. The budget-matched baseline settles the question directly. We will also sweep both methods from 100 to the full training set and report the two curves. Our method is useful only where its curve sits above SheepDog's.

Why not just prompt an LLM to judge veracity, which is immune to style by construction? It is not immune in practice. SheepDog reports LLMs to be mediocre zero-shot detectors. MARiA found a prompted Mixtral near chance before heavy prompt tuning, and unstable under formatting changes afterward. MisSpans reports F1 below 0.5 for fifteen LLMs on fine-grained false content. We include the zero-shot reference so readers can see the gap on the same attacked test sets.

Why not just normalize features, since transformers already use layer normalization? Layer normalization does not fix the length of the pooled vector that meets the classifier, and it does nothing about the classifier weights, whose norms drift under class imbalance (Zhao et al.). More importantly, length removal alone may simply not be enough if style rotates features. The cosine-head-without-projection ablation measures exactly this. If it matches the full method, the proposal shrinks to a one-line change, which would still be worth reporting.

Why not just use temperature scaling or class reweighting? These rescale or shift logits uniformly. They cannot tell a style direction from a content direction, so they can improve calibration but not robustness to restyling. SPA's results suggest that fixing the geometry during training beats patching logits afterward. Our temperature-scaled baseline tests whether that carries over to news.

Decisive Experiments

The first check is the diagnostic, and it comes before any claims about the method. If the radial part plus a rank-16 angular subspace explains less than about half of the attack-induced logit change on PolitiFact and LUN, the hypothesis is wrong. We would then report the measurement as a negative result showing that style attacks act on content-entangled directions, and stop. The second check compares attacked F1 against the number of generated rewrites for our method, full SheepDog, and budget-matched SheepDog, on PolitiFact, GossipCop and LUN, averaged over five seeds. The third is transfer: we estimate $U$ from GPT-3.5 rewrites and test on rewrites from Llama 3 and in styles not seen during calibration, where a single fixed subspace is most likely to break. The fourth measures content sensitivity: the fraction of entity-swap and certainty-flip probes that change the prediction, and the gap in scores on MisSpans gold-false spans. Style invariance that comes at the cost of this sensitivity counts as failure, whatever the attacked F1. We also report expected calibration error on attacked sets. The spherical head is expected to cut overconfidence on restyled fakes, and SPA's audio results give a reason to look for that effect here.