Negation as Rotation: Testing Whether Phase Geometry Explains Inverse Scaling on Negated Prompts

Larger language models often get worse at following negated instructions. Asked to pick the inappropriate ending of a story, or to complete "Ibuprofen is not a…", they fall back on the familiar affirmative answer. Fine-tuning on negated examples fixes this only by giving up accuracy on ordinary prompts. We propose a geometric explanation, borrowed from phase retrieval, together with a small architectural test of it. In real-valued spaces, turning a vector $v$ into $-v$ by any smooth path of scalings has to pass through zero, and at zero the content is erased. In complex-valued spaces, $v$ can reach $-v$ by rotating along a circle, and its length never changes. Our hypothesis is that learned negation in ordinary transformers behaves like the real case. The model shrinks the part of its representation that carries the proposition instead of flipping it, so the model is left to guess, and the affirmative pretraining prior wins more often in larger models.

The proposal has two parts. First, a diagnostic study on the Pythia checkpoints asks whether negation in larger and longer-trained models moves hidden states by shrinking them or by reorienting them. Second, we build a "polarity adapter," a small module that rotates paired hidden dimensions by a context-dependent angle, and compare it against matched real-valued operators. The central comparison is between operators that stay length-preserving along their whole learning path and operators that must pass through an erasing state. If rotation removes the trade-off between affirmative and negated accuracy and flattens the inverse-scaling slope, while a real gate with the same parameter count does not, we will have evidence that geometry, and not only data scarcity, contributes to inverse scaling on negation.

Core Claim

Phase retrieval supplies the key observation. For real signals, the two global minimizers $\pm x$ are disconnected. Nearly optimal bounds for the global geometric landscape of phase retrieval shows that they are separated by a region near the origin containing saddles, with a local maximum at zero. In the complex case, the minimizers form a connected circle $\{e^{i\theta}x\}$. We carry this over to negation. Suppose a network represents "not" as a learned real multiplier on some subspace, starting near the identity. To reach sign inversion it must pass through a scaling of zero, which is a projection that deletes the proposition's content. A rotation by a learned angle can reach inversion at $\theta=\pi$ while keeping the norm fixed at every step.

We make the adapter precise as follows. Let $h$ be a hidden state and $P$ a learned map onto $k$ pairs of dimensions, read as a complex vector $z = Ph \in \mathbb{C}^k$. Let $\theta(c)$ be an angle predicted from the local context $c$. The adapter outputs

$$h' = h + P^{\dagger}\big((e^{i\theta(c)} - 1)\odot Ph\big),$$

where $P^{\dagger}$ maps back to the residual stream and $\odot$ is elementwise multiplication. The matched real control uses a gate $g(c)\in[0,1]$:

$$h' = h + P^{\top}\big(-2g(c)\,Ph\big).$$

At $g=1/2$ this control annihilates the subspace, and it reaches inversion only at $g=1$. Two predictions follow. The real gate, and the model's native mechanism, will sit near the erasing midpoint when supervision is ambiguous, which produces the roughly 50% averaged original-plus-negated accuracy reported by Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts. The rotation should avoid that collapse and should avoid the trade-off between affirmative and negated accuracy that the same paper observed after fine-tuning.

A rotation on paired real dimensions is mathematically the same as complex multiplication. We claim no advantage for complex arithmetic as such. The claim concerns the operator family: an operator that is connected to the identity and preserves length along its entire path.

Relation to Prior Work

Three papers motivate the proposal by documenting the gap. Inverse Scaling: When Bigger Isn't Better catalogs tasks where larger models do worse and names "strong prior" as a cause. Emergent Inabilities? Inverse Scaling Over the Course of Pretraining shows on Pythia that Memo Trap, Pattern Match Suppression, and TruthfulQA degrade as training continues, and degrade faster in larger models. Language models are not naysayers shows GPT-neo-125M beating its 6B sibling on negated factual cloze, and reports that 10-shot prompting does not help. That last finding points to a problem in the representation, which is the level our diagnostic probes. None of these papers proposes a mechanism at the level of hidden-state geometry. We do, and we test it.

Two papers constrain the design. The trade-off result in Can Large Language Models Truly Understand Prompts? means any fix must be judged on affirmative and negated accuracy together. Logical Consistency of Large Language Models in Fact-Checking finds that QLoRA fine-tuning on logical primitives generalizes to compound formulas while instruction prompting hurts. We therefore train on the same kind of primitive data and treat parameter-efficient fine-tuning as the main competitor, not as a straw man.

The closest architectural work is ComplexFormer. It rotates queries and keys by learned semantic-plus-positional angles to change attention scores, and it evaluates on GSM8K and MATH. Our proposal differs in three ways. We rotate the content in the residual stream, not the attention scores. The rotation angle depends on the operator words in the context, not on position. We evaluate on negation and inverse scaling, which ComplexFormer never probes. Grounding the "Not" is the nearest attack on negation. It merges negation tokens with their attributes and scales them by a boost $\beta>1$ in vision-language detectors. Boosting changes length, whereas our operator changes direction and holds length fixed. We port their idea to text as a baseline. Earlier complex-valued networks, Complex Transformer and SurReal, show that treating magnitude and phase as coupled beats simply concatenating the real and imaginary channels on physical signals. SurReal's separation into a scaling part and a rotation part is the decomposition we exploit. Neither paper touches language or logic.

Research Program

The first stage is diagnostic and does not depend on the adapter working. Using the Pythia suite from 410M to 12B parameters across its saved checkpoints, we build minimal pairs of affirmative and negated sentences from the Negated Prompts suite, MKR-NQ, and Logical Fact-Checking templates. At each layer we split the hidden-state difference between each pair into two parts: a shrinking component that runs along the affirmative state, and a reorienting component orthogonal to it. We track two quantities, the norm ratio and the cosine similarity, against model size and training tokens. The hypothesis predicts that larger, longer-trained models encode negation increasingly as a small shrinking perturbation. It also predicts that a per-model "erasure index" will track the inverse-scaling curves of Michaelov and Bergen on Memo Trap and Pattern Match Suppression. If the index does not correlate with those curves, the geometric story fails regardless of what the adapter does.

The second stage inserts polarity adapters into a few middle-to-deep layers of each frozen Pythia model. The angle predictor is learned, not driven by a parser. We train on paired primitive data in the style of the Logical Fact-Checking suite, covering negation, conjunction, and disjunction, and on held-in negated-prompt templates. We then evaluate on held-out benchmarks. Throughout training we log the norm of the adapter's subspace component. For the real gate we expect to see it dip toward zero, and for the rotation we expect it to stay constant. This directly checks the proposed mechanism, not only the final accuracy.

Baselines

The naive reference point is the unmodified Pythia model at each scale, evaluated zero-shot and 10-shot. For context we also cite published numbers from other model families that we will not re-run. On RTE-neg, Language models are not naysayers reports 80.7% accuracy for InstructGPT and 21.1% for GPT-J-6B. Can Large Language Models Truly Understand Prompts? reports a 31.3% gap between the best models and humans on negated prompts. These are not our models, so the comparisons that decide the question will come only from our own runs.

The strongest existing method is QLoRA supervised fine-tuning on logical primitives, following Logical Consistency of Large Language Models in Fact-Checking. That paper reports average gains of 14% in logical consistency and 19% in accuracy across FreebaseLFC, NELLLFC, and WikiLFC. It is the strongest competitor because it already achieves compositional transfer at low cost. We will run it on the same Pythia models with the same training data. Its LoRA rank will be set so the trainable parameter count equals the adapter's, the same layers will be targeted, and the hyperparameter search budget will be identical. Beating it means matching or exceeding its negated-prompt and consistency gains while losing less than one point of affirmative accuracy, where the trade-off result predicts LoRA will lose more.

Our second external baseline is a text port of NEGTOME from Grounding the "Not". We merge the negation particle with the embedding of its operand and multiply the result by a learned boost, and we train it on the same data with the same parameter budget.

The ablations that isolate the key ingredient are all parameter-matched. The real gated reflection above passes through erasure. A hard-switched Householder reflection is length-preserving but disconnected from the identity, and is trained with a straight-through estimator. A rotation fixed at $\pi$ removes the learned continuous angle. The learned rotation is the full proposal. If the hard reflection matches the learned rotation, then length preservation is doing the work and connectivity to the identity is not, and we will report that narrower claim.

Why not just…?

Why not just scale up? The evidence runs against it. Inverse Scaling: When Bigger Isn't Better reports that GPT-4 still shows inverse scaling on some tasks. Michaelov and Bergen find that on TruthfulQA-MC2, more pretraining tokens predict lower accuracy ($t=-23.89$). Our diagnostic adds a mechanistic check: if the erasure index grows with scale, more scale makes the problem worse.

Why not just LoRA fine-tune on negated and logical data? This is the real threat. The Logical Fact-Checking results show that it works. Its weak point is the trade-off with affirmative accuracy reported by Jang et al. The parameter-matched LoRA comparison, which reports affirmative and negated accuracy separately, settles this. If LoRA shows no trade-off and matches us on negated accuracy, the adapter has no practical advantage, although the diagnostic findings would still stand on their own.

Why not just use a real sign flip on a subspace? In real space, $-1$ is already an exact negation, and a skeptic could reasonably say complex numbers add nothing. We agree that a rotation on paired dimensions is a real-valued operation. The question is how the operator is reached during learning. A continuously gated real flip must pass through zero, and a hard flip must be switched discretely. Both variants appear in our ablations, so this alternative is tested directly and not argued away.

Why not just merge negation tokens and boost them? Boosting makes the negation cue louder but does not tell the model which direction the proposition should move. It also depends on explicit negation words, which leaves out implicit polarity such as "rarely" or antonyms, the cases where the naysayer paper finds models at chance on synonym-versus-antonym recognition. The ported NEGTOME baseline, evaluated on those implicit-polarity items, will show whether this limitation matters in practice.

Decisive Checks

We evaluate on four groups of benchmarks. The first is the nine-task Negated Prompts suite, scored on original prompts, negated prompts, and their average. The second is RTE-neg, MKR-NQ, and synonym-versus-antonym recognition. The third is the Logical Fact-Checking suite scored for consistency, meaning whether the answer to $\neg q$ equals the negation of the answer to $q$. The fourth is the Inverse Scaling Prize tasks Memo Trap, Hindsight Neglect, and Pattern Match Suppression, run through the inverse-scaling evaluation code.

The proposal succeeds only if several things hold together. The erasure index must correlate with inverse-scaling curves across Pythia sizes and checkpoints. The learned rotation must raise negated accuracy without lowering affirmative accuracy by more than one point, while matched LoRA and the real gate do lower it. The slope of negated accuracy against log parameter count must become non-negative with the adapter in place. The logged subspace norms must dip for the real gate and not for the rotation. If the rotation helps but the norms do not behave as predicted, the benefit comes from something other than the proposed geometry and we will say so.