Larger language models often get worse at simple logic. When a prompt says "inappropriate" instead of "appropriate", or "few" instead of "most", big models tend to ignore the small word and give the familiar, memorized answer. This effect is called inverse scaling. We propose to treat the affirmative wording and the negated wording of the same task as two data "domains", and to borrow a tool from domain adaptation to close the gap between them. That tool is an adversary: a small classifier that tries to guess the domain from the model's internal state, while the model is trained to hide it. The twist is that the adversary is shown the correct answer. So the model only has to hide how the question was phrased once the answer is fixed, and it still keeps the negation signal it needs to reach that answer.
The payoff should show up most clearly in ensembles. Soft voting averages the full probability outputs of several models, not just their top picks. It usually helps because members make different mistakes. Under negation, however, members share the same pretraining prior and are wrong together, and wrong with confidence, so soft voting should amplify the error rather than cancel it. Our central, falsifiable prediction is this. Soft voting over unadapted models gains nothing over its best member on negated items, and may lose ground. After label-conditioned alignment, the members' prior-driven errors decorrelate and the ensemble gain returns.
Core Claim
The obvious way to combine these directions is a gradient reversal layer that makes representations indistinguishable between affirmative and negated prompts. A gradient reversal layer passes values through unchanged in the forward pass and flips the sign of gradients in the backward pass, which trains the feature extractor to defeat the domain classifier. Applying it here would be a mistake. Negation flips the label, so a representation that truly cannot tell whether "not" was present has thrown away the operator the task depends on. This is the representation-collapse failure the domain adaptation literature already warns about, taken to its limit.
Our claim is that the correct invariance is conditional. Given the true answer $y$, the model's decision-time representation $h$ should carry no information about whether the prompt was affirmative or negated ($d \in \{A, N\}$). In other words, $h$ should encode "the answer is no", not "the answer is the opposite of the memorized yes". We train a low-rank adapter with
$$\min_{\theta}\max_{\phi}\; \mathcal{L}_{\text{task}}(\theta) - \lambda\, \mathcal{L}_{\text{disc}}\big(D_\phi(h_\theta(x), y),\, d\big),$$
where $\theta$ are the adapter weights, $h_\theta(x)$ is the hidden state at the answer position, $D_\phi$ is a small discriminator that receives both $h$ and an embedding of the gold label $y$ and predicts the domain $d$, and $\lambda$ trades off task loss against the adversarial term.
This setup targets two open problems at once. The first is the zero-sum effect reported by Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts: fine-tuning on negated data degrades performance on affirmative prompts. Tying the two domains together in representation space should resist that trade-off. The second is the out-of-domain collapse shown by Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism, where models shown affirmative demonstrations fail on negated queries.
Research Program
Three lines of evidence motivate the work. Can Large Language Models Truly Understand Prompts? tested OPT and GPT-3 up to 175B parameters and found that larger models do worse on negated instructions. Rarely a problem? found that accuracy after "few"-type quantifiers stays below 20% and falls with scale. The syllogism study above found that even when models are handed a valid reasoning chain, LLaMA-65B drops to 22.8% on the sports task under out-of-domain negation and GPT-4 falls to about 66%. Inverse Scaling: When Bigger Isn't Better names the relevant cause "strong prior": memorized sequences override the instruction.
The mechanism comes from two sources. The adversarial minimax game with a gradient reversal layer comes from Adversarial Domain Adaptation for Classification of Prostate Histopathology Whole-Slide Images, which uses a feature extractor, a domain discriminator and a consistency regularizer to raise cross-site accuracy from 54.3% to 77.1%. Adversarial Topic-Aware Prompt-Tuning for Cross-Topic Automated Essay Scoring shows the same adversary can act on a small tunable prompt module inside a language model. We depart from both by conditioning the discriminator on the label, because in our setting the domain variable interacts with the label rather than being independent noise.
The ensemble side builds on Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems. That paper has each model score every candidate answer with a single forward pass, normalizes the scores by length, and combines them as a product of experts. It works because members' errors are diverse. Our question is exactly whether that diversity survives negation.
We would build paired training data from the syllogism benchmark's sports and occupation tasks, using the base and negated word pairs, and from negated factual cloze items. We then train adapters on three model families at small, medium and large scale. The small members matter because small models sometimes beat large ones under negation: GPT-neo-125M beats the 6B version on negated factual cloze in Language models are not naysayers, and LLaMA-7B beats 65B on the sports task. Soft voting then combines members across scales and families, both before and after adaptation.
Two analyses carry the argument. The first is pairwise error correlation among members on negated items. The second decomposes each member's error into a global label bias, which is the "no-ratio" tendency to favor one label regardless of the derivation, and an item-level component in which the model follows its typicality prior.
Baselines
The naive reference point is each unadapted base model with the syllogism benchmark's own prompts, run at every scale. For this we have reported numbers: 22.8% for LLaMA-65B on the sports task under out-of-domain negation and about 66% for GPT-4, both from the syllogism study.
The second reference is instruction tuning alone, since instruction-tuned models are the cheapest fix a practitioner would reach for. Language models are not naysayers reports 80.7% on RTE-neg for InstructGPT against 21.1% for GPT-J-6B. We run the instruction-tuned variants of our open families zero-shot.
The strongest trained competitor is supervised low-rank fine-tuning on the same paired affirmative and negated data, in the style of Logical Consistency of Large Language Models in Fact-Checking. That paper reports an average 14% gain in logical consistency and 19% in accuracy from fine-tuning on simple logical primitives. It is the strongest competitor because it uses identical supervision and nothing else, so any gain we report must come from the adversary itself. We give it the same adapter rank, data, number of steps and hyperparameter budget. Its learning-rate sweep is matched in size to our joint sweep over learning rate and $\lambda$.
The strongest ensemble competitor is the length-normalized product-of-experts selection of Uncertainty-Aware Answer Selection. It gains about 4% on GSM8K, 3% on MMLU and 5% on ARC over majority voting, but we found no reported result on Lacuna for it on any negation benchmark. We run it over the same member pool, alongside simple cross-scale soft voting and majority voting.
Two ablations isolate the key ingredient. The first is an unconditional discriminator that does not see $y$. We predict it hurts negated accuracy, because it erases the negation operator. The second is the conditional discriminator with $\lambda = 0$, which reduces to plain supervised fine-tuning.
Beating the baselines means three things. On held-out framings, out-of-domain negated accuracy must exceed the matched supervised fine-tuning by a margin larger than the seed-to-seed spread. Affirmative accuracy must fall by no more than its unadapted value minus that spread. And the soft-voting gain over the best single member on negated items must turn from zero or negative to positive.
Why not just…?
Why not just fine-tune on mixed affirmative and negated data? This is the most serious threat, and the fact-checking paper shows it works in distribution. Our bet concerns transfer and retention. Can Large Language Models Truly Understand Prompts? reports that negation fine-tuning costs affirmative accuracy. The syllogism results suggest that learning negation in one framing does not carry over to another. The matched comparison settles the question: we train on the sports and occupation tasks and evaluate on the weight-transitivity task, on the quantifier stimuli and on negated prompts. If supervised fine-tuning matches us on all of them, the adversary adds nothing, and we will report that.
Why not just recalibrate outputs by dividing out the model's prior, with no training at all? The syllogism failures are partly a global label bias, and a training-free label recalibration could remove much of it. We take this seriously and include recalibration as a baseline. The decisive analysis is our error decomposition. If label bias explains most of the out-of-domain collapse, recalibration will close that part of the gap, and our method is justified only by the remaining item-level errors. The clearest test of those is the quantifier stimuli, where the correct word depends on the item's typicality rather than on a fixed label.
Why not just soft-vote across scales, letting weak-prior small models outvote large ones? This is cheap, and it is the natural reading of the file's non-monotonic scaling evidence. The problem is how soft voting treats the large, confidently wrong model. Confidence is precisely what soft voting rewards, so it hands the heaviest weight to the member most driven by the prior. Error correlation and ensemble gain on unadapted members test this directly. If cross-scale voting already helps on negated items, our "amplification" premise is wrong, and the proposal narrows to single-model adaptation.
Why not just wait for scale or use the largest instruction-tuned model as a judge? The evidence points against this. Inverse scaling persists to 175B parameters. GPT-4 itself falls to about 66% under out-of-domain negation. Inverse Scaling: When Bigger Isn't Better reports that some instruction tuning and reinforcement learning from human feedback worsen the effect, which makes a large model an unreliable judge of exactly these items.
Proposed Experiments
Training uses paired items from the syllogism benchmark's sports and occupation tasks and from negated ConceptNet cloze items.
Evaluation covers held-out framings. From the syllogism benchmark we use the weight-transitivity task and all four settings (base, fictional entities, in-domain negation, out-of-domain negation). We add the 960 quantifier stimuli, scored by accuracy and by surprisal sensitivity; RTE-neg and the negation-inference sets from the negation benchmark suite; the negated-prompt versions of PIQA, ARC-Easy and COPA; and the fact-checking consistency suites, scored by whether $\text{LLM}(\neg q) = \neg\,\text{LLM}(q)$.
We report affirmative and negated accuracy separately so any trade-off stays visible, and we use five seeds throughout. Because Episodic Training for Domain Generalization documents both $\lambda$ sensitivity and feature forgetting in adversarial setups, we report full $\lambda$ sweeps rather than a single tuned value.
Three checks are decisive. First, the conditional adversary must beat matched supervised fine-tuning on held-out framings while keeping affirmative accuracy. Second, the unconditional adversary must do worse than both, which confirms that conditioning on the label is the operative ingredient. Third, pairwise error correlation among adapted members must drop on negated items, and the soft-voting gain over the best member must become positive where it was not before.
If the error decomposition shows that recalibration alone closes the out-of-domain gap on the syllogism tasks, we will restrict the claim to item-level prior-following, as measured by the quantifier stimuli.
References
- Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts
- Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism
- Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers
- Inverse Scaling: When Bigger Isn't Better
- Adversarial Domain Adaptation for Classification of Prostate Histopathology Whole-Slide Images
- Adversarial Topic-Aware Prompt-Tuning for Cross-Topic Automated Essay Scoring
- Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems
- Language models are not naysayers: an analysis of language models on negation benchmarks
- Logical Consistency of Large Language Models in Fact-Checking
- Episodic Training for Domain Generalization