Language models often get the right answer for the wrong reason. On natural language inference (NLI, deciding whether one sentence follows from another), they learn that high word overlap means "entailed." They then fail on pairs like "The doctor was paid by the actor" versus "The doctor paid the actor." Hand-written rules that generate counter-examples fix part of this, but not all. The best-known result adds a few hundred rule-generated inversions and passives and lifts BERT's accuracy on non-entailed lexical-overlap cases from 0.28 to 0.73. Yet the passive examples produced no clear gain on passive sentences, a ceiling the authors call "the Passive Wall" (Syntactic Data Augmentation Increases Robustness to Inference Heuristics). Large language models (LLMs) can write far more varied syntactic rewrites than rules can. Their free-form output, however, tends to change wording without changing structure, and it can silently corrupt the label.
This note proposes a specific fix. The generator LLM first writes an explicit plan, a short chain of thought (intermediate reasoning written out before the answer), stating who does what to whom and which grammatical relations the rewrite will change. A dependency parser then checks the finished sentence against that plan. Any rewrite whose parse does not match its plan is thrown away. The label of each surviving example follows mechanically from the verified role mapping, not from the LLM's judgment. The scientific payoff is a test of why the Passive Wall exists. If varied, verified passive and embedding data breaks through it, the wall was a data problem. If it does not, that is evidence for the "representational inadequacy" explanation.
Core Claim
The claim is that chain-of-thought helps syntactic augmentation because it produces a machine-checkable commitment, not because the reasoning is richer. A plan such as "agent = actor, patient = doctor; after rewriting, patient becomes passive subject, agent moves into a by-phrase" can be compared with a parser's output. Unplanned rewrites give the checker nothing to compare against. The claim therefore predicts that plan-plus-verification beats both unplanned generation with a generic parse filter and planning without verification. It also predicts that the gain concentrates on the structural subcases of HANS where rules failed: passives, the subsequence heuristic, and the constituent heuristic. HANS is the Heuristic Analysis for NLI Systems benchmark (Right for the Wrong Reasons).
Relation to Prior Work
Two papers supply the method.
Controllable Data Augmentation with Chain-of-Thought Attribute Manipulation (CoTAM) has an LLM decompose a sentence into attributes, plan a change to one attribute, and rewrite the sentence. On SST-2 it reports 79.12% versus 74.28% for label flipping without the plan. We reuse the decompose, plan, rewrite structure. CoTAM, however, manipulates semantic attributes such as sentiment and topic, and it trusts the rewrite. We manipulate grammatical relations and verify every rewrite externally.
Adaptive Problem Generation via Symbolic Representations shows the value of editing a formal representation and checking it with a solver. It measures an embedding cosine distance of 0.37 for its symbolic edits versus 0.07 for edits made in plain text. It works only on math problems written as SymPy code. A dependency parse is our analogue of the solver, and it is weaker: it checks structure, not truth.
Several papers motivate the proposal.
- The rule-based study above documents the Passive Wall.
- HANS shows that NLI models collapse on examples that contradict the heuristics (Right for the Wrong Reasons).
- BLiMP shows that even large pretrained models stay weak on argument structure and island effects.
One paper constrains the design. MultiBLiMP 1.0 finds that instruction-tuned chat models underperform their base models on grammatical minimal pairs. The instruction-tuned generators we would use are therefore exactly the kind of model whose grammar we should not trust unchecked, and this is the reason for the external parser check.
Two further papers mark the boundaries of the idea.
- Chain of Thought Prompting Elicits Knowledge Augmentation appends reasoning chains to a small model's input as hints. It does not use them to build new training sentences.
- SynCode enforces formal grammars during decoding. It only works where a complete grammar exists, which natural language lacks.
Research Program
We start from MNLI premises, taking the same kind of seed sentences as the rule-based study. An open-weights instruction model, from the Qwen2.5 family used by Opt-Sym, receives a sentence and a target construction. The constructions are:
- a passive with an agent phrase;
- a passive without an agent phrase;
- a subject–object role swap under a passive;
- a relative clause that embeds a false subsequence;
- an embedding under non-factive verbs or adverbs, which creates the constituent heuristic's traps.
The model first emits a structured plan in a fixed template:
- the predicate and its thematic roles, meaning who acts and who is affected;
- the intended grammatical function of each role after rewriting;
- the required auxiliary and morphology changes;
- the entailment relation the rewrite should bear to the original.
It then writes the sentence.
The verifier parses both the original and the rewrite with a Universal Dependencies parser trained on English-EWT. It accepts the rewrite only if three conditions hold. First, the content lemmas are preserved. Second, the dependency arcs realize the planned mapping, for example the planned patient appears as nsubj:pass and the planned agent as an obl headed by "by". Third, two differently trained parsers agree.
Parsers themselves have limited coverage on unusual constructions (Automatic Manipulation of Training Corpora). We therefore track the rejection rate for each construction, and we have humans audit a random sample of both accepted and rejected rewrites. The audit estimates the verifier's precision and how often it wrongly rejects good rewrites.
The final example pairs the original premise with the rewrite as hypothesis. Its label is computed from the verified mapping. A meaning-preserving passive yields entailment. A role-swapped passive or a false embedded subsequence yields non-entailment. We fine-tune BERT on MNLI plus these examples at two sizes: the rule-based study's 400 examples and roughly 8% of training size, the scale used in the HANS paper.
Baselines
Plain MNLI. The naive reference point is BERT fine-tuned on MNLI alone. The rule-based study reports 0.28 accuracy on HANS non-entailed lexical-overlap cases for this model (Syntactic Data Augmentation Increases Robustness to Inference Heuristics).
Rule-based augmentation. The strongest existing method is that paper's rule-based inversion plus passivization. It reports 0.73 on the same subset, with no loss on MNLI. We rerun it on our exact seed premises and at our augmentation sizes, rather than relying only on the published figure. For the passive-specific and subsequence subsets, no reported result was found on Lacuna, so our rerun supplies those numbers.
Unplanned LLM rewriting. This is the competitor most likely to erase the gain. The same generator rewrites into the target construction without a plan. It is tested twice: unfiltered, and filtered by a generic parse check. The generic check requires only that the rewrite parses as the target construction and preserves the content lemmas.
Planning without verification. This is a CoTAM-style baseline: the same plan template, but no parser check, and the label is taken from the plan.
Fairness conditions. Every condition uses the same seed sentences, generator, and number of accepted training examples. Each also gets the same fine-tuning hyperparameter search with five random seeds. Generator token budgets are reported, so that planning's extra cost is visible.
Success criterion. The proposal beats the field if it exceeds the rule-based method and the unplanned-plus-filter baseline on the passive, subsequence, and constituent non-entailed subsets. The margin must exceed the variation across seeds, it must at least match 0.73 on lexical overlap, and it must cause no MNLI drop.
Isolating each ingredient. Two ablations separate the two parts of the method. Comparing "plan + verify" with "unplanned + generic filter" isolates the plan. Comparing "plan + verify" with "plan only" isolates the verifier.
Why not just…?
Why not just keep the rules? Rules are cheap and already reach 0.73 on lexical overlap. By their authors' own report, however, they did not move performance on passives and struggled on subsequence cases (Syntactic Data Augmentation Increases Robustness to Inference Heuristics). The likely reason is that rules produce a narrow band of surface forms. The rerun on our seeds tests this directly. If rules match our method on those subsets, the proposal fails.
Why not just prompt the LLM directly and filter with a parser? This is the real threat. A strong generic filter may catch most bad rewrites without any plan. The evidence against it is indirect: unplanned text edits tend to be superficial (0.07 versus 0.37 diversity in Adaptive Problem Generation via Symbolic Representations), and CoTAM's plan beat plain flipping on SST-2. Neither result is about syntax. The unplanned-plus-filter ablation settles the question. If it ties, the finding becomes "verification, not reasoning, is what matters," and we would report it that way.
Why not just use a bigger pretrained model? Scale has not closed these gaps. GPT-2 sits at 83.0% on BLiMP against 88.6% for humans, and near chance on island effects (BLiMP). Instruction tuning lowers grammatical competence (MultiBLiMP 1.0). We will also report HANS results for a larger encoder trained on plain MNLI, to show how much of the gap scale alone closes.
Why not constrain decoding with a grammar? SynCode-style masking needs a complete formal grammar, which natural language lacks. It also guarantees only well-formedness, not the preserved roles that make the label correct (SynCode).
Proposed Experiments
Main evaluation. The primary benchmark is HANS, 30,000 examples, reported for each heuristic and each subcase. The passive, subsequence, and constituent non-entailed subsets are the key measurements. The MNLI matched and mismatched sets check that general accuracy does not fall.
Generator audit. We evaluate generator output quality through the human audit and the per-construction rejection rates. We also report the structural diversity of each augmentation set, using the embedding-distance measure from Opt-Sym, so that "more varied passives" is measured rather than asserted.
Decisive checks. Three results decide the proposal.
- Does plan plus verification break the Passive Wall where rules and unplanned rewriting do not? A yes favors the data-shortfall explanation over representational inadequacy.
- Does removing the verifier lower label accuracy in the audit, and lower HANS accuracy with it? This would confirm that an unchecked chat-model generator is a liability.
- Do gains scale from 400 examples to about 8% of training size without MNLI loss?
A clean negative result on the first check is also informative. If even varied, verified passives leave BERT at the wall, that is the strongest evidence yet for the representational-inadequacy explanation.
References
- Syntactic Data Augmentation Increases Robustness to Inference Heuristics
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
- Controllable Data Augmentation for Few-Shot Text Mining with Chain-of-Thought Attribute Manipulation
- Adaptive Problem Generation via Symbolic Representations
- BLiMP: The Benchmark of Linguistic Minimal Pairs for English
- MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
- Chain of Thought Prompting Elicits Knowledge Augmentation
- SynCode: LLM Generation with Grammar Augmentation
- Automatic Manipulation of Training Corpora to Make Parsers Accept Real-world Text