Large language models are now routinely used as graders. A strong model reads a candidate answer, sometimes beside a reference solution or a competing answer, and outputs a score or a preference. This "LLM-as-a-judge" setup drives leaderboards, preference data for alignment, and regression tests. We already know these judges can be swayed by surface cues: answer length, fake citations, labels saying who wrote the text, and their own familiar phrasing. One basic question has not been tested. If a correct answer is rewritten into a different grammatical shape with the same meaning, does the judge still call it correct? Examples of such rewrites are passive voice ("the root is found by…"), a negated contrary ("this method does not fail when…"), a conditional frame, a nominalization, or the dropped punctuation and verbless fragments common in informal text.
We propose a controlled audit of this question and a test of what causes the failures it finds. The core claim is that LLM judges have a measurable syntactic-invariance failure: equivalent answers in non-canonical grammatical form receive different verdicts, at rates above the judge's own run-to-run noise. The audit also asks which of two explanations fits. The first is that the judge simply prefers text it finds predictable, the low-perplexity "familiarity" effect. The second is that it misreads the scope of negation and conditional operators, the way older parsers misattached modifiers in long sentences. The two explanations call for different fixes, so separating them is the point of the work, not a side analysis.
What Prior Work Establishes and Leaves Open
The motivating evidence comes from two places that have not been joined. Syntactic Framing Fragility showed that rephrasing an ethical question with "should not," "even if," or double negation flips direct answers from 23 LLMs, with a mean Syntactic Variation Index of 0.53. That audit asks models for their own yes/no decisions. It never puts them in the grader's seat, and it never perturbs a third-party text being graded. The judge-bias literature, in turn, perturbs everything except grammar. Justice or Prejudice? (CALM) injects 12 bias cues, including verbosity, tone, and fabricated citations, while holding correctness fixed. The Silent Judge shows that provenance and recency labels shift verdicts (up to +30% verdict shift for GPT-4o on ELI5) while the written rationales never mention the cue. In both cases the perturbations are additions or metadata, not changes to the sentence structure of the answer.
Two papers constrain how the results should be read. Self-Preference Bias in LLM-as-a-Judge traces self-preference to judges favoring low-perplexity text. Any syntactic effect we find could therefore be perplexity under another name, so we must control for it. Does syntax matter? shows that fine-tuned RoBERTa builds its own task-shaped internal syntax that beats parser trees. That cuts against the expectation that modern models mishandle structure, and it means we should not assume structural failure; we test for it.
The older parsing literature supplies the method. Automatic Manipulation of Training Corpora defines two concrete informal-text transforms that break dependency parsers: dropping terminal punctuation and standalone verbless noun phrases. We reuse both as perturbation families, now aimed at an LLM grader rather than a parser. Parsing Run Amok argued that extraction should run on head–role–filler relations rather than full trees, because attachment ambiguity overwhelms full parsing. That idea becomes one candidate defense: grade a relation-level abstraction of the answer rather than its surface form.
Core Claim and Mechanism Test
On correctness-only grading tasks, LLM judges flip their verdicts on meaning-preserving syntactic rewrites more often than on resampled or re-ordered identical inputs. Flips should concentrate in rewrites that move a polarity or conditional operator away from the word it governs. For a judge $J$, a set of answers $a$, and a rewrite family $T$, we define the syntactic flip rate as
$$\mathrm{SFR}(J,T) = \Pr_a\big[\,J(a) \neq J(T(a))\,\big] - \Pr_a\big[\,J(a) \neq J'(a)\,\big],$$
where $J'(a)$ is the same judge re-run on the unchanged answer (different sample, swapped position in pairwise mode). The second term is the judge's own noise floor, so an SFR above zero is an effect of the rewrite itself.
To separate the two mechanisms, we fit, across all rewrite pairs, a logistic model of whether a flip occurs:
$$\Pr[\text{flip}] = \sigma\big(\beta_0 + \beta_1\,\Delta\mathrm{PPL} + \beta_2\,\Delta d_{\mathrm{op}} + \beta_3\,\Delta\mathrm{len}\big).$$
Here $\sigma$ is the logistic function. $\Delta\mathrm{PPL}$ is the change in the judge model's own log-perplexity on the answer; for closed models where this is unavailable, we use an open proxy from the same family. $\Delta d_{\mathrm{op}}$ is the change in dependency distance between any negation or conditional operator and the head it scopes over, from a parser checked on these sentences. $\Delta\mathrm{len}$ controls for the length change that rewrites inevitably introduce. The familiarity account predicts that $\beta_1$ carries the effect. The scope-misreading account predicts that $\beta_2$ stays significant after perplexity is controlled.
Research Program
We first build a perturbation suite of eight rewrite families. Five are meaning-preserving but structurally marked: passive, cleft, nominalization, fronted adverbial, and relative-clause stacking. One applies polarity: the "not the contrary" rewrite, plus a conditional frame mirroring the operators in Syntactic Framing Fragility. Two are informal: punctuation omission and noun-phrase fragments, following Automatic Manipulation of Training Corpora. Rewrites are generated by a strong model under explicit instructions. Each is kept only if it passes two-way entailment checks by two independent models and, for a stratified 15% sample, a human equivalence check. We report the human rejection rate so the suite's validity is visible.
We then apply the suite on the judge side. We test frontier and open judges and the reward-model classifiers covered by RewardBench, in three formats: reference-guided correctness grading, single-answer Likert grading on a correctness-only rubric, and pairwise comparison in which only one side is rewritten. Restricting rubrics to correctness matters. On fluency rubrics a judge may legitimately prefer canonical prose, so we keep those as a contrast condition rather than the headline. In a second arm we leave the answer alone and rewrite the rubric itself between positive and negative framing ("score 5 if faithful" versus "do not score 5 unless no claim is ungrounded"), directly transferring the framing manipulation into the grading instruction.
Finally, we fit the mechanism model above. We also check, as The Silent Judge did for metadata cues, whether rationales accompanying a flip mention phrasing at all, or instead invent a content defect.
Baselines
The naive reference point is the unperturbed judge's noise floor: the same model on the same answer under a fresh sample and, in pairwise mode, swapped order, as recommended in Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Every SFR is reported net of this floor. A rewrite family counts as harmful only if its flip rate exceeds the floor with a bootstrap 95% interval clear of zero.
The strongest existing judge configuration is a chain-of-thought judge with position swapping. It is the strongest competitor because Syntactic Framing Fragility found that eliciting reasoning substantially reduced framing swings, and MT-Bench reports roughly 80–85% GPT-4 agreement with human experts using this kind of setup. On RewardBench, the strongest reward model reported in the file is ArmoRM-Llama3-8B at an overall score of 89.0. For syntactic-rewrite flip rates on any of these judges, there is no reported result found on Lacuna. The closest reported figure is the 0.53 mean variation index for direct ethical answers, which is not a grading task. The chain-of-thought judge runs on identical items, with the same decoding settings and the same token budget per verdict as every defense arm. It is "beaten" in the defense experiments if a defense lowers SFR by a statistically reliable margin at matched cost, without lowering agreement with human labels on unperturbed MT-bench items.
Three defenses isolate the key ingredient. The first is paraphrase averaging: score $k$ random paraphrases and take the majority. The second is canonical rewriting: one rewrite into plain active declaratives before grading. The third is relation-level abstraction in the spirit of Parsing Run Amok and Molecular Facts: convert the answer to minimal self-contained claims with explicit polarity and then grade. Comparing canonical rewriting against paraphrase averaging at equal calls tests whether normalization does anything beyond averaging out noise. Comparing relation abstraction against canonical rewriting tests whether making operator scope explicit matters beyond smoothing style.
Why not just…?
Why not just use chain-of-thought? This is the real threat, since chain-of-thought is known to dampen framing effects in direct decisions. Two things keep it from settling the question in advance. Reduced is not removed: the framing audit reports dampening, not elimination. And The Silent Judge shows reasoning traces can rationalize cue-driven verdicts with a 0% cue acknowledgment rate. Our chain-of-thought arm settles it empirically. If chain-of-thought drives SFR to the noise floor across all eight families, we report that as the main finding and drop the defense program.
Why not just average over paraphrases? Averaging cancels idiosyncratic flips but not shared directional ones. If judges systematically mark down negated-contrary phrasings, random paraphrases of such answers inherit the bias. The equal-budget comparison against canonical rewriting tells us which case holds.
Why not just control for perplexity, since self-preference already explains phrasing effects? If the mechanism model shows $\beta_2$ collapsing once $\Delta\mathrm{PPL}$ is included, then this is the familiarity effect of Self-Preference Bias under a new name. The right fix would then be perplexity-aware calibration, not structural normalization. We treat this as a live outcome. The polarity rewrites are the sharpest test, because they can lower perplexity (a common idiom) while lengthening operator scope.
Why not just parse the answer with an off-the-shelf parser and grade the tree? The informal families are exactly the inputs on which UD parsers are documented to break (Automatic Manipulation of Training Corpora), and parser trees underperform learned structure in Does syntax matter?. We use a parser only as a measuring instrument, checked on a hand-annotated subset, never inside the grader.
Proposed Experiments and Decisive Checks
The data come from the correctness-bearing CALM sources in Justice or Prejudice?: GSM8K, MATH, ScienceQA, and TruthfulQA. For each source we take paired correct and incorrect candidate answers. We add MT-bench items with human votes from Judging LLM-as-a-judge, and ELI5 and LitBench pairs from The Silent Judge as an open-ended contrast. RewardBench prompt–response pairs test reward-model judges.
Metrics are SFR net of noise, the change in accuracy against gold correctness labels, agreement with human MT-bench votes, the rewrite-acknowledgment rate in rationales, and the mechanism coefficients.
The decisive checks are three. First, does any rewrite family push a chain-of-thought, position-swapped judge's SFR reliably above its noise floor on correctness grading? If not, the hypothesis fails. Second, does the operator-distance coefficient survive perplexity and length controls? This decides between scope misreading and familiarity. Third, does relation-level abstraction beat paraphrase averaging at equal calls, specifically on polarity rewrites, without lowering agreement on unperturbed items? That is the condition under which structural normalization earns its place as a judge-side defense.
References
- Syntactic Framing Fragility: An Audit of Robustness in LLM Ethical Decisions
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Self-Preference Bias in LLM-as-a-Judge
- Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa
- Automatic Manipulation of Training Corpora to Make Parsers Accept Real-world Text
- Parsing Run Amok: Relation-Driven Control for Text Analysis
- RewardBench: Evaluating Reward Models for Language Modeling
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
- Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification