Form or Meaning? Auditing LLM Judges with Pivot Paraphrases and Real Web Boilerplate

Large language models are now routinely used as graders ("LLM-as-a-judge"): given a question, a candidate answer, and often a reference answer, the model assigns a score or picks the better of two answers. The selling point is that a judge reads for meaning, unlike word-overlap metrics such as ROUGE. Nobody has cleanly tested that claim. We do not know whether a judge's score still tracks how many words the candidate shares with the reference once meaning is held fixed. We also do not know whether a judge is swayed by the navigation menus, cookie notices and copyright lines that retrieval-augmented systems copy from scraped web pages into their answers. This proposal builds a controlled audit that answers both questions. It uses three kinds of edits. The first is meaning-preserving, vocabulary-changing rewrites produced by round-trip translation through a pivot language. The second is meaning-irrelevant insertions of real web boilerplate. The third is small meaning-changing edits. A trustworthy judge should ignore the first two and react sharply to the third.

In two sentences: we separate how much a judge responds to wording from how much it responds to content, using translation round trips to change wording at fixed meaning and real boilerplate to add noise at fixed meaning. We then test whether averaging a judge's score over several pivot rewrites removes the wording effect better than cheaper fixes such as instructions, same-language paraphrasing, or repeated sampling.

Core Claim

Recent work on judge robustness treats sensitivity in one direction only. Noise-Response Calibration perturbs inputs with synthetic typos and dropped words. It flags judges whose scores do not deteriorate as untrustworthy, which treats sensitivity to form as good. Work on the lexical gap points the other way. Yang et al. find that many correct answers share zero content words with the question, so a good evaluator must be insensitive to vocabulary differences. Both views are right about different perturbations, and no existing protocol measures them together. Our claim is that judge trustworthiness should be scored as a ratio of two effects. The numerator is the score change caused by meaning-changing edits. The denominator is the score change caused by meaning-preserving edits, meaning either vocabulary shifts or boilerplate. The pivot-translation literature supplies a principled generator for the denominator's vocabulary shifts, with graded lexical distance.

Concretely, for a judge $J$, candidate $c$ and reference $r$, we estimate the slope $\beta$ in

$$J(c', r) = \alpha + \beta \cdot \mathrm{overlap}(c', r) + \gamma \cdot h(c') + \epsilon,$$

where $c'$ ranges over human- and entailment-verified pivot rewrites of $c$, $\mathrm{overlap}$ is content-word overlap with the reference, and $h(c')$ is the human quality rating of the underlying answer. A judge that reads for meaning has $\beta \approx 0$. A judge that secretly counts words has $\beta > 0$, which means it behaves partly like ROUGE.

Relation to Prior Work

The method's lineage is bilingual pivoting. Callison-Burch, Koehn and Osborne defined paraphrase probability by summing over foreign phrases $f$, $P(e_2 \mid e_1)=\sum_f P(f\mid e_1)P(e_2\mid f)$. Using that to fill vocabulary gaps raised test unigram coverage from 48% to 90%. ParaEval applied pivot paraphrases to summary evaluation and raised correlation with humans on DUC 2003 from Pearson 0.622 (ROUGE-1) to 0.657. We reuse the marginalization idea, but at the level of whole answers and judge scores rather than phrase tables and n-gram counts.

Both papers also constrain the design. ParaEval's multi-word paraphrases were judged correct only 68% of the time, and Callison-Burch reports 50–70% accuracy on newly covered items. Pivot rewrites therefore cannot be assumed to preserve meaning. They must be filtered and spot-checked by humans before they can serve as "fixed meaning" controls.

The motivation for the audit comes from three sources. The Silent Judge shows GPT-4o flipping pairwise verdicts by up to +30% under recency tags and +18% under "EXPERT" labels, with a 0% rate of acknowledging the cue in its rationale. Its cues are metadata labels, though. We test natural surface features, vocabulary and boilerplate, that arise without any adversary. Judging LLM-as-a-judge documents position and verbosity biases, and verbosity is a related form-over-content effect. How Reliable is Multilingual LLM-as-a-Judge? finds cross-lingual judge agreement around Fleiss' Kappa 0.3. We ask whether part of that inconsistency is plain wording sensitivity that is visible within English.

On the boilerplate side, BoilerNet and Web2Text remove boilerplate from web pages using the page's tree structure. Web2Text reports that cleaning lifts ClueWeb12 retrieval MAP from 0.137 to 0.154. Both tools need that structure. Once a generator copies a cookie notice into a plain-text answer, it cannot be removed that way. That is exactly the regime a judge faces, and no study measures its effect on judges. We use these extractors differently: as a source of real boilerplate strings to inject, not only as cleaners. Disentangling Reasoning Tokens and Boilerplate Tokens shows that predictable template tokens dominate fine-tuning. That result concerns training dynamics. We study inference-time judging, and cite it only to motivate a later extension to trained judges.

Research Program

The first component is a contrastive test suite built on existing human-rated data. We use three sources. The first is SummEval (SummEval), with 12,800 expert and crowd ratings of 100 CNN/DailyMail articles and references available. The second is the reference-guided, non-code categories of MT-Bench. The third is the ELI5 and LitBench pairwise sets from The Silent Judge, which make our shift rates directly comparable to theirs.

For each candidate answer we produce three families of variants. The first family is pivot rewrites. Each answer is translated into German, French and Chinese and back, and also chained through two pivots, which pushes lexical distance further. A variant is kept only if a natural language inference model trained on MNLI judges bidirectional entailment with the original. A stratified sample is also human-checked for meaning preservation, and we report the acceptance rate against ParaEval's 68%.

The second family is boilerplate insertions. Boilerplate blocks are labeled in CleanEval and GoogleTrends-2017 pages, using gold labels where available and BoilerNet output otherwise. These blocks are spliced into answers at the start, the end, or mid-text, at matched lengths. A length-matched neutral filler control separates boilerplate from verbosity bias.

The third family is meaning-changing minimal edits: negations, entity swaps and number changes, verified by hand on a sample. We also cover the reference side. The same pivot operation is applied to the reference instead of the candidate, which changes overlap without touching the answer being graded.

The second component is measurement across four judges drawn from those studied by Fu and Liu: GPT-4o, Gemini-2.0-Flash, Llama-3.3-70b and Qwen-2.5-72b. They are run in pointwise 1–10 grading and pairwise modes. For each judge we measure four things. The first is the overlap slope $\beta$ above. The second is the verdict shift rate under boilerplate, using the Silent Judge definition. The third is the shift under meaning-changing edits. The fourth is the meaning-to-form sensitivity ratio. Following the Silent Judge, we also measure how often rationales mention wording or boilerplate when the verdict moved.

The third component is a mitigation, pivot-marginalized judging. The judge scores $K$ verified pivot rewrites of each candidate, and the final score is their average, $\bar J(c,r)=\frac{1}{K}\sum_{k=1}^{K} J(\mathrm{rt}_k(c), r)$, where $\mathrm{rt}_k$ is round-trip translation through pivot $k$. This is the Callison-Burch sum with a uniform weight over pivots and a judge score in place of a phrase probability. The hypothesis is narrow: averaging over pivots should drive $\beta$ toward zero, because the vocabulary of each rewrite is set by different translation paths rather than by the candidate's author. We do not expect it to fix boilerplate, since translation will usually preserve a cookie notice faithfully. Boilerplate is handled in the audit, and the relevant mitigation there is a plain-text boilerplate filter, compared below.

Baselines

The naive reference point is the judge run once on the original input with the standard MT-Bench-style prompt. It defines the uncorrected slope, shift rates and human correlation. On MT-Bench, Judging LLM-as-a-judge reports GPT-4 agreement with human experts of about 80–85%. For boilerplate-free shortcut cues, the Silent Judge reports GPT-4o shifts of +30% (recency, ELI5) and +18% (expert provenance, ELI5). Those are the magnitudes against which boilerplate effects will be read. On SummEval, expert inter-annotator agreement is Krippendorff's alpha 0.71, which caps meaningful judge–human agreement. We found no reported result on Lacuna for an LLM judge's correlation with SummEval humans. The Distributional LLM-as-a-Judge paper reports only that it beats single-point judging there, without a printed value. We therefore report our own raw-judge numbers as the reference.

The strongest instruction-level competitor is chain-of-thought prompting with an explicit rubric telling the judge to disregard wording, formatting and page furniture and grade meaning only. It is the cheapest fix practitioners would actually deploy. It is run with the same judges and the same test suite, with prompt wording tuned on a held-out 10% split under the same tuning budget we give our own method.

The strongest competitor to pivot marginalization is monolingual paraphrase marginalization. Here, the judge's own model family or a different chat model rewrites the candidate $K$ times in English, filtered by the same entailment check, and the scores are averaged. We also run self-consistency, meaning $K$ sampled judgments of the unchanged input. Every method gets exactly $K$ judge calls plus its rewrite calls, and we report cost.

For boilerplate, the competitor is a sentence-level boilerplate filter applied to the answer text before judging. We fine-tune a small classifier on CleanEval boilerplate versus content sentences, because DOM-based BoilerNet cannot run on plain text.

Pivot marginalization "beats" a baseline if it reduces $\beta$ by a margin whose bootstrap confidence interval excludes zero, without lowering correlation with human ratings on unperturbed SummEval, and without lowering sensitivity to meaning-changing edits. The last condition matters because a method that flattens all scores would trivially shrink $\beta$. The key ablation is pivot versus same-language paraphrase at matched $K$ and filtering. Varying $K$ and the number of pivot languages isolates whether diversity of translation paths is the active ingredient.

Why not just…?

Why not just tell the judge to ignore wording and boilerplate? The Silent Judge found that GPT-4o, writing full rationales, still shifted verdicts under superficial cues and never acknowledged them. Instruction therefore does not reliably reach the part of the judge that is biased. Fu and Liu similarly conclude that prompt engineering is not a fix for cross-lingual inconsistency. Still, their cues were explicit labels, and ours are diffuse. The instructed-rubric baseline could genuinely suffice for vocabulary effects. If it drives $\beta$ to zero, we will report that and drop the mitigation claim, keeping the audit.

Why not just paraphrase in English instead of translating? This is the most serious threat, and it is the main ablation, not an afterthought. There are two reasons to expect a difference, but neither is settled. First, rewrites by a chat model carry that model's style. Self-Preference Bias in LLM-as-a-Judge shows judges favor text resembling their own outputs, so English paraphrasing may raise scores for reasons unrelated to meaning. Second, different pivot languages give systematically different lexical paths, which is the diversity the original pivot formulation exploited. If English paraphrase matches pivot marginalization on $\beta$, human correlation and meaning sensitivity at equal cost, the correct conclusion is that input-side paraphrase averaging is the useful idea and pivoting is incidental. We will state it that way.

Why not just strip the boilerplate first? For HTML inputs, tools like BoilerNet already do this well. The judge's problem is boilerplate that has already migrated into a generated plain-text answer, where structural extractors do not apply and a plain-text filter may miss short or domain-specific lines. Stripping also answers a different question from the audit, which asks whether judges are swayed at all and whether they reward or penalize answers that mimic scraped pages. The sentence-level filter baseline measures how much residual effect survives cleaning.

Why not just sample the judge several times? Self-consistency averages over the judge's randomness, but every sample sees identical wording. If the bias is a stable response to lexical overlap, repeated sampling cannot remove it. The matched-$K$ comparison tests this directly. A self-consistency result that reduces $\beta$ as much as pivoting would falsify our mechanism.

Decisive Checks

The first check is whether $\beta$ is meaningfully positive for any of the four judges on SummEval and MT-Bench once human quality is controlled. If all slopes are near zero, LLM judges already bridge the lexical gap. That is a useful negative result, and the mitigation half of the proposal should be abandoned. The second is whether real boilerplate insertions produce verdict shifts comparable to the Silent Judge's metadata cues, beyond the length-matched filler control, and whether rationales ever mention the inserted text. The third is whether the meaning-to-form ratio ranks judges differently from raw human correlation. That would show the audit adds information that existing agreement numbers hide. As a secondary check, we test whether judges with larger within-English form sensitivity also show lower cross-lingual Kappa on Fu and Liu's XQuAD and WikiLingua data. With only four judges this is suggestive at best, and it will be reported as such. The fourth is the matched-cost comparison of pivot, English-paraphrase and self-consistency averaging, which decides whether bilingual pivoting earns a place in modern evaluation or only in its history.