Systems that answer questions by first fetching documents, known as retrieval-augmented generation (RAG), are now routinely scored by another large language model acting as a grader, an approach called "LLM-as-a-judge." In multi-turn professional consultations, such as a client asking a legal assistant five follow-up questions about a tenancy dispute, the grader's prompt usually holds a great deal: the whole dialogue, the retrieved statutes, the system's answer, and an expert reference. Language models are known to perform worse when relevant information is buried among irrelevant text. The obvious question has not been tested: does the grader suffer from the same problem? If it does, published comparisons of conversational RAG systems in law and medicine partly measure how much clutter the grader was shown, and not only how good the answers were.
This note proposes to measure that effect directly and to fix it using a tool from conversational search. The tool is query rewriting, which turns a context-dependent follow-up ("what if he refuses?") into a standalone question ("can a landlord withhold a deposit if the tenant refuses to sign the inspection report under Article X?"). The proposal applies rewriting to the grader's input instead of the search engine's input. The grader then sees only a standalone question, the expert reference broken into atomic legal points, and the answer. It never sees the raw history or the retrieved passages. The central prediction is that grader distraction is worse in domains where the grader lacks knowledge, because a grader that does not know the law cannot tell a relevant statute from a lexically similar irrelevant one.
Core Claim
Our claim has two parts. First, a standard LLM judge evaluating multi-turn legal RAG changes its scores, and its preferences between systems, when content-irrelevant material is added to its prompt while the answer stays exactly the same. This sensitivity should be larger on legal consultation than on general-domain conversational QA. Second, a context-isolated judge removes most of this sensitivity. It works by rewriting the dialogue into a standalone question, decomposing the reference into atomic points, and excluding retrieved evidence. It should do so without losing the ability to catch real errors, and it should agree better with human legal experts. If both parts hold, some system rankings on the LexRAG benchmark should change once the grader is no longer distracted. That includes the size of its reported "noisy context trap."
Where this sits in prior work
The work that motivates this proposal is LexRAG. It is a benchmark of 1,013 five-turn Chinese legal consultations over 17,228 statutory articles. It found that LLM query rewriting is needed for retrieval, and that noisy retrieved statutes push generation quality below a no-retrieval baseline. It scores answers with a Qwen-2.5-72B judge on five dimensions against expert answers.
LexRAG treats its judge as a fixed measuring instrument. We treat that judge as an object of study. We do not assume the LexRAG judge's prompt contains the retrieved statutes. Instead, we test judge configurations that do and do not include history and evidence. The point is to find out how much of any measured gap depends on that choice.
Evidence on contextual distraction comes from studies of the models being graded, not of graders. Reader accuracy drops as more documents are retrieved (Improving Zero-shot Reader by Reducing Distractions). Buried numbers break calculation in document analysis (UDA). History causes "contextual drowning" in conversational retrieval, measured by a Historical Interference Rate (Learning Contextual Retrieval for Robust Conversational Search). None of these studies examines a model in the grading role.
Two judge papers supply the method and also constrain it. ConsJudge reduces sensitivity to how the grading prompt is worded. It samples several prompts, picks the most central reasoning path, and trains the judge with Direct Preference Optimization. This raised consistency from 0.75–0.84 to over 0.90 on HotpotQA and TriviaQA. But every prompt variant it uses sees the same input content. ConsJudge therefore cannot remove a bias that all variants share, such as being distracted by an irrelevant statute.
LeMAJ decomposes legal answers into atomic "Legal Data Points" and tags each as correct, incorrect, irrelevant, or missing. It reaches Pearson 0.700 with human correctness ratings on a LegalBench subset, against 0.170 for ROUGE-L. However, it assumes clean, single-document context. We reuse its decomposition as one ingredient and test it in the multi-turn, noisy setting it leaves open.
Our rewriting step borrows from conversational query rewriting. Examples are zero-shot LLM rewriters (Rewriting Conversational Utterances with Instructed LLMs) and topic-switch pruning in CHIQ. Note one point from Informative Query Rewriting: it argues rewrites should be maximally informative for retrievers. We argue the opposite for graders. A grader needs the question to be complete, not expanded. We test that difference explicitly.
Research Program
The first component is a perturbation audit. For each graded instance from LexRAG, we fix a dialogue turn, a system answer, and the expert reference. We then build variants of the judge's prompt that change only material irrelevant to the answer's correctness.
The perturbations come in five kinds. The first inserts $k$ statutes from the same legal area that are lexically close but not applicable; these are the hard distractors. The second inserts $k$ statutes from an unrelated area. The third prepends turns from a different consultation, simulating a topic switch. The fourth replaces the raw history with an oracle rewrite. The fifth reorders evidence.
We define the judge's noise sensitivity as the mean absolute score change across these perturbations. We also define a pairwise flip rate: how often the judge's preference between two systems' answers reverses under perturbation. To make sure robustness is not bought by numbness, we also apply controlled answer corruptions. One kind swaps a cited article for a wrong one; another drops a required legal condition. We measure detection rate on these corruptions. A good judge should have low noise sensitivity and high detection.
We run the same audit on TopiOCQA (TopiOCQA). It is a general-domain conversational benchmark with frequent topic switches, and it serves as the low-knowledge-gap control. The domain-gap hypothesis predicts that hard distractors hurt the judge more on LexRAG than on TopiOCQA. We also run it at two judge sizes, roughly 7B and 72B. If the effect simply disappears at 72B, the problem is one of scale, not a structural flaw.
The second component is the context-isolated judge. It has three stages:
- Rewrite. An LLM rewrites turn $t$ and its history into a standalone question.
- Decompose. The expert reference is broken into atomic legal points, following LeMAJ.
- Grade. The judge grades the answer against only the standalone question and the reference points. Any statute the answer cites is fetched by article identifier, so that citation correctness can still be checked without including the retrieved pool.
The rewrite and decomposition depend only on the dialogue and reference, not on the system being graded. They are therefore computed once per turn and cached, which keeps the extra cost small when many systems are compared.
Rewrite errors are the main risk of this design. A wrong rewrite would make the judge grade the wrong question. We therefore measure how faithful the rewrite is, using human checks on a sample. We also test a fallback that includes the raw last turn alongside the rewrite.
The third component is human validation. We will collect expert ratings for about 600 answers sampled across systems and turns on LexRAG. Each answer gets factuality and completeness ratings from three law-trained annotators. This gives a ground truth for both per-answer correlation and system-level ranking agreement. No such ratings for judge robustness are available in the file, so this collection is necessary.
Baselines
The naive reference point is the vanilla full-context judge as LexRAG uses it. That means Qwen-2.5-72B-Instruct with chain-of-thought over the history, retrieved statutes, answer, and expert reference. LexRAG reports it gives the best systems about 7.4, against 8.0 for expert answers. We will run it with the released LexiT toolkit (CSHaitao/LexRAG) so prompts match the published setup. ROUGE-L against the expert answer serves as a lexical floor; LeMAJ reports it at Pearson 0.170 for legal correctness.
The strongest competitor on robustness is ConsJudge. It is the only judge in the file trained explicitly for consistency. Its reported consistency of over 0.90 shows it can absorb some kinds of variation. We will reimplement it with the same base model and the same number of prompt variants as described. We will train it on the same LexRAG training split, with the same preference-pair budget our method uses for any tuning. It will then be evaluated on the same perturbed instances. No reported result was found on Lacuna for ConsJudge, or any judge, under context-injection perturbations on LexRAG.
The strongest competitor on agreement with experts is LeMAJ-style atomic judging given the full noisy context. LeMAJ reports Pearson 0.700 on correctness. That figure was measured on clean single-document LegalBench, not on LexRAG, so it is a reference point and not a direct target.
Our method beats these baselines if two things happen on the human-rated LexRAG set. First, it lowers noise sensitivity and flip rate relative to both ConsJudge and full-context LeMAJ, with confidence intervals that do not overlap. Second, it matches or exceeds their Pearson correlation with expert ratings and their corruption detection rate.
Two ablations isolate the key ingredient. The first removes the rewrite and gives the judge the raw history plus reference points. This tests whether rewriting, rather than evidence removal, matters. The second removes all history and gives only the last turn. This tests whether a standalone question is needed at all.
Why not just…?
Why not just drop the history and retrieved passages from the judge's prompt, since there is an expert reference anyway? This is the most serious alternative, and it is our "last turn only" ablation. The risk is that follow-up turns in legal consultation depend heavily on earlier turns, through pronouns and ellipsis. LexRAG found that the last query alone fails for retrieval for exactly this reason. A judge that does not know what "he" refers to may reward an answer to the wrong question. If the last-turn judge matches our judge on expert correlation and corruption detection, we will report that the rewriting step is unnecessary. The contribution would then reduce to the audit plus a simple recommendation, and we would say so.
Why not just use ConsJudge's multi-prompt consistency training? ConsJudge reduces variance from how the grading prompt is worded. But every variant still reads the same distractor statutes. A bias that all variants share survives consensus, and may even be reinforced by it. The perturbation audit tests this directly. If ConsJudge's flip rate under hard-distractor insertion is as low as ours, then content isolation adds nothing beyond consistency training, and that would count against our proposal.
Why not just use a larger judge? Scale may reduce distraction. But Improving Low-resource QA by Augmenting Question Information and TheoremQA both indicate that size alone does not supply missing domain knowledge. Our hypothesis says distraction comes from that missing knowledge. Running the audit at two sizes settles the question. If the 72B judge is insensitive on LexRAG, the problem is one of scale and our fix matters mainly for cheap judges. That is still useful, but it is a narrower claim.
Why not just use lexical or embedding metrics? They are immune to prompt clutter because they ignore context. But LeMAJ shows they barely track legal correctness, with ROUGE-L at 0.170 and DeepEval at 0.018. They stay in the comparison only as a floor.
Decisive Checks
The proposal stands or falls on three results.
- Does the vanilla judge actually move? Same-area statute insertion must shift its LexRAG scores and flip pairwise preferences more than unrelated-area insertion does. The effect must be clearly larger than on TopiOCQA. If no shift appears, the premise is false and we stop.
- Does isolation fix the right thing? The context-isolated judge must cut sensitivity while keeping corruption detection and expert correlation. It must also beat the last-turn-only ablation on turns 3–5, where dependence on earlier turns is greatest.
- Do conclusions change? We re-rank a fixed set of LexRAG systems under both the vanilla and isolated judges. The systems are raw-history retrieval, last-query retrieval, zero-shot rewriting, informative rewriting, and CHIQ-style rewriting. We report Kendall's tau against the expert ranking, and the size of the gap between noisy retrieval and no retrieval under each judge. A changed ranking, or a shrunken noisy-context trap that agrees better with the experts, would show that the grader's own distraction has been shaping conclusions in domain conversational RAG.
References
- LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- Learning Contextual Retrieval for Robust Conversational Search
- Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models
- LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation
- Rewriting Conversational Utterances with Instructed Large Language Models
- CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search
- Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
- TopiOCQA: Open-domain Conversational Question Answering with Topic Switching
- Improving Low-resource Question Answering by Augmenting Question Information
- TheoremQA: A Theorem-driven Question Answering Dataset