TL;DR
When LLMs act as judges, asking a single model call to evaluate too many criteria at once causes severe oversight failures, a vulnerability that adversaries can easily exploit. This paper introduces sharding—splitting a large set of evaluation criteria into smaller groups across separate API calls—which restores expert-level accuracy and robustly blocks adversarial presentation attacks without requiring larger compute budgets or more advanced models.
What Problem Does This Paper Address?
Using LLMs as automated judges ("LLM-as-a-judge") is now standard practice for evaluating code, long-form text, and complex rubrics. Typically, developers pass an entire artifact along with a massive checklist of requirements into a single prompt.
The authors reveal a fundamental flaw in this setup: a decision-load bottleneck. As the number of requirements (verdicts) per call grows, the LLM's ability to ground its decisions in actual evidence degrades. Instead of thoroughly checking every item, the overloaded model defaults to surface-level cues. Worse yet, this blind spot is highly exploitable. An adversary can execute a "presentation attack" by keeping the underlying work fixed but tweaking its formatting or summary to trick an overloaded judge into approving genuinely unmet criteria.
Methods
To solve this, the paper isolates decision load from the computational budget using a technique called sharding:
- Sharding: The total pool of $K$ criteria is partitioned into $S$ smaller, disjoint groups. Each group is evaluated by a separate parallel LLM call that still sees the full context but only outputs a fraction of the verdicts.
- Baselines for Comparison: The authors compare sharding against a standard Solo judge (one call for all criteria) and a Full-budget solo judge (one call given the same total token/tool budget as the sharded panel, often utilizing step-by-step instructions).
- Defenses Against Adaptive Attacks: For sophisticated adversaries who adaptively persuade judges on a criterion-by-criterion basis, the authors layer debate-style opposition (assigning a skeptical challenger to argue why a criterion is unmet) on top of the sharded base.
Datasets
The authors evaluated their approach across diverse, expert-labeled domains requiring deep contextual understanding:
- PaperBench (JudgeEval): Code-replication research submissions evaluated against dense hierarchical rubrics.
- JudgmentBench: Expert-lawyer grading of legal contracts and work products against professional rubrics.
- ROBoto2: Expert Cochrane risk-of-bias assessments of clinical trials.
- SWE-Bench Pro & Oracle Cyber-Verification (ARVO): Software engineering and memory-safety verification tasks used as direct-reading controls.
- FActScore: A structural control dataset consisting of easy, independent atomic fact checks.
Key Insights and Ideas
- Compute Doesn't Cure Overload: Simply giving a single LLM call more tokens or forcing it to reason step-by-step does not fix the decision-load bottleneck. The issue is how attention is divided across multiple distinct outputs, not a lack of raw compute.
- Weaker Judges Can Beat Stronger Ones: Remarkably, a weaker model (like Claude Sonnet) using a sharded approach can outperform a more capable model (like Claude Opus) evaluating holistically.
- Context-Dependent Benefits: Sharding provides massive accuracy gains when the criteria require an integrated understanding of a large shared context (e.g., legal contracts or research papers). However, for independent tasks where text can be evaluated locally (like checking separate code patches in SWE-Bench), sharding offers no distinct advantage.
Key Results
- Restoring Accuracy: On PaperBench, as the decision load scaled to 116 criteria, the holistic judge's agreement with human experts plummeted. Sharding completely arrested this decline, maintaining a high Cohen's kappa (~0.73 to 0.79) across all workloads.
- Replication on Legal Work: On JudgmentBench, a single judge's agreement with expert lawyers dropped sharply past 12 criteria. At full load, the sharded judge significantly outperformed both the single judge and the full-budget single judge controls.
- Thwarting Adversaries: In a best-of-8 presentation attack on PaperBench, an adversary successfully doubled the over-acceptance rate of unmet criteria against a holistic judge. Sharding consistently mitigated this vulnerability across all tested submissions.
- Defeating Adaptive Persuasion: Against an adversary that iteratively rewrote compliance memos to target specific criteria, plain sharding eventually faltered. However, adding a one-sided opposing advocate (debate) successfully restricted the cumulative attack surface from 0.55 down to 0.39.
Limitations and Future Work
While highly effective, sharding introduces a clear trade-off: token cost. Because the entire context (which can be up to 183,000 tokens in legal tasks) must be re-sent to every parallel shard call, API costs scale linearly with the number of shards. The authors note that intermediate capacity-sized grouping (e.g., batches of 32 criteria) offers a practical sweet spot that preserves accuracy while drastically reducing token overhead compared to maximal sharding (1 criterion per call). Future work will likely focus on leveraging platform-level context caching to minimize these redundant costs.
Summary updated