Sharding Prevents LLM Oversight Failures and Adversarial Exploitation

Victor Akinwande, J. Zico Kolter, Aran NayebiarXiv 2026 · arXiv

TL;DR

When LLMs act as judges, asking a single model call to evaluate too many criteria at once causes severe oversight failures, a vulnerability that adversaries can easily exploit. This paper introduces sharding—splitting a large set of evaluation criteria into smaller groups across separate API calls—which restores expert-level accuracy and robustly blocks adversarial presentation attacks without requiring larger compute budgets or more advanced models.

What Problem Does This Paper Address?

Using LLMs as automated judges ("LLM-as-a-judge") is now standard practice for evaluating code, long-form text, and complex rubrics. Typically, developers pass an entire artifact along with a massive checklist of requirements into a single prompt.

The authors reveal a fundamental flaw in this setup: a decision-load bottleneck. As the number of requirements (verdicts) per call grows, the LLM's ability to ground its decisions in actual evidence degrades. Instead of thoroughly checking every item, the overloaded model defaults to surface-level cues. Worse yet, this blind spot is highly exploitable. An adversary can execute a "presentation attack" by keeping the underlying work fixed but tweaking its formatting or summary to trick an overloaded judge into approving genuinely unmet criteria.

Methods

To solve this, the paper isolates decision load from the computational budget using a technique called sharding:

  • Sharding: The total pool of $K$ criteria is partitioned into $S$ smaller, disjoint groups. Each group is evaluated by a separate parallel LLM call that still sees the full context but only outputs a fraction of the verdicts.
  • Baselines for Comparison: The authors compare sharding against a standard Solo judge (one call for all criteria) and a Full-budget solo judge (one call given the same total token/tool budget as the sharded panel, often utilizing step-by-step instructions).
  • Defenses Against Adaptive Attacks: For sophisticated adversaries who adaptively persuade judges on a criterion-by-criterion basis, the authors layer debate-style opposition (assigning a skeptical challenger to argue why a criterion is unmet) on top of the sharded base.

Datasets

The authors evaluated their approach across diverse, expert-labeled domains requiring deep contextual understanding:

  • PaperBench (JudgeEval): Code-replication research submissions evaluated against dense hierarchical rubrics.
  • JudgmentBench: Expert-lawyer grading of legal contracts and work products against professional rubrics.
  • ROBoto2: Expert Cochrane risk-of-bias assessments of clinical trials.
  • SWE-Bench Pro & Oracle Cyber-Verification (ARVO): Software engineering and memory-safety verification tasks used as direct-reading controls.
  • FActScore: A structural control dataset consisting of easy, independent atomic fact checks.

Key Insights and Ideas

  • Compute Doesn't Cure Overload: Simply giving a single LLM call more tokens or forcing it to reason step-by-step does not fix the decision-load bottleneck. The issue is how attention is divided across multiple distinct outputs, not a lack of raw compute.
  • Weaker Judges Can Beat Stronger Ones: Remarkably, a weaker model (like Claude Sonnet) using a sharded approach can outperform a more capable model (like Claude Opus) evaluating holistically.
  • Context-Dependent Benefits: Sharding provides massive accuracy gains when the criteria require an integrated understanding of a large shared context (e.g., legal contracts or research papers). However, for independent tasks where text can be evaluated locally (like checking separate code patches in SWE-Bench), sharding offers no distinct advantage.

Key Results

  • Restoring Accuracy: On PaperBench, as the decision load scaled to 116 criteria, the holistic judge's agreement with human experts plummeted. Sharding completely arrested this decline, maintaining a high Cohen's kappa (~0.73 to 0.79) across all workloads.
Agreement with gold standard and errors caught across different decision loads on PaperBench.
  • Replication on Legal Work: On JudgmentBench, a single judge's agreement with expert lawyers dropped sharply past 12 criteria. At full load, the sharded judge significantly outperformed both the single judge and the full-budget single judge controls.
Figure 2: Second dataset with expert labels (JudgmentBench, expert-lawyer grading of legal work products, full source contract as context, judge Claude Opus 4.8). (a) A single judge’s agreement with the lawyer labels falls as the number of criteria per call grows, with the decline beginning near twenty criteria; decision load therefore limits accuracy even for the strongest model tier tested. (b) At full load a sharded judge exceeds both the single judge and the full-budget single judge, and the full-budget single judge does not close the gap. Error bars are task-clustered 95 percent intervals over the thirteen legal tasks. Sonnet-tier results are in Appendix B (Tables S9 and S10).
  • Thwarting Adversaries: In a best-of-8 presentation attack on PaperBench, an adversary successfully doubled the over-acceptance rate of unmet criteria against a holistic judge. Sharding consistently mitigated this vulnerability across all tested submissions.
Reduction in over-acceptance under a best-of-8 presentation attack across four submissions.
  • Defeating Adaptive Persuasion: Against an adversary that iteratively rewrote compliance memos to target specific criteria, plain sharding eventually faltered. However, adding a one-sided opposing advocate (debate) successfully restricted the cumulative attack surface from 0.55 down to 0.39.

Limitations and Future Work

While highly effective, sharding introduces a clear trade-off: token cost. Because the entire context (which can be up to 183,000 tokens in legal tasks) must be re-sent to every parallel shard call, API costs scale linearly with the number of shards. The authors note that intermediate capacity-sized grouping (e.g., batches of 32 criteria) offers a practical sweet spot that preserves accuracy while drastically reducing token overhead compared to maximal sharding (1 criterion per call). Future work will likely focus on leveraging platform-level context caching to minimize these redundant costs.

Summary updated

Abstract

Giving an LLM judge more compute does not necessarily make it check more requirements. When one call must return many verdicts, some decisions become weakly grounded in the evidence, even when that call receives the same token or tool budget as a panel of separate calls. Across expert-graded research replications, legal work, and clinical-trial assessments, agreement with experts falls as the number of verdicts per call grows. We identify sharding as the intervention that mitigates this failure in model-based oversight. Sharding partitions the requirements into smaller groups, assigns each group to a separate call, and aggregates the verdicts. Against a single call with the panel's full budget, sharding improves agreement while holding the model, evidence, total budget, and per-decision budget fixed. Overall, we find that a sharded weaker judge can outperform a more capable holistic judge and match that judge even when the latter receives the panel's full budget. Additionally, we find that sharding exhibits robustness against adversaries. A best-of-N adversary can hold the underlying work fixed, vary only its presentation, and increase an overloaded judge's acceptance of genuinely unmet criteria severalfold. Wherever sharding reduces baseline error, it removes this adversarial advantage, keeping over-acceptance low even as the adversary's search widens. Sharding does not address attacks that persuade the judge separately on each criterion rather than exploiting overload. In that setting, we find that debate-style opposition on top of sharding withstands such adaptive re-optimization.
BibTeX
@misc{akinwande2026sharding,
  author = {Victor Akinwande and J. Zico Kolter and Aran Nayebi},
  title = {{Sharding Prevents LLM Oversight Failures and Adversarial Exploitation}},
  year = {2026},
  eprint = {2608.06422},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.06422},
}
Figure