Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

Yuanzhe Liu, Bastian Lipka, Avraham Shinnar, Jatin Ganhotra, Yao Ma, Martin HirzelarXiv 2026 · arXiv

Evaluating AI coding agents is a high-stakes game of "telephone." If a benchmark task is too vague (under-specification), the agent has to guess internal naming conventions or interface choices that it couldn't possibly know, leading to unfair failures. If the task is too detailed (over-specification), the benchmark "spoils" the challenge by leaking the solution, making the agent look smarter than it actually is.

A new paper from researchers at RPI and IBM Research, titled "Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks," proposes a principled way to thread this needle.

TL;DR

The authors introduce a framework to categorize "hints" in coding benchmarks into Selection (which code elements to show) and Projection (what details to reveal about them). They propose DefUse, a method that reveals only the minimal interface needed for tests to pass, effectively reducing guesswork without giving away the solution "spoilers."

What Problem Does This Paper Address?

Repository-level coding benchmarks (like SWE-bench) ask agents to fix bugs or add features in real-world codebases. However, these benchmarks often suffer from a "misalignment" between the task description and the hidden tests.

Imagine an agent writes a perfectly functional feature but names a new function fetch_user_data(). If the hidden test suite specifically looks for a function named get_user_info(), the agent fails. This is guesswork. To fix this, recent benchmarks have started adding "hints" derived from the ground-truth solution.

The danger here is over-specification. If a hint says, "Create a function called get_user_info that uses a trie data structure to optimize lookups," it has leaked the core engineering decision (the "spoiler"). The benchmark is no longer measuring the agent's ability to design a solution, only its ability to follow instructions.

Methods

The authors decompose hint construction into two design decisions:

  1. Identifier Selection: Which program entities (files, functions, classes) should we tell the agent about?
  2. Identifier Projection: What specific attributes (name, signature, docstring, implementation details) should we reveal for each selected entity?

The DefUse Policy

The authors propose DefUse for selection. As shown in the pipeline below, it identifies the "handshake" between the new code and the new tests.

Figure 2: DefUse hints generation pipeline.

Specifically, it selects identifiers that are:

  • Defined in the solution and Used in the tests.
  • Defined in the tests and Used in the solution.

The Declaration Policy

For projection, they propose Declaration. This reveals only structural information: the name, kind (function/class), file path, and signature. It deliberately omits "semantic" information like docstrings or behavioral requirements that often contain implementation spoilers.

The authors also developed an LLM-based "Hint Leakage Judge" to quantify three metrics:

  • Necessity ($N$): How unrecoverable is this info without the hint?
  • Disclosure ($D$): How much of the solution's internal logic is leaked?
  • Guidance ($G$): How much does it direct the agent's specific actions?

Datasets

The study evaluates these policies across five major repository-level benchmarks:

  • Python: SWE-bench Pro, FEA-bench, Doc2Feat-bench, and SWE-rebench.
  • Java: Multi-SWE-bench (MSB Java).

Key Insights and Ideas

  • The "Handshake" is Enough: DefUse selection provides the minimal set of identifiers needed to resolve name-mismatch artifacts. Any identifier not in the DefUse set is an internal design choice that the agent should be able to make independently.
  • Structural vs. Semantic: Structural attributes (like function signatures) provide high value with almost zero leakage. Semantic attributes (like "Requirements" in SWE-bench Pro) significantly boost solve rates but do so by providing "procedural guidance"—essentially telling the agent how to do its job.
  • Solve Rates are Deceptive: A higher solve rate doesn't always mean a better benchmark. If a hint scheme increases the solve rate by providing spoilers, it reduces the "shelf-life" of the benchmark as models improve.

Key Results

The authors found that existing benchmarks vary wildly in how much they disclose. As seen in the table below, DefUse is significantly more parsimonious than existing methods like those used in SWE-bench Pro.

Table 2: Number of identifiers disclosed per selection method, counted on the same instances as the selection tables under the Declaration projection. Total is the number of identifiers over all instances and Mean the number per instance. DefUse is empty on the instances whose tests reference no new identifier: it is non-empty on 20/31, 44/142, 66/131, 9/26 and 29/40 instances, with 3.2, 1.8, 2.4, 2.0 and 6.7 identifiers per non-empty instance. The other selections are non-empty on every instance.

In ablation studies (Table 5), moving from "Empty" hints to "Declaration" hints (using the DefUse selection) nearly tripled the solve rate for DeepSeek V4 Flash on SWE-bench Pro (from 10.3% to 29.4%).

Table 5: Hint attribute ablation with DeepSeek V4 Flash on SWE-bench Pro (68 instances) un- der DefUse selection.

Crucially, while adding "All Hints" (including semantic spoilers) boosted the rate to 72.1%, the "Leakage Judge" confirmed this gain was bought with high disclosure and prescriptive guidance, effectively "solving" the problem for the model.

Limitations and Future Work

The authors acknowledge several limitations:

  • Budgetary Constraints: Due to the high cost of running these agents, they limited experiments to a single "trajectory" (one attempt) per instance, which can be sensitive to stochastic variation.
  • LLM Judge Subjectivity: While the LLM judge showed high agreement on structural attributes, it was less consistent on semantic descriptions, highlighting the difficulty of quantifying "meaning."
  • Generalizability: The study focused on a subset of popular benchmarks and a specific agent harness (mini-SWE-agent).

Future work could involve refining the "Hint Value Margin" to help benchmark creators automatically audit their tasks for spoilers before release.

Summary updated

Abstract

Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking. Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure. Since future-looking benchmarks should be difficult, over-specification reduces a benchmark's shelflife of community adoption. Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification. For instance, SWE-bench Pro, FEA-bench, Doc2Feat-bench, and Multi-SWE-bench all tackle under-specification by augmenting task statements with hints derived from ground truth. We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork. Based on these results, we recommend techniques for deriving parsimonious hints. We will release our hints so the community can obtain more reliable signals from popular benchmarks.
BibTeX
@misc{liu2026between,
  author = {Yuanzhe Liu and Bastian Lipka and Avraham Shinnar and Jatin Ganhotra and Yao Ma and Martin Hirzel},
  title = {{Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks}},
  year = {2026},
  eprint = {2610.22760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.22760},
}
Figure