Evaluating AI coding agents is a high-stakes game of "telephone." If a benchmark task is too vague (under-specification), the agent has to guess internal naming conventions or interface choices that it couldn't possibly know, leading to unfair failures. If the task is too detailed (over-specification), the benchmark "spoils" the challenge by leaking the solution, making the agent look smarter than it actually is.
A new paper from researchers at RPI and IBM Research, titled "Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks," proposes a principled way to thread this needle.
TL;DR
The authors introduce a framework to categorize "hints" in coding benchmarks into Selection (which code elements to show) and Projection (what details to reveal about them). They propose DefUse, a method that reveals only the minimal interface needed for tests to pass, effectively reducing guesswork without giving away the solution "spoilers."
What Problem Does This Paper Address?
Repository-level coding benchmarks (like SWE-bench) ask agents to fix bugs or add features in real-world codebases. However, these benchmarks often suffer from a "misalignment" between the task description and the hidden tests.
Imagine an agent writes a perfectly functional feature but names a new function fetch_user_data(). If the hidden test suite specifically looks for a function named get_user_info(), the agent fails. This is guesswork. To fix this, recent benchmarks have started adding "hints" derived from the ground-truth solution.
The danger here is over-specification. If a hint says, "Create a function called get_user_info that uses a trie data structure to optimize lookups," it has leaked the core engineering decision (the "spoiler"). The benchmark is no longer measuring the agent's ability to design a solution, only its ability to follow instructions.
Methods
The authors decompose hint construction into two design decisions:
- Identifier Selection: Which program entities (files, functions, classes) should we tell the agent about?
- Identifier Projection: What specific attributes (name, signature, docstring, implementation details) should we reveal for each selected entity?
The DefUse Policy
The authors propose DefUse for selection. As shown in the pipeline below, it identifies the "handshake" between the new code and the new tests.
Specifically, it selects identifiers that are:
- Defined in the solution and Used in the tests.
- Defined in the tests and Used in the solution.
The Declaration Policy
For projection, they propose Declaration. This reveals only structural information: the name, kind (function/class), file path, and signature. It deliberately omits "semantic" information like docstrings or behavioral requirements that often contain implementation spoilers.
The authors also developed an LLM-based "Hint Leakage Judge" to quantify three metrics:
- Necessity ($N$): How unrecoverable is this info without the hint?
- Disclosure ($D$): How much of the solution's internal logic is leaked?
- Guidance ($G$): How much does it direct the agent's specific actions?
Datasets
The study evaluates these policies across five major repository-level benchmarks:
- Python: SWE-bench Pro, FEA-bench, Doc2Feat-bench, and SWE-rebench.
- Java: Multi-SWE-bench (MSB Java).
Key Insights and Ideas
- The "Handshake" is Enough: DefUse selection provides the minimal set of identifiers needed to resolve name-mismatch artifacts. Any identifier not in the DefUse set is an internal design choice that the agent should be able to make independently.
- Structural vs. Semantic: Structural attributes (like function signatures) provide high value with almost zero leakage. Semantic attributes (like "Requirements" in SWE-bench Pro) significantly boost solve rates but do so by providing "procedural guidance"—essentially telling the agent how to do its job.
- Solve Rates are Deceptive: A higher solve rate doesn't always mean a better benchmark. If a hint scheme increases the solve rate by providing spoilers, it reduces the "shelf-life" of the benchmark as models improve.
Key Results
The authors found that existing benchmarks vary wildly in how much they disclose. As seen in the table below, DefUse is significantly more parsimonious than existing methods like those used in SWE-bench Pro.
In ablation studies (Table 5), moving from "Empty" hints to "Declaration" hints (using the DefUse selection) nearly tripled the solve rate for DeepSeek V4 Flash on SWE-bench Pro (from 10.3% to 29.4%).
Crucially, while adding "All Hints" (including semantic spoilers) boosted the rate to 72.1%, the "Leakage Judge" confirmed this gain was bought with high disclosure and prescriptive guidance, effectively "solving" the problem for the model.
Limitations and Future Work
The authors acknowledge several limitations:
- Budgetary Constraints: Due to the high cost of running these agents, they limited experiments to a single "trajectory" (one attempt) per instance, which can be sensitive to stochastic variation.
- LLM Judge Subjectivity: While the LLM judge showed high agreement on structural attributes, it was less consistent on semantic descriptions, highlighting the difficulty of quantifying "meaning."
- Generalizability: The study focused on a subset of popular benchmarks and a specific agent harness (mini-SWE-agent).
Future work could involve refining the "Hint Value Margin" to help benchmark creators automatically audit their tasks for spoilers before release.
Summary updated