TL;DR
Benchmark Factory is an autonomous "meta-agent" system that creates high-quality, custom coding benchmarks directly from a company's internal production code. By using "quality anchors"—automated checks that ensure stronger models consistently outperform weaker ones—it eliminates the need for expensive manual labeling while producing benchmarks that reliably rank AI agents.
What Problem Does This Paper Address?
As software teams increasingly deploy AI coding agents, they face a "measurement gap." Public benchmarks like SWE-bench are useful for general capabilities, but they don't reflect the specific internal libraries, APIs, and coding standards of a private organization.
Creating custom benchmarks manually is a massive undertaking, requiring experts to curate tasks and verify solutions. Conversely, fully automated generation often results in "noisy" benchmarks that are either too easy, too hard, or simply broken, leading to misleading results. The authors seek to automate this curation process while maintaining the rigor of a human-validated suite.
Methods
The core of the paper is the Benchmark Factory loop, an agentic workflow that treats benchmark creation as an iterative engineering task:
- Research: An LLM meta-agent explores production data (landed code changes) and writes sourcing queries to find potential tasks.
- Launch: The system runs a "ladder" of models with known relative strengths (e.g., a "top-tier" model vs. a "bottom-tier" model) on these tasks.
- Analyze (Quality Anchors): Instead of human labels, the system uses label-free quality anchors. A benchmark is only accepted if the models' performance matches their known hierarchy (Capability Ordering) and the tasks fall within a useful range of difficulty (Difficulty Band).
- Hill-Climb (Repair): If a candidate benchmark fails the anchors, the meta-agent diagnoses the failure. It then "repairs" the benchmark by refining the sourcing query (e.g., looking for larger code changes), editing prompts, or adjusting the grading threshold.
- Holdout: To prevent the agent from "overfitting" the benchmark to specific model quirks, a portion of the tasks is kept in a holdout set that is never used to drive repairs.
Datasets
The researchers applied Benchmark Factory to five developer cohorts within Meta, spanning various programming languages and repository structures.
- Source Material: Thousands of "landed" (merged) production code changes.
- Models Used: The "ladder" included various versions of Claude (Opus 4.5 through 4.8) and GPT (5.4 and 5.5). Note: These version numbers likely represent internal iterations or aliases used for the study's relative ranking.
- Evaluation: The system was validated against a 5% sample (30 tasks) reviewed by human engineering experts.
Key Insights and Ideas
- Validation via Relative Strength: The most clever insight is that you don't need to know the "correct" answer for every task to know if a benchmark is valid. If a model known to be superior cannot beat a weaker model on your benchmark, the benchmark (or the grader) is likely the problem, not the models.
- Repairing Data vs. Tuning Scores: When a benchmark fails, the meta-agent is instructed to prioritize changing the sourcing query (finding better tasks) over simply lowering the passing grade. This ensures the final suite consists of meaningful engineering challenges.
- Negative Controls as a Safety Net: The authors tested "broken" setups (e.g., trivial tasks, shuffled labels, or graders that don't look at the code). The quality anchors successfully rejected these 100% of the time, proving the system isn't just "rubber-stamping" every task it finds.
Key Results
- Human Alignment: Expert raters gave the generated tasks a quality score of 2.3/3 and the automated grader verdicts a 2.6/3.
- Generalization (H1): Benchmarks built using one family of models (Claude) successfully ranked an entirely different family (GPT) in the correct order (GPT-5.5 > GPT-5.4, $p=0.004$).
- Sensitivity: The benchmarks were sensitive enough to detect performance gains from "high reasoning effort" settings in models, showing a 6.2 percentage point improvement that was consistent across all five cohorts.
- Efficiency: A full benchmark for a new team can be generated in just a few hours, involving roughly 480 to 2,400 agent runs, depending on how many repair rounds are needed.
Limitations and Future Work
- Under-specification: The primary failure mode identified by human experts was "under-specified tasks"—where the original commit message didn't provide enough context for an AI to solve the problem from scratch.
- LLM Grader Bias: While the system uses a "ladder" to mitigate bias, there is always a risk of "self-preference" (an LLM grader favoring its own model family).
- Scope Drift: Current graders sometimes struggle to distinguish between "useful extra work" and "unnecessary changes" (scope drift).
- Future Directions: The authors suggest future work should focus on automatically rewriting under-specified task descriptions and developing more nuanced graders that can better handle complex, multi-file changes.
Summary updated