TL;DR
While generalist LLM coding agents can successfully run iterative data curation loops out of the box, they suffer from an "execution-research gap"—preferring to tweak local heuristics rather than explore novel strategies. However, by applying structured method-adaptation scaffolding that forces agents to operationalize prior literature, they can autonomously discover advanced curation policies. Remarkably, these scaffolded agents can find a 10k-sample training set that outperforms strong human-designed baselines built on ten times more data.
What Problem Does This Paper Address?
Curating high-quality training data is one of the most critical yet labor-intensive phases of modern AI development. Because no single data-filtering technique works universally, human engineers spend countless hours in a manual loop: proposing a data policy (e.g., filtering, deduplication, or re-weighting), training a model, evaluating it against noisy benchmarks, and revising the policy based on the feedback.
This paper investigates whether generalist coding agents (like Claude Code or open-source alternatives via OpenHands) can fully automate this iterative data-curation loop. To study this systematically, the authors address a core gap in existing benchmarks: data-centric benchmarks (like DataComp) evaluate static datasets rather than the discovery process, while machine learning agent benchmarks typically optimize hyperparameters or training code rather than the underlying data distribution.
Methods
The authors introduce CURATION-BENCH, an agent-centric benchmark designed to isolate data curation as the sole independent variable. The environment fixes the model architecture, training recipe, and evaluation suite, while providing the agent with a standard terminal workspace.
The agent's workflow is formulated as an interactive policy-search loop:
- Inspect the candidate data pool and logs from previous runs.
- Propose and implement a data curation policy via Python scripts.
- Submit the curated dataset to an automated pipeline that enforces strict budget constraints and runs a contamination audit (to prevent data leakage from evaluation sets).
- Observe benchmark feedback and iterate.
To improve the agent's research quality, the authors introduce a Scaffolding Ladder consisting of four conditions:
- Open-Prompt: Minimal instructions detailing only the environment interface.
- Light I (Data Strategies): Provides a list of broad curation concepts (e.g., deduplication, semantic diversity).
- Light II (Research Papers): Grants access to paper-derived "skill cards" summarizing data curation literature.
- Heavy I (Self-Research): Imposes a binding protocol requiring a written observation, hypothesis, and minimal change log before execution.
- Heavy II (Adapt Papers): Forces the agent to explicitly cite, instantiate, and adapt a method from the skill cards for every non-baseline iteration.
Datasets
The benchmark evaluates agents across both multimodal instruction tuning and vision-language pretraining:
- LLaVA-665K: A heterogeneous mixture of 665,298 samples (including COCO, GQA, OCR-VQA, TextVQA, and ShareGPT). Agents were tasked with selecting a strict 10k, 20k, or 50k budget.
- Vision-Flan: A human-annotated instruction-tuning dataset spanning 191 task types.
- DataComp Small: A 12.8M image-text pair filtering track dataset used to evaluate CLIP-style pretraining.
- Evaluation: Models were evaluated using VLMEvalKit across 8 diverse benchmarks (including MMBench, MMMU, MathVista, and OCRBench).
Key Insights and Ideas
- The Execution-Research Gap: Out-of-the-box agents are excellent executors but narrow researchers. They reliably write scripts and parse logs, but fall into "vibe optimization"—making shallow, local adjustments (like changing random seeds or slightly altering source ratios) rather than shifting to entirely new policy families.
- Awareness vs. Procedural Constraints: Light scaffolds expand the agent's vocabulary in its written plans but fail to change its actual code execution. Breakthroughs require heavy, binding procedural constraints (like the Adapt Papers scaffold) that force the agent to pay the implementation cost of complex methods.
- Autonomous Method Composition: When properly scaffolded, the agent autonomously composed a sophisticated hybrid policy without human intervention. It combined an EL2N-style top-loss proxy (to select highly informative samples) with an assistant-loss noise filter (to prune incoherent or corrupted text responses).
- Curation as a Compute Axis: When agent iteration budgets were scaled from 10 to 50 iterations, downstream performance continued to improve without a clear plateau. This suggests that searching over existing data is a highly viable way to scale compute when raw data is scarce or expensive.
Key Results
- Matching Human Baselines: Under simple open-prompting, generalist agents matched or exceeded strong published human-designed data selection baselines (such as ICONS and ARDS) within just 10 iterations.
- 10x Data Efficiency: Under the Heavy II (Adapt Papers) scaffold, Claude Code achieved a peak average score of 34.9 on the LLaVA fine-tuning task. This outperformed the 100k-sample ARDS baseline while using only 10k samples—effectively achieving a 10x data efficiency gain.
- Generalization: The framework successfully extended beyond subset selection to a data-rewriting task, where the agent used an external multimodal LLM tool to identify model failure modes and target specific data slices for rewriting.
Limitations and Future Work
The authors note that their current evaluations focus heavily on vision-language instruction tuning and a small-scale CLIP pretraining setup. It remains an open question whether these agent behaviors and scaffolding benefits translate identically to code generation, mathematics, or massive-scale language model pretraining.
Additionally, the heavy scaffolds combine multiple elements (structured logging, method grounding, and category rotation), meaning the exact causal impact of each individual constraint is not yet fully isolated. Future work could explore more granular ablation of these constraints and expand the agent's toolkit to include more complex data actions like synthetic data generation loops.
Summary updated