TL;DR
Change2Task is a novel framework that automatically transforms historical pull requests (PRs) into verified, executable coding tasks on modern, healthy versions of a repository. By bridging the gap between historical developer intent and evolved codebases, it provides a scalable, high-fidelity data source for training and evaluating coding agents while reducing environment overhead by over 50%.
What Problem Does This Paper Address?
Coding agents—AI systems designed to fix bugs, add features, and write tests—require a massive supply of "executable substrate." Each training or evaluation task needs a specific repository state, a clear specification, and a reliable verifier (tests).
Currently, researchers face a trilemma:
- Historical Benchmarks: (e.g., SWE-bench) are grounded in real intent but are "frozen" at old, often unmaintained repository revisions.
- Synthetic Data: Can be generated at scale but often lacks the nuance and complex intent of real-world developer work.
- Infrastructure Cost: Building and storing unique Docker environments for thousands of individual tasks is prohibitively expensive (costing hundreds of thousands of dollars for large-scale datasets).
Change2Task addresses these by "porting" historical changes forward to modern codebases, allowing multiple tasks to be hosted on a single, well-maintained environment.
Methods
The Change2Task workflow (illustrated in Figure 2) operates in four distinct stages:
- Deriving Task Evidence: The system extracts the "maintenance intent" from a merged PR, including the implementation patch, the description, and the specific tests (checks) that verify the change.
- Establishing the Modern Task Base: It identifies a healthy, recent descendant revision of the repository that still hosts the relevant code logic.
- Constructing the Task State: This is the core innovation. To create a "task state" (e.g., a version of the code where a bug is present or a feature is missing), the system uses three levels of adaptive construction (see Figure 3):
- Level 1: Patch Reversal: A simple git-reverse of the original patch.
- Level 2: Code Mapping: Uses symbol alignment to map historical changes to modern files, even if the code has been refactored or moved.
- Level 3: Agent Reconstruction: An LLM-based agent receives the PR evidence and modern context to "un-implement" the change, ensuring the resulting state is semantically equivalent to the historical "before" state.
- Lifecycle & Scope Validation: Every task must pass a strict "pass-fail-pass" lifecycle check. The target tests must pass on the healthy base, fail on the constructed task state, and pass again after a restoration patch is applied.
Datasets
The authors evaluated Change2Task using 1,130 construction-eligible changes sourced from 12 public benchmark collections, including SWE-bench, FEA-Bench, and PyMigBench.
The process resulted in a finalized corpus of 900 paired tasks across five families:
- Bug Fix (500 cases)
- Feature Addition (100 cases)
- Test Generation (100 cases)
- API Migration (100 cases)
- Security Repair (100 cases)
Key Insights and Ideas
- Semantic Subtraction: Instead of just rolling back a repository, Level 3 construction performs a "scoped semantic subtraction." It removes a specific feature or bug fix while preserving all other improvements and refactors made to the repository in the intervening years.
- Environment Multi-tenancy: By hosting multiple task variants on one healthy base, the system drastically reduces the "image sprawl" common in coding agent benchmarks.
- Fidelity over Textual Match: The authors emphasize that the modern task doesn't need to look exactly like the old code; it just needs to represent the same behavioral obligation.
Key Results
- High Success Rate: Change2Task achieved a 79.6% verified task construction success rate across all families.
- Superior Recovery: On a matched set of Bug Fix candidates, it recovered 29.2% more verified tasks than the previous state-of-the-art baseline (SWE-smith PR Mirror).
- Agent Consistency: When evaluated by coding agents (like GPT-4o or Claude 3.5 Sonnet), the reconstructed modern tasks showed a 98.0% outcome agreement with their historical counterparts.
- Efficiency Gains: Reusing modern bases reduced environment setup time by 58.4% and storage requirements by 71.2%. The total pipeline expenditure was reduced by 10.8%.
The distribution of why certain tasks failed construction is detailed in the terminal rejection reasons heatmap (Figure 6), showing that lifecycle validation and environment preflight are the primary bottlenecks.
Limitations and Future Work
- Lineage Dependence: The system requires a traceable PR and a descendant revision that still hosts the relevant behavior. If a feature is completely deprecated and removed in later versions, the task cannot be reconstructed.
- Language Scope: The current study focused on Python and Java. Future work aims to extend this to more languages and complex build systems.
- Oracle Evolution: As APIs evolve, the "target checks" (tests) themselves may need adaptation to remain valid on modern versions, a process the authors plan to further automate.
Summary updated