Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

Haomin Qi, Xingliang Wang, Xuanqi Gao, Baihui Sang, Xin Zhang, Minghua Ma, Pengfei Gao, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi ZhangarXiv 2026 · arXiv

TL;DR

Change2Task is a novel framework that automatically transforms historical pull requests (PRs) into verified, executable coding tasks on modern, healthy versions of a repository. By bridging the gap between historical developer intent and evolved codebases, it provides a scalable, high-fidelity data source for training and evaluating coding agents while reducing environment overhead by over 50%.

What Problem Does This Paper Address?

Coding agents—AI systems designed to fix bugs, add features, and write tests—require a massive supply of "executable substrate." Each training or evaluation task needs a specific repository state, a clear specification, and a reliable verifier (tests).

Currently, researchers face a trilemma:

  1. Historical Benchmarks: (e.g., SWE-bench) are grounded in real intent but are "frozen" at old, often unmaintained repository revisions.
  2. Synthetic Data: Can be generated at scale but often lacks the nuance and complex intent of real-world developer work.
  3. Infrastructure Cost: Building and storing unique Docker environments for thousands of individual tasks is prohibitively expensive (costing hundreds of thousands of dollars for large-scale datasets).

Change2Task addresses these by "porting" historical changes forward to modern codebases, allowing multiple tasks to be hosted on a single, well-maintained environment.

Methods

The Change2Task workflow (illustrated in Figure 2) operates in four distinct stages:

  1. Deriving Task Evidence: The system extracts the "maintenance intent" from a merged PR, including the implementation patch, the description, and the specific tests (checks) that verify the change.
  2. Establishing the Modern Task Base: It identifies a healthy, recent descendant revision of the repository that still hosts the relevant code logic.
  3. Constructing the Task State: This is the core innovation. To create a "task state" (e.g., a version of the code where a bug is present or a feature is missing), the system uses three levels of adaptive construction (see Figure 3):
  4. Level 1: Patch Reversal: A simple git-reverse of the original patch.
  5. Level 2: Code Mapping: Uses symbol alignment to map historical changes to modern files, even if the code has been refactored or moved.
  6. Level 3: Agent Reconstruction: An LLM-based agent receives the PR evidence and modern context to "un-implement" the change, ensuring the resulting state is semantically equivalent to the historical "before" state.
  7. Lifecycle & Scope Validation: Every task must pass a strict "pass-fail-pass" lifecycle check. The target tests must pass on the healthy base, fail on the constructed task state, and pass again after a restoration patch is applied.
Figure 2: Overview of the Change2Task workflow.

Datasets

The authors evaluated Change2Task using 1,130 construction-eligible changes sourced from 12 public benchmark collections, including SWE-bench, FEA-Bench, and PyMigBench.

The process resulted in a finalized corpus of 900 paired tasks across five families:

  • Bug Fix (500 cases)
  • Feature Addition (100 cases)
  • Test Generation (100 cases)
  • API Migration (100 cases)
  • Security Repair (100 cases)

Key Insights and Ideas

  • Semantic Subtraction: Instead of just rolling back a repository, Level 3 construction performs a "scoped semantic subtraction." It removes a specific feature or bug fix while preserving all other improvements and refactors made to the repository in the intervening years.
  • Environment Multi-tenancy: By hosting multiple task variants on one healthy base, the system drastically reduces the "image sprawl" common in coding agent benchmarks.
  • Fidelity over Textual Match: The authors emphasize that the modern task doesn't need to look exactly like the old code; it just needs to represent the same behavioral obligation.

Key Results

  • High Success Rate: Change2Task achieved a 79.6% verified task construction success rate across all families.
  • Superior Recovery: On a matched set of Bug Fix candidates, it recovered 29.2% more verified tasks than the previous state-of-the-art baseline (SWE-smith PR Mirror).
  • Agent Consistency: When evaluated by coding agents (like GPT-4o or Claude 3.5 Sonnet), the reconstructed modern tasks showed a 98.0% outcome agreement with their historical counterparts.
  • Efficiency Gains: Reusing modern bases reduced environment setup time by 58.4% and storage requirements by 71.2%. The total pipeline expenditure was reduced by 10.8%.

The distribution of why certain tasks failed construction is detailed in the terminal rejection reasons heatmap (Figure 6), showing that lifecycle validation and environment preflight are the primary bottlenecks.

Terminal rejection reasons heatmap

Limitations and Future Work

  • Lineage Dependence: The system requires a traceable PR and a descendant revision that still hosts the relevant behavior. If a feature is completely deprecated and removed in later versions, the task cannot be reconstructed.
  • Language Scope: The current study focused on Python and Java. Future work aims to extend this to more languages and complex build systems.
  • Oracle Evolution: As APIs evolve, the "target checks" (tests) themselves may need adaptation to remain valid on modern versions, a process the authors plan to further automate.

Summary updated

Abstract

Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification. To expand this supply, we present Change2Task, a system grounded in repository history that converts merged pull requests into verified tasks on healthy modern revisions of the same repository. It aligns historical evidence with evolved code, reconstructs task states through Patch Reversal, Code Mapping, or Agent Reconstruction, and validates the lifecycle from a healthy base to a task state and a restored state. By deriving multiple tasks grounded in developer evidence from maintained environments, Change2Task provides executable data for coding agent training and evaluation while reducing repeated environment setup, storage, and task construction effort. We evaluate the system through five common and widely adopted coding agent task families: Bug Fix, Feature Addition, Test Generation, Application Programming Interface Migration, and Security Repair. Starting from 1,130 source changes eligible for construction, Change2Task achieves 79.6% verified task construction success across these task families. On a matched candidate set, it recovers 29.2% more verified tasks than a construction baseline based on pull requests. Historical and reconstructed cases achieve up to 98.0% matched outcome agreement under agent evaluation, while reuse of modern bases reduces measured expenditure across the complete pipeline by 10.8%.
BibTeX
@misc{qi2026change,
  author = {Haomin Qi and Xingliang Wang and Xuanqi Gao and Baihui Sang and Xin Zhang and Minghua Ma and Pengfei Gao and Yu Kang and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang and Qi Zhang},
  title = {{Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments}},
  year = {2026},
  eprint = {2607.28591},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.28591},
}
Figure