CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents

Silin Chen, Yuling Shi, Heng Lian, Zihan Zhang, Xiaodong Gu, Haotian Lin, Zhensu SunarXiv 2026 · arXiv

TL;DR

CrossSWE is a new benchmark designed to evaluate coding agents on tasks that require simultaneous, coordinated modifications across multiple interdependent repositories. It reveals a significant performance gap in state-of-the-art models: even the strongest model, GPT-6.1 Sol, achieves only a 58.48% success rate, primarily due to the difficulty of maintaining consistency across repository boundaries.

What Problem Does This Paper Address?

Current evaluations for AI coding agents, such as the popular SWE-bench, have largely focused on "single-repository" tasks. While these benchmarks have grown in complexity—featuring longer horizons and larger codebases—they remain bounded by the walls of a single project.

In the real world, software is rarely an island. A change to a shared API in a library (Repo A) often necessitates immediate, corresponding updates in a downstream application (Repo B). This "joint development" introduces unique challenges: agents must reason about cross-repo dependencies, propagate changes correctly, and ensure interface consistency. Existing benchmarks that include multiple repositories often use the secondary repo merely as "context" (read-only), whereas CrossSWE requires the agent to actively patch multiple repositories to pass a single test suite.

Figure 1: Comparison of single-repository SWE tasks and cross-repository SWE tasks.

Methods

The authors developed a rigorous four-stage pipeline to construct CrossSWE:

  1. Candidate Collection: Mining GitHub for merged pull requests and issues that explicitly reference changes across multiple repositories (e.g., "API evolution" or "dependency migration").
  2. Human Screening: Manually verifying that the candidates represent genuine joint development rather than just documentation updates.
  3. Joint-Necessity Verification: This is the benchmark's "secret sauce." For every task involving repositories A and B, the authors verify that applying only the patch for A or only the patch for B results in a test failure. Only when both patches are applied together must the tests pass. This ensures the task truly requires coordinated cross-repo reasoning.
  4. Problem Statement Construction: Converting fragmented developer discussions into self-contained, natural-language task specifications.
Figure 4: Overview of the CrossSWE benchmark construction framework.

Datasets

The paper introduces CrossSWE, which consists of:

  • 171 real-world instances spanning 165 distinct repositories.
  • 20 technical domains and 13 programming languages.
  • A mix of repository counts: 165 instances involve two repos, while others involve up to five.

Compared to existing benchmarks like SWE-bench Pro, CrossSWE tasks are significantly more "dense" in terms of modifications. Over 56% of instances modify more than ten files, and two-thirds change over 200 lines of code across the participating repositories.

Figure 3: Distribution of reference-patch sizes across CrossSWE, SWE-bench Pro, and SWE-Bench ProMax.

Key Insights and Ideas

  • The "Consistency Killer": The most prevalent failure mode (44.2% for GPT models) was cross-repository inconsistency. Agents often fixed one side of the interface but failed to update the other to match, leading to compilation errors or protocol mismatches.
  • Joint vs. Independent Difficulty: The authors ran an experiment where they took the same cross-repo tasks but modified them so only one side needed a fix (while keeping the other repo as context). Performance jumped significantly (e.g., GPT-6.1 Sol went from 58.48% to 64.55%). This proves that the act of coordinated editing is fundamentally harder than just reasoning about multiple repos.
  • Exploration Strategies Matter: Closed-source models like GPT-6.1 Sol and Claude 3.5 Sonnet tend to be "decisive," making larger, concentrated edits early. In contrast, open-source models like DeepSeek V4.1 Flash spend much more time exploring repository structures and alternating between small edits and tests. While DeepSeek was more "active," it didn't necessarily translate to higher success rates.

Key Results

The evaluation of five frontier LLMs shows that cross-repository development remains a "frontier" challenge:

Model Overall Pass@1 Avg. Steps Avg. Cost ($)
GPT-6.1 Sol 58.48% 32.9 $0.3496
DeepSeek V4.1 Flash 52.63% 152.7 $0.2881
Claude 3.5 Sonnet 50.29% 29.2 $0.5217
Qwen3.8-Flash-Next 45.03% 182.1 $0.2939
GLM-5.3 Flash 43.86% 141.7 $0.3411

Notably, while GPT-6.1 Sol is the most effective, it is also highly efficient, requiring roughly 5x fewer interaction steps than DeepSeek to reach a solution.

Limitations and Future Work

  • Scale of Interdependence: The vast majority of tasks (165/171) involve only two repositories. Real-world microservice architectures or massive monorepos might involve dozens of interdependent units.
  • Repository Discovery: In CrossSWE, the agent is given the relevant repositories. In a real-world scenario, an agent might first need to discover which repositories in a massive organization need to be changed—a task the authors suggest for future benchmark versions.
  • Environment Complexity: The current setup uses a unified Docker environment. Future work could explore more complex network-isolated environments to better simulate cloud-native development.

Summary updated

Abstract

As coding agents achieve increasingly high success rates on software engineering benchmarks, recent efforts have sought to increase task complexity through longer horizons and more extensive code modifications. However, real-world software development often spans multiple interdependent repositories, introducing additional challenges in dependency reasoning, change propagation, and interface consistency. Rather than further increasing complexity within individual repositories, we introduce CrossSWE, a benchmark for cross-repository joint development comprising 171 real-world instances spanning 165 repositories, 20 technical domains, and 13 programming languages. CrossSWE verifies the necessity of coordinated changes by testing whether partial repository modifications are insufficient to achieve the intended behavior. Our evaluation of five LLMs reveals substantial challenges, with the strongest model, GPT-6.1 Sol, achieving only 58.48% Pass@1. Furthermore, cross-repository inconsistencies emerge as the most prevalent failure mode (44.2%), highlighting the difficulty of maintaining consistency across interdependent repositories.
BibTeX
@misc{chen2026crossswe,
  author = {Silin Chen and Yuling Shi and Heng Lian and Zihan Zhang and Xiaodong Gu and Haotian Lin and Zhensu Sun},
  title = {{CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents}},
  year = {2026},
  eprint = {2611.01484},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01484},
}
Figure