TL;DR
CrossSWE is a new benchmark designed to evaluate coding agents on tasks that require simultaneous, coordinated modifications across multiple interdependent repositories. It reveals a significant performance gap in state-of-the-art models: even the strongest model, GPT-6.1 Sol, achieves only a 58.48% success rate, primarily due to the difficulty of maintaining consistency across repository boundaries.
What Problem Does This Paper Address?
Current evaluations for AI coding agents, such as the popular SWE-bench, have largely focused on "single-repository" tasks. While these benchmarks have grown in complexity—featuring longer horizons and larger codebases—they remain bounded by the walls of a single project.
In the real world, software is rarely an island. A change to a shared API in a library (Repo A) often necessitates immediate, corresponding updates in a downstream application (Repo B). This "joint development" introduces unique challenges: agents must reason about cross-repo dependencies, propagate changes correctly, and ensure interface consistency. Existing benchmarks that include multiple repositories often use the secondary repo merely as "context" (read-only), whereas CrossSWE requires the agent to actively patch multiple repositories to pass a single test suite.
Methods
The authors developed a rigorous four-stage pipeline to construct CrossSWE:
- Candidate Collection: Mining GitHub for merged pull requests and issues that explicitly reference changes across multiple repositories (e.g., "API evolution" or "dependency migration").
- Human Screening: Manually verifying that the candidates represent genuine joint development rather than just documentation updates.
- Joint-Necessity Verification: This is the benchmark's "secret sauce." For every task involving repositories A and B, the authors verify that applying only the patch for A or only the patch for B results in a test failure. Only when both patches are applied together must the tests pass. This ensures the task truly requires coordinated cross-repo reasoning.
- Problem Statement Construction: Converting fragmented developer discussions into self-contained, natural-language task specifications.
Datasets
The paper introduces CrossSWE, which consists of:
- 171 real-world instances spanning 165 distinct repositories.
- 20 technical domains and 13 programming languages.
- A mix of repository counts: 165 instances involve two repos, while others involve up to five.
Compared to existing benchmarks like SWE-bench Pro, CrossSWE tasks are significantly more "dense" in terms of modifications. Over 56% of instances modify more than ten files, and two-thirds change over 200 lines of code across the participating repositories.
Key Insights and Ideas
- The "Consistency Killer": The most prevalent failure mode (44.2% for GPT models) was cross-repository inconsistency. Agents often fixed one side of the interface but failed to update the other to match, leading to compilation errors or protocol mismatches.
- Joint vs. Independent Difficulty: The authors ran an experiment where they took the same cross-repo tasks but modified them so only one side needed a fix (while keeping the other repo as context). Performance jumped significantly (e.g., GPT-6.1 Sol went from 58.48% to 64.55%). This proves that the act of coordinated editing is fundamentally harder than just reasoning about multiple repos.
- Exploration Strategies Matter: Closed-source models like GPT-6.1 Sol and Claude 3.5 Sonnet tend to be "decisive," making larger, concentrated edits early. In contrast, open-source models like DeepSeek V4.1 Flash spend much more time exploring repository structures and alternating between small edits and tests. While DeepSeek was more "active," it didn't necessarily translate to higher success rates.
Key Results
The evaluation of five frontier LLMs shows that cross-repository development remains a "frontier" challenge:
| Model | Overall Pass@1 | Avg. Steps | Avg. Cost ($) |
|---|---|---|---|
| GPT-6.1 Sol | 58.48% | 32.9 | $0.3496 |
| DeepSeek V4.1 Flash | 52.63% | 152.7 | $0.2881 |
| Claude 3.5 Sonnet | 50.29% | 29.2 | $0.5217 |
| Qwen3.8-Flash-Next | 45.03% | 182.1 | $0.2939 |
| GLM-5.3 Flash | 43.86% | 141.7 | $0.3411 |
Notably, while GPT-6.1 Sol is the most effective, it is also highly efficient, requiring roughly 5x fewer interaction steps than DeepSeek to reach a solution.
Limitations and Future Work
- Scale of Interdependence: The vast majority of tasks (165/171) involve only two repositories. Real-world microservice architectures or massive monorepos might involve dozens of interdependent units.
- Repository Discovery: In CrossSWE, the agent is given the relevant repositories. In a real-world scenario, an agent might first need to discover which repositories in a massive organization need to be changed—a task the authors suggest for future benchmark versions.
- Environment Complexity: The current setup uses a unified Docker environment. Future work could explore more complex network-isolated environments to better simulate cloud-native development.
Summary updated