SWE-QA: Can Language Models Answer Repository-level Code Questions?

Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, Xiaodong Gu

Abstract

Understanding and reasoning about entire software repositories is an essential capability for intelligent software engineering tools. While existing benchmarks such as CoSQA and CodeQA have advanced the field, they predominantly focus on small, self-contained code snippets. These setups fail to capture the complexity of real-world repositories, where effective understanding and reasoning often require navigating multiple files, understanding software architecture, and grounding answers in long-range code dependencies. In this paper, we present SWE-QA, a repository-level code question answering (QA) benchmark designed to facilitate research on automated QA systems in realistic code environments. SWE-QA involves 576 high-quality question-answer pairs spanning diverse categories, including intention understanding, cross-file reasoning, and multi-hop dependency analysis. To construct SWE-QA, we first crawled 77,100 GitHub issues from 11 popular repositories. Based on an analysis of naturally occurring developer questions extracted from these issues, we developed a two-level taxonomy of repository-level questions and constructed a set of seed questions for each category. For each category, we manually curated and validated questions and collected their corresponding answers. As a prototype application, we further develop SWE-QA-Agent, an agentic framework in which LLM agents reason and act to find answers automatically. We evaluate six advanced LLMs on SWE-QA under various context augmentation strategies. Experimental results highlight the promise of LLMs, particularly our SWE-QA-Agent framework, in addressing repository-level QA, while also revealing open challenges and pointing to future research directions.

Introduction

Most existing code benchmarks focus on isolated functions or small snippets. However, real-world software engineering happens at the repository level. Developers rarely ask questions about a single line of code in a vacuum; they ask about how a feature is implemented across multiple files, why specific architectural choices were made, or how data payloads are modified as they traverse the system.

Current Large Language Models (LLMs) often struggle with these "long-range" dependencies. There is a significant gap between being a good "auto-complete" tool and being a truly "repository-aware" assistant. SWE-QA aims to quantify this gap and provide a framework for models to bridge it by testing their ability to navigate complex, multi-file codebases.

Methodology

The researchers contributed both a comprehensive benchmark and a novel agentic system designed for deep codebase reasoning.

Benchmark Construction

The construction of SWE-QA followed a rigorous four-stage pipeline. First, the team analyzed over 77,000 GitHub issues to identify naturally occurring developer questions. They developed a two-level taxonomy (What, Why, Where, How) and used a "seed-and-instantiate" approach. By parsing repositories into typed graphs using tree-sitter, they generated 576 high-quality, human-validated QA pairs grounded in real-world code.

The four-stage pipeline for SWE-QA benchmark construction: Seed Collection, Question Instantiation, Answer Collection, and Data Validation.

SWE-QA-Agent

To address the limitations of standard Retrieval-Augmented Generation (RAG), which often lacks context, the authors developed SWE-QA-Agent. This autonomous agent uses a ReAct (Reason + Act) loop to interact with the codebase. The agent is equipped with tools to:

  • GetRepoStructure: View the file tree to understand the layout.
  • SearchContent: Perform semantic searches to find relevant snippets.
  • ReadFile: Inspect specific files using command-line tools like cat and grep.
  • Finish: Synthesize gathered evidence into a final, grounded answer.

Datasets and Evaluation

SWE-QA comprises 576 QA pairs across 11 popular Python repositories, including major projects like Scikit-learn, Pytest, and Matplotlib, totaling over 3 million lines of code.

Distribution of the 576 QA pairs across the selected Python repositories.

Evaluation was conducted using an "LLM-as-Judge" approach (utilizing GPT-5) and corroborated by a study with professional software engineers to ensure the metrics aligned with human judgment of completeness and reasoning quality.

Key Insights and Results

Experimental results revealed several critical findings regarding the current state of repository-level reasoning:

  • Taxonomy of Difficulty: Models perform well on "What" (definitions) and "Why" (design rationale) questions, likely because these are often documented in docstrings. However, they struggle significantly with "How" (procedural implementation) and "Where" (locational tracing), which require tracing data flows across multiple files.
  • The Power of Agency: The iterative "plan-act-observe" loop of the SWE-QA-Agent significantly outperformed static RAG. For instance, DeepSeek-V3 improved its score from 34.38 to 42.70 (+8.32 points) when using the agentic framework.
  • Top Performers: Claude 3.7 Sonnet paired with the SWE-QA-Agent achieved the highest overall score of 47.82, surpassing commercial tools like Cursor (45.55).
  • Repository Brittleness: Performance varied significantly across repositories. Models often struggled with pytest due to its unconventional coding patterns and high architectural complexity, highlighting a lack of robustness in handling diverse coding styles.

Conclusion and Future Work

SWE-QA establishes a new frontier for evaluating LLMs in realistic software engineering environments. While agentic frameworks show great promise, significant challenges remain in cross-file procedural reasoning and handling architectural complexity. Future work will focus on expanding the benchmark to other programming languages and exploring "dynamic" QA scenarios where the codebase evolves over time.

Summary updated

Abstract

Understanding and reasoning about entire software repositories is an essential capability for intelligent software engineering tools. While existing benchmarks such as CoSQA and CodeQA have advanced the field, they predominantly focus on small, self-contained code snippets. These setups fail to capture the complexity of real-world repositories, where effective understanding and reasoning often require navigating multiple files, understanding software architecture, and grounding answers in long-range code dependencies. In this paper, we present SWE-QA, a repository-level code question answering (QA) benchmark designed to facilitate research on automated QA systems in realistic code environments. SWE-QA involves 576 high-quality question-answer pairs spanning diverse categories, including intention understanding, cross-file reasoning, and multi-hop dependency analysis. To construct SWE-QA, we first crawled 77,100 GitHub issues from 11 popular repositories. Based on an analysis of naturally occurring developer questions extracted from these issues, we developed a two-level taxonomy of repository-level questions and constructed a set of seed questions for each category. For each category, we manually curated and validated questions and collected their corresponding answers. As a prototype application, we further develop SWE-QA-Agent, an agentic framework in which LLM agents reason and act to find answers automatically. We evaluate six advanced LLMs on SWE-QA under various context augmentation strategies. Experimental results highlight the promise of LLMs, particularly our SWE-QA-Agent framework, in addressing repository-level QA, while also revealing open challenges and pointing to future research directions.
BibTeX
@misc{peng2025swe,
  author = {Weihan Peng and Yuling Shi and Yuhang Wang and Xinyun Zhang and Beijun Shen and Xiaodong Gu},
  title = {{SWE-QA: Can Language Models Answer Repository-level Code Questions?}},
  year = {2025},
  eprint = {2509.14635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.14635},
}
Figure