A Tool for Generating Exceptional Behavior Tests With Large Language Models

Linghan Zhong, Samuel Yuan, Jiyang Zhang, Yu Liu, Pengyu Nie, Junyi Jessy Li, Milos Gligoric

TL;DR

Researchers have developed exLong, a tool that leverages fine-tuned Large Language Models (LLMs) to automatically generate tests for exceptional code behaviors (error handling). By providing the model with specific execution context—like stack traces and the logical conditions required to trigger an error—exLong significantly outperforms general-purpose LLMs and traditional testing tools in creating runnable, effective exception tests.

What Problem Does This Paper Address?

In software engineering, "happy path" testing (verifying that code works when everything goes right) is the norm. However, robust software must also handle "exceptional behaviors"—situations where things go wrong, like invalid inputs or missing resources.

Writing Exceptional Behavior Tests (EBTs) is notoriously difficult and tedious. Developers often neglect them due to time constraints, and traditional automated tools (like Randoop or EvoSuite) struggle to target specific error-throwing statements or produce human-readable code. The challenge lies in the "reachability" problem: to trigger a specific exception, a test must navigate a complex maze of conditional logic and state setups.

Methods

The authors present exLong, a framework built on a fine-tuned version of CodeLlama. The core innovation isn't just the model, but the structured context provided to it during inference.

To generate a test that hits a specific throw statement, exLong gathers three critical pieces of information:

  1. Stack Traces: The sequence of method calls required to reach the target code.
  2. Guard Expressions: A logical formula representing the constraints (if-statements, assignments) that must be satisfied to reach the throw statement. exLong uses symbolic substitution to simplify these conditions for the LLM.
  3. Relevant Non-EBTs: Existing "happy path" tests from the same repository. These serve as "few-shot" examples, teaching the LLM the project's specific coding conventions and how to set up the necessary objects.

The tool supports two modes:

  • Developer-oriented: A developer selects a specific method and a line of code that throws an exception, and exLong generates the test.
  • Machine-oriented: The tool automatically scans an entire repository and attempts to generate EBTs for every throw statement it finds.

To make the tool accessible, the authors also implemented a quantized version (using Ollama), allowing it to run on consumer-grade hardware with 75% less memory usage.

Datasets

The researchers evaluated exLong using a dataset derived from CodeSearchNet, specifically focusing on Java projects hosted on GitHub.

  • For the developer-oriented evaluation, they used a subset of 434 examples where stack traces could be successfully extracted.
  • For the machine-oriented evaluation, they used 649 examples across 30 different projects.

Key Insights and Ideas

  • Logic over Luck: Traditional random testing relies on "getting lucky" to hit an error state. By feeding the LLM "Guard Expressions," exLong essentially gives the model the "cheat code" or the logical requirements needed to trigger the error.
  • Style Matters: By including non-exceptional tests in the prompt, the generated tests don't just work—they look like they belong in the codebase, following the project's specific API usage and naming conventions.
  • Quantization is Viable: The study found that while quantization (reducing model precision to save memory) does lead to a small drop in performance (~13% fewer executable tests), the tool remains highly effective and much more practical for developers without massive GPU clusters.

Key Results

In head-to-head comparisons, exLong demonstrated a clear lead:

  • Vs. LLMs: exLong generated 83.8% more executable EBTs than CAT-LM (a specialized test-gen model) and 9.9% more than GPT-3.5.
  • Throw Coverage: In the developer-oriented case, exLong achieved a 59.45% ThrowCov (the percentage of tests that successfully triggered the target exception), compared to 48.39% for GPT-3.5 and a mere 30.03% for CAT-LM.
  • Vs. Traditional Tools: When compared to Randoop and EvoSuite, exLong covered the most target throw statements across 30 projects, often succeeding where the analysis-based tools failed due to complex state requirements.

Limitations and Future Work

The current implementation of exLong is tailored for Java and Maven-based projects. While the underlying logic of using stack traces and guard expressions is language-agnostic, the static and dynamic analysis tools used to gather that context would need to be rebuilt for languages like Python or C#.

Additionally, the tool relies on the existence of some non-exceptional tests to provide context. In a "greenfield" project with zero tests, exLong's performance might degrade. Future work could involve exploring how to generate these initial context-providing tests or improving the symbolic analysis to handle even more complex, multi-threaded execution paths.

Summary updated

Abstract

Exceptional behavior tests (EBTs) are crucial in software development for verifying that code correctly handles unwanted events and throws appropriate exceptions. However, prior research has shown that developers often prioritize testing "happy paths", e.g., paths without unwanted events over exceptional scenarios. We present exLong, a framework that automatically generates EBTs to address this gap. exLong leverages a large language model (LLM) fine-tuned from CodeLlama and incorporates reasoning about exception-throwing traces, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. Our demonstration video illustrates how exLong can effectively assist developers in creating comprehensive EBTs for their project (available at https://youtu.be/Jro8kMgplZk).
BibTeX
@inproceedings{zhong2025tool,
  author = {Linghan Zhong and Samuel Yuan and Jiyang Zhang and Yu Liu and Pengyu Nie and Junyi Jessy Li and Milos Gligoric},
  title = {{A Tool for Generating Exceptional Behavior Tests With Large Language Models}},
  booktitle = {SIGSOFT FSE Companion},
  year = {2025},
  url = {https://openreview.net/forum?id=xXSDmQlZQt},
}

Loading related content...

Figure