A Tool for Generating Exceptional Behavior Tests With Large Language Models

Linghan Zhong, Samuel Yuan, Jiyang Zhang, Yu Liu, Pengyu Nie, Junyi Jessy Li, Milos Gligoric

Abstract

Exceptional behavior tests (EBTs) are crucial in software development for verifying that code correctly handles unwanted events and throws appropriate exceptions. However, prior research has shown that developers often prioritize testing "happy paths", e.g., paths without unwanted events over exceptional scenarios. We present exLong, a framework that automatically generates EBTs to address this gap. exLong leverages a large language model (LLM) fine-tuned from CodeLlama and incorporates reasoning about exception-throwing traces, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. Our demonstration video illustrates how exLong can effectively assist developers in creating comprehensive EBTs for their project.

Introduction

In modern software engineering, ensuring robustness requires more than just verifying the "happy path"—the execution sequence where all inputs are valid and resources are available. Robust systems must gracefully handle exceptional behaviors, such as invalid user inputs, missing file resources, or network failures, by throwing and managing appropriate exceptions. Despite their importance, Exceptional Behavior Tests (EBTs) are frequently neglected by developers due to the complexity of navigating the "reachability problem": triggering a specific throw statement often requires satisfying an intricate web of conditional logic and state configurations.

Traditional automated testing tools like Randoop and EvoSuite often struggle with EBTs because they rely on random or search-based strategies that may never hit the specific edge cases required to trigger an exception. We introduce exLong, a framework designed to bridge this gap by leveraging the reasoning capabilities of Large Language Models (LLMs) to generate targeted, human-readable EBTs.

The exLong Framework

The core of exLong is a fine-tuned CodeLlama model that operates within a structured analysis and generation pipeline. Unlike general-purpose LLMs that may hallucinate or fail to understand complex project dependencies, exLong is provided with rich execution context to solve the reachability problem.

The exLong framework pipeline: from task definition to EBT generation.

As illustrated in the framework overview, exLong gathers three critical pieces of information during its analysis phase:

  1. Stack Traces: These provide the sequence of method calls necessary to reach the target code, helping the model understand the execution flow and the necessary call hierarchy.
  2. Guard Expressions: These are the logical constraints (e.g., if statements) that must be satisfied to trigger the throw statement. exLong simplifies these expressions using symbolic substitution, providing the LLM with a clear "logical map" to the error state.
  3. Relevant Non-EBTs: By including existing "happy path" tests from the repository in the prompt, exLong learns the project's specific API usage, naming conventions, and object setup procedures. This "few-shot" learning approach ensures the generated tests are stylistically consistent with the existing codebase.

The tool is designed for versatility, offering two primary modes of operation. In developer-oriented mode, a developer selects a specific method and a line of code that throws an exception, and exLong generates the corresponding test. In machine-oriented mode, the tool automatically scans an entire repository to identify and generate EBTs for every throw statement it finds. Furthermore, exLong supports a quantized version (via Ollama), which reduces memory usage by 75%, making it accessible for use on consumer-grade hardware without requiring massive GPU clusters.

Evaluation and Results

We evaluated exLong using datasets derived from CodeSearchNet, focusing on Java projects hosted on GitHub. The evaluation covered 434 developer-oriented examples and 649 machine-oriented examples across 30 different projects.

Quantitative comparison of exLong against GPT-3.5 and CAT-LM across key metrics.

As shown in the performance table, exLong significantly outperforms both general-purpose and specialized LLMs. In terms of Throw Coverage (ThrowCov%)—the percentage of generated tests that successfully trigger the intended exception—exLong achieved 59.45%, surpassing GPT-3.5 (48.39%) and CAT-LM (30.03%). Additionally, exLong produced the highest percentage of compilable (82.10%) and runnable (67.36%) tests.

When compared to traditional tools like Randoop and EvoSuite, exLong demonstrated superior capability in covering unique throw statements. While traditional tools often fail due to complex state requirements, exLong's ability to reason about guard expressions allows it to bypass the reachability problem by directly constructing the necessary state in the test code.

Conclusion

exLong represents a significant step forward in automating the generation of robust test suites. By combining the generative power of LLMs with precise static and dynamic analysis context, it effectively addresses the reachability problem inherent in EBTs. While currently optimized for Java and Maven environments, the underlying principles of using guard expressions and stack traces are broadly applicable. Future work will focus on extending support to other programming languages and reducing the dependency on existing non-EBTs for context, ensuring that even "greenfield" projects can benefit from automated exception testing.

Summary updated

Abstract

Exceptional behavior tests (EBTs) are crucial in software development for verifying that code correctly handles unwanted events and throws appropriate exceptions. However, prior research has shown that developers often prioritize testing "happy paths", e.g., paths without unwanted events over exceptional scenarios. We present exLong, a framework that automatically generates EBTs to address this gap. exLong leverages a large language model (LLM) fine-tuned from CodeLlama and incorporates reasoning about exception-throwing traces, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. Our demonstration video illustrates how exLong can effectively assist developers in creating comprehensive EBTs for their project (available at https://youtu.be/Jro8kMgplZk).
BibTeX
@inproceedings{zhong2025tool,
  author = {Linghan Zhong and Samuel Yuan and Jiyang Zhang and Yu Liu and Pengyu Nie and Junyi Jessy Li and Milos Gligoric},
  title = {{A Tool for Generating Exceptional Behavior Tests With Large Language Models}},
  booktitle = {SIGSOFT FSE Companion},
  year = {2025},
  url = {https://openreview.net/forum?id=xXSDmQlZQt},
}
Figure