LlamaRestTest: Effective REST API Testing with Small Language Models

Myeongsoo Kim, Saurabh Sinha, Alessandro Orso

TL;DR

LlamaRestTest is a specialized testing framework that uses fine-tuned, quantized Llama3-8B models to solve the "hard parts" of REST API testing: generating realistic inputs and identifying complex parameter dependencies. By integrating these small language models (SLMs) with reinforcement learning and real-time server feedback, the tool outperforms much larger models like GPT-3.5 and state-of-the-art fuzzers in both code coverage and bug detection.

What Problem Does This Paper Address?

Automated REST API testing usually relies on OpenAPI Specifications (OAS). However, these specifications are often incomplete. They might define a parameter as a "string" but fail to mention it requires a specific coordinate format (like WGS84) or that it cannot be used simultaneously with another parameter.

Existing black-box testing tools (fuzzers) struggle with these Inter-Parameter Dependencies (IPDs) and complex data formats, leading to a "wall" of 400-level client errors and poor code coverage. While Large Language Models (LLMs) can help interpret these constraints from human-readable descriptions, using massive, proprietary models like GPT-4 is expensive, slow, and raises privacy concerns. This paper asks: Can we fine-tune a much smaller, open-source model to perform as well—or better—than the giants?

Methods

The authors introduce LlamaRestTest, which splits the testing task between two specialized models derived from Llama3-8B:

  1. LlamaREST-IPD: Fine-tuned to identify dependency rules (e.g., "Parameter A requires Parameter B").
  2. LlamaREST-EX: Fine-tuned to generate semantically valid example values based on parameter names and descriptions.

Key Technical Approaches: * Fine-Tuning with QLoRA: The models were trained using Quantized Low-Rank Adaptation, allowing high-performance tuning on consumer-grade hardware. * Quantization (GGUF): The authors explored 2-bit, 4-bit, and 8-bit integer formats to balance inference speed with accuracy. * Dynamic Feedback Loop: Unlike tools that only look at the static API spec, LlamaRestTest analyzes server error messages during execution. If a request fails, the models interpret the error message to refine the next test case. * RL Integration: These models are integrated into ARAT-RL, a reinforcement learning-based testing engine that prioritizes which API endpoints and parameter combinations to explore next.

Datasets

  • For Fine-Tuning:
    • IPD Dataset: Based on Martin-Lopez’s dependency database, resulting in 551 parameters with known dependencies.
  • EX Dataset: Mined from APIs-guru, encompassing over 4,133 OpenAPI specifications and 1.8 million parameter sets to learn realistic value patterns.
  • For Evaluation: 12 real-world services, including high-profile APIs like Spotify and YouTube, and complex ones like Ohsome (OpenStreetMap history).

Key Insights and Ideas

  • Specialization Trumps Scale: A fine-tuned 8B model (LlamaREST-EX) achieved a 72.44% success rate in generating valid inputs, actually beating the much larger GPT-3.5 (68.82%) used in previous tools like RESTGPT.
  • The Power of "Listening": The most significant performance gains came from the model's ability to read server responses. When a server says, "You need to define one of the boundary parameters," LlamaREST-IPD identifies this as a "one-of" dependency and adjusts immediately.
  • Quantization as a Feature, Not a Bug: While 2-bit quantization led to some accuracy loss, 8-bit quantization provided the best balance. It was fast enough to allow for significantly more test iterations within the same time budget, leading to higher overall code coverage than the unquantized "full" model.
  • Ablation Success: Removing the IPD component caused the largest drop in coverage, proving that understanding how parameters interact is the single most important factor in testing modern APIs.

Key Results

  • Bug Discovery: LlamaRestTest identified 204 internal server errors (500 status codes), significantly more than competitors like RESTler (130) or EvoMaster (130).
  • Coverage Boost: It outperformed state-of-the-art tools by 10.0–22.7 percentage points in method coverage.
  • Efficiency: The 8-bit quantized model processed 37.5 operations per run, compared to only 23 operations for the base Llama3 model, representing a massive gain in testing throughput.
  • Zero-Shot vs. Fine-Tuning: The base Llama3-8B model was nearly useless for IPD detection (identifying only 2/17 rules), whereas the fine-tuned version identified 12/17.

Limitations and Future Work

  • Domain Specificity: The model struggled with APIs requiring deep domain-specific financial knowledge (e.g., the FDIC API) or highly specific regional formats (e.g., "en-GB" vs "en").
  • 2-Bit Precision Loss: At the extreme 2-bit quantization level, the model occasionally hallucinated or lost the "logic" required for complex scenarios, leading to lower coverage in services like Spotify.
  • Future Directions: The authors suggest expanding the approach to handle more diverse API types (beyond REST) and further refining the dynamic interaction between the LLM and the testing engine to handle even more complex multi-step stateful scenarios.

Summary updated

Abstract

Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on OpenAPI specifications. Although Large Language Models (LLMs) have shown promising test-generation abilities, their application to REST API testing remains mostly unexplored. We present LlamaRestTest, a novel approach that employs two custom LLMs-created by fine-tuning and quantizing the Llama3-8B model using mined datasets of REST API example values and inter-parameter dependencies-to generate realistic test inputs and uncover inter-parameter dependencies during the testing process by analyzing server responses. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller models to outperform much larger models in detecting actionable parameter-dependency rules and generating valid inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing, balancing effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools use RESTGPT-enhanced specifications. Finally, through an ablation study, we show that each component of LlamaRestTest contributes to its overall performance.
BibTeX
@inproceedings{kim2025llamaresttest,
  author = {Myeongsoo Kim and Saurabh Sinha and Alessandro Orso},
  title = {{LlamaRestTest: Effective REST API Testing with Small Language Models}},
  booktitle = {Proceedings of the ACM on Software Engineering},
  year = {2025},
  doi = {10.1145/3715737},
}

Loading related content...

Figure