Abstract
Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on OpenAPI specifications. Although Large Language Models (LLMs) have shown promising test-generation abilities, their application to REST API testing remains mostly unexplored. We present LlamaRestTest, a novel approach that employs two custom LLMs—created by fine-tuning and quantizing the Llama3-8B model using mined datasets of REST API example values and inter-parameter dependencies—to generate realistic test inputs and uncover inter-parameter dependencies during the testing process by analyzing server responses. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller models to outperform much larger models in detecting actionable parameter-dependency rules and generating valid inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing, balancing effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools use RESTGPT-enhanced specifications.
Introduction
Automated REST API testing typically relies on OpenAPI Specifications (OAS). However, these specifications are often incomplete, frequently omitting critical details such as specific coordinate formats or complex Inter-Parameter Dependencies (IPDs). For instance, a parameter might be defined as a generic "string" while requiring a specific format like WGS84, or two parameters might be mutually exclusive.
Existing black-box testing tools, or fuzzers, often struggle with these constraints, resulting in a high volume of 400-level client errors and limited code coverage. While Large Language Models (LLMs) can interpret these constraints from human-readable descriptions, relying on massive, proprietary models like GPT-4 is often impractical due to high costs, latency, and privacy concerns. LlamaRestTest addresses this by investigating whether a smaller, open-source model can be fine-tuned to match or exceed the performance of larger models in this specialized domain.
Methodology
LlamaRestTest utilizes two specialized models derived from Llama3-8B to handle the most challenging aspects of REST API testing.
The framework consists of two primary components:
- LlamaREST-IPD: This model is fine-tuned to identify dependency rules, such as when one parameter requires the presence of another.
- LlamaREST-EX: This model is fine-tuned to generate semantically valid example values based on parameter names and descriptions.
Technical Implementation
The models were developed using Quantized Low-Rank Adaptation (QLoRA), enabling high-performance fine-tuning on consumer-grade hardware. To optimize inference speed and efficiency, the authors explored various quantization formats (GGUF), specifically 2-bit, 4-bit, and 8-bit integers.
A key innovation in LlamaRestTest is its Dynamic Feedback Loop. Unlike static tools, LlamaRestTest analyzes server error messages in real-time. If a request fails, the models interpret the error message to refine subsequent test cases. These models are integrated into ARAT-RL, a reinforcement learning-based testing engine that intelligently prioritizes API endpoints and parameter combinations.
Experimental Setup
The models were fine-tuned using two primary datasets:
- IPD Dataset: Derived from Martin-Lopez’s dependency database, containing 551 parameters with known dependencies.
- EX Dataset: Mined from APIs-guru, covering over 4,133 OpenAPI specifications and 1.8 million parameter sets to learn realistic value patterns.
The evaluation was conducted across 12 real-world services, including high-traffic APIs like Spotify and YouTube, as well as complex services like Ohsome (OpenStreetMap history).
Results and Discussion
The evaluation revealed several key insights:
- Specialization vs. Scale: The fine-tuned 8B model (LlamaREST-EX) achieved a 72.44% success rate in generating valid inputs, outperforming the much larger GPT-3.5 (68.82%) used in RESTGPT.
- Dynamic Learning: The ability to "listen" to server responses provided significant gains. For example, when a server indicates a missing boundary parameter, LlamaREST-IPD identifies the dependency and adjusts the test generation immediately.
- Quantization Efficiency: 8-bit quantization offered the best balance between speed and accuracy. It allowed for significantly more test iterations within the same time budget, leading to higher overall code coverage compared to the unquantized model.
- Bug Discovery and Coverage: LlamaRestTest identified 204 internal server errors (500 status codes), surpassing competitors like RESTler and EvoMaster (both at 130). It also outperformed state-of-the-art tools by 10.0–22.7 percentage points in method coverage.
The importance of fine-tuning was further highlighted by the fact that the base Llama3-8B model identified only 2 out of 17 IPD rules, whereas the fine-tuned version identified 12.
Conclusion
LlamaRestTest demonstrates that small language models, when properly fine-tuned and quantized, can be more effective and efficient than large-scale models for specialized tasks like REST API testing.
However, limitations remain. The model struggled with highly domain-specific APIs (e.g., financial data) or regional formats. Additionally, extreme 2-bit quantization led to some accuracy loss and occasional hallucinations. Future work will focus on expanding the approach to other API types and further refining the dynamic interaction between the LLM and the testing engine to handle complex, multi-step stateful scenarios.
Summary updated