Abstract
As REST APIs have become widespread in modern web services, comprehensive testing of these APIs has become increasingly crucial. Due to the vast search space consisting of operations, parameters, and parameter values along with their complex dependencies and constraints, current testing tools suffer from low code coverage, leading to suboptimal fault detection. To address this limitation, we present a novel tool, AutoRestTest, which integrates the Semantic Operation Dependency Graph (SODG) with Multi-Agent Reinforcement Learning (MARL) and large language models (LLMs) for effective REST API testing. AutoRestTest determines operation-dependent parameters using the SODG and employs five specialized agents (operation, parameter, value, dependency, and header) to identify dependencies of operations and generate operation sequences, parameter combinations, and values. AutoRestTest provides a command-line interface and continuous telemetry on successful operation count, unique server errors detected, and time elapsed. Upon completion, AutoRestTest generates a detailed report highlighting errors detected and operations exercised. In this paper, we introduce our tool and present preliminary results.
Introduction
Testing REST APIs presents a significant challenge due to the combinatorial explosion of the search space. Effective testing requires navigating a complex web of HTTP methods, headers, and parameters, often with intricate state dependencies. For instance, a GET request for a specific resource might require an identifier generated by a preceding POST request. Traditional automated testing tools, such as fuzzers, often fail to achieve high code coverage because they struggle to satisfy these dependencies, frequently resulting in "400 Bad Request" errors that prevent exploration of deeper application logic.
Methodology
AutoRestTest addresses these challenges through a three-pillar architecture combining graph-based modeling, Multi-Agent Reinforcement Learning (MARL), and Large Language Models (LLMs).
- Semantic Property Dependency Graph (SPDG): To manage parameter dependencies, AutoRestTest parses the OpenAPI Specification (OAS) and utilizes GloVe word embeddings to calculate semantic similarity between response fields and request parameters. This allows the tool to intelligently link outputs to inputs, significantly narrowing the search space.
- Multi-Agent Reinforcement Learning (MARL): The tool employs five specialized agents—Operation, Parameter, Value, Dependency, and Header—that use Q-learning to optimize their actions. While the Operation Agent is rewarded for uncovering errors (4xx/5xx), the other agents are incentivized by successful (2xx) responses, creating a cooperative-competitive environment that drives deep API exploration.
- LLM Integration: AutoRestTest leverages LLMs (e.g., GPT-4o) to generate realistic, context-aware data for request bodies and headers. A "Mutator" component further enhances this by probabilistically altering these valid inputs to discover edge-case bugs that standard models might overlook.
Evaluation and Results
The tool was evaluated against four real-world services: FDIC, OMDb, OhSome, and Spotify. Performance was compared to state-of-the-art tools including RESTler, EvoMaster, ARAT-RL, and MoRest.
In a one-hour testing window, AutoRestTest demonstrated superior performance, exercising 26 unique operations across the four services, compared to just 12 by the next best tool, ARAT-RL. Notably, AutoRestTest was the only tool capable of successfully processing operations in the complex OhSome API (achieving 12 successful operations where others achieved zero) and the only one to detect a 5xx Internal Server Error in the production-grade Spotify API.
Conclusion and Future Work
AutoRestTest significantly advances automated REST API testing by combining semantic analysis with multi-agent learning and intelligent data generation. While the use of LLMs introduces some latency and cost—partially mitigated by a caching system using Python’s shelve library—the tool's ability to achieve high coverage and detect deep logic flaws is evident. Future work will focus on expanding support for various OAS versions and enhancing stateful testing capabilities for long-duration operation sequences.
Summary updated