Abstract
Examples in web API specifications are critical for API testing, developer understanding, and the training of API-focused chatbots. However, the majority of existing API specifications lack human-written examples. This paper introduces Iterated-Calls In-Context Learning (ICICL), a novel technique designed to generate high-quality, diverse examples for web API parameters. By leveraging a retrieval-based prompting strategy and an iterative ensemble approach, ICICL overcomes the traditional trade-off between correctness and diversity in Large Language Model (LLM) outputs. Our evaluation demonstrates that ICICL significantly improves performance across downstream tasks, including a 116% increase in code coverage during automated testing and a 50% reduction in time for developer comprehension tasks.
Introduction
Modern software ecosystems rely heavily on REST APIs, typically documented using the OpenAPI specification. While these specifications define the structure of data, they frequently omit concrete examples. In a study of over 13,000 API parameters, researchers found that only approximately 15% contained human-written examples. This "example gap" hinders automated fuzzing tools, which struggle to generate valid data to reach deep code paths, and complicates the onboarding process for developers and AI assistants alike.
To address this, we present ICICL, a method that moves beyond standard In-Context Learning (ICL) to generate synthetic examples that are both semantically accurate and sufficiently varied for robust testing and documentation.
Methodology: The ICICL Framework
ICICL utilizes a three-stage pipeline to transform raw API parameter definitions into a curated set of examples.
1. Retrieval-Based Prompting
Rather than relying on the LLM to generate examples in a vacuum, ICICL utilizes a "Parameter Bank" containing over 13,000 parameters mined from public OpenAPI specifications. When a new parameter requires examples, the system uses the BM25 algorithm to retrieve semantically similar parameters from the bank, providing the LLM with relevant few-shot context.
2. Iterated Calls and the Greedy Anchor
Standard ICL often faces a trade-off: low-temperature "greedy" decoding produces correct but repetitive results, while high-temperature decoding produces diversity at the cost of "hallucinations." ICICL resolves this through an ensemble approach:
- The Greedy Anchor: The LLM (Falcon-40B) is first called with the most similar retrieved matches using greedy decoding (temperature 0) to establish a highly reliable baseline.
- Context Sampling: The system then generates 10 additional prompt contexts, each featuring a different mix of retrieved examples. The LLM is called 10 times (temperature 0.5) to explore the valid format space of the parameter.
3. Post-Processing and Selection
Candidates are filtered based on data type constraints. To ensure quality, the system uses BERT embeddings to select the top three examples that maintain semantic proximity to the "Greedy Anchor" while maximizing internal diversity.
Evaluation and Results
We evaluated ICICL through intrinsic metrics (correctness and diversity) and extrinsic downstream applications.
Intrinsic Performance
Manual verification of 1,000 sampled parameters showed that 74.3% of ICICL-generated examples were semantically correct. By varying the prompt context rather than just model randomness, ICICL successfully generated diverse formats for complex parameters like dates, currency codes, and identifiers.
Extrinsic Impact
- Automated Testing (Fuzzing): Integrating ICICL examples into four industry-standard API fuzzers (including EvoMaster and MoREST) resulted in an average 116% increase in branch coverage. By providing valid data formats, the fuzzers bypassed initial validation layers (4xx errors) to test deeper business logic.
- AI Chatbots: On the SeqATIS and SGD benchmarks, synthetic examples improved intent recognition and slot-filling accuracy by 3% and 5%, respectively, demonstrating that LLMs can effectively learn API mappings from synthetic data.
- Human Productivity: In a pilot study with expert developers, participants completed API-integration tasks 50% faster when provided with ICICL examples compared to baseline documentation.
Discussion and Limitations
While ICICL represents a significant step forward, it introduces new challenges. The "Trust Trap" is a primary concern: developers in our study often treated synthetic examples as ground truth, occasionally copying incorrect values into their code. Additionally, the computational cost of making 11 LLM calls per parameter is higher than single-shot methods. Finally, while 74.3% correctness is high, the remaining error rate suggests that ICICL is best suited as a productivity aid or a testing tool rather than a fully autonomous documentation generator for mission-critical systems.
Conclusion
ICICL demonstrates that iterated-calls in-context learning can effectively bridge the example gap in Web API specifications. By balancing correctness with diversity through context sampling, ICICL provides a scalable solution for improving API testability, documentation clarity, and the performance of AI-driven developer tools.
Summary updated