Improving Examples in Web API Specifications using Iterated-Calls In-Context Learning

Kush Jain, Kiran Kate, Jason Tsay, Claire Le Goues, Martin Hirzel

Abstract

Examples in web API specifications are critical for API testing, developer understanding, and the training of API-focused chatbots. However, the majority of existing API specifications lack human-written examples. This paper introduces Iterated-Calls In-Context Learning (ICICL), a novel technique designed to generate high-quality, diverse examples for web API parameters. By leveraging a retrieval-based prompting strategy and an iterative ensemble approach, ICICL overcomes the traditional trade-off between correctness and diversity in Large Language Model (LLM) outputs. Our evaluation demonstrates that ICICL significantly improves performance across downstream tasks, including a 116% increase in code coverage during automated testing and a 50% reduction in time for developer comprehension tasks.

Introduction

Modern software ecosystems rely heavily on REST APIs, typically documented using the OpenAPI specification. While these specifications define the structure of data, they frequently omit concrete examples. In a study of over 13,000 API parameters, researchers found that only approximately 15% contained human-written examples. This "example gap" hinders automated fuzzing tools, which struggle to generate valid data to reach deep code paths, and complicates the onboarding process for developers and AI assistants alike.

To address this, we present ICICL, a method that moves beyond standard In-Context Learning (ICL) to generate synthetic examples that are both semantically accurate and sufficiently varied for robust testing and documentation.

Methodology: The ICICL Framework

ICICL utilizes a three-stage pipeline to transform raw API parameter definitions into a curated set of examples.

1. Retrieval-Based Prompting

Rather than relying on the LLM to generate examples in a vacuum, ICICL utilizes a "Parameter Bank" containing over 13,000 parameters mined from public OpenAPI specifications. When a new parameter requires examples, the system uses the BM25 algorithm to retrieve semantically similar parameters from the bank, providing the LLM with relevant few-shot context.

2. Iterated Calls and the Greedy Anchor

Standard ICL often faces a trade-off: low-temperature "greedy" decoding produces correct but repetitive results, while high-temperature decoding produces diversity at the cost of "hallucinations." ICICL resolves this through an ensemble approach:

  • The Greedy Anchor: The LLM (Falcon-40B) is first called with the most similar retrieved matches using greedy decoding (temperature 0) to establish a highly reliable baseline.
  • Context Sampling: The system then generates 10 additional prompt contexts, each featuring a different mix of retrieved examples. The LLM is called 10 times (temperature 0.5) to explore the valid format space of the parameter.

3. Post-Processing and Selection

Candidates are filtered based on data type constraints. To ensure quality, the system uses BERT embeddings to select the top three examples that maintain semantic proximity to the "Greedy Anchor" while maximizing internal diversity.

Evaluation and Results

We evaluated ICICL through intrinsic metrics (correctness and diversity) and extrinsic downstream applications.

Intrinsic Performance

Manual verification of 1,000 sampled parameters showed that 74.3% of ICICL-generated examples were semantically correct. By varying the prompt context rather than just model randomness, ICICL successfully generated diverse formats for complex parameters like dates, currency codes, and identifiers.

Extrinsic Impact

  • Automated Testing (Fuzzing): Integrating ICICL examples into four industry-standard API fuzzers (including EvoMaster and MoREST) resulted in an average 116% increase in branch coverage. By providing valid data formats, the fuzzers bypassed initial validation layers (4xx errors) to test deeper business logic.
  • AI Chatbots: On the SeqATIS and SGD benchmarks, synthetic examples improved intent recognition and slot-filling accuracy by 3% and 5%, respectively, demonstrating that LLMs can effectively learn API mappings from synthetic data.
  • Human Productivity: In a pilot study with expert developers, participants completed API-integration tasks 50% faster when provided with ICICL examples compared to baseline documentation.

Discussion and Limitations

While ICICL represents a significant step forward, it introduces new challenges. The "Trust Trap" is a primary concern: developers in our study often treated synthetic examples as ground truth, occasionally copying incorrect values into their code. Additionally, the computational cost of making 11 LLM calls per parameter is higher than single-shot methods. Finally, while 74.3% correctness is high, the remaining error rate suggests that ICICL is best suited as a productivity aid or a testing tool rather than a fully autonomous documentation generator for mission-critical systems.

Conclusion

ICICL demonstrates that iterated-calls in-context learning can effectively bridge the example gap in Web API specifications. By balancing correctness with diversity through context sampling, ICICL provides a scalable solution for improving API testability, documentation clarity, and the performance of AI-driven developer tools.

Summary updated

Abstract

Examples in web API specifications can be essential for API testing, API understanding, and even building chat-bots for APIs. Unfortunately, most API specifications lack human-written examples. This paper introduces a novel technique for generating examples for web API specifications. We start from in-context learning (ICL): given an API parameter, use a prompt context containing a few examples from other similar API parameters to call a model to generate new examples. However, while ICL tends to generate correct examples, those lack diversity, which is also important for most downstream tasks. Therefore, we extend the technique to iterated-calls ICL (ICICL): use a few different prompt contexts, each containing a few examples,to iteratively call the model with each context. Our intrinsic evaluation demonstrates that ICICL improves both correctness and diversity of generated examples. More importantly, our extrinsic evaluation demonstrates that those generated examples significantly improve the performance of downstream tasks of testing, understanding, and chat-bots for APIs.
BibTeX
@inproceedings{jain2025improving,
  author = {Kush Jain and Kiran Kate and Jason Tsay and Claire Le Goues and Martin Hirzel},
  title = {{Improving Examples in Web API Specifications using Iterated-Calls In-Context Learning}},
  booktitle = {AST},
  year = {2025},
  url = {https://openreview.net/forum?id=3Ojw6kHR7P},
}
Figure