TL;DR
Researchers from CMU and IBM have developed ICICL (Iterated-Calls In-Context Learning), a technique that automatically generates realistic data examples for Web API specifications. By combining smart retrieval with multiple, varied LLM prompts, they achieved a 116% increase in code coverage during automated security testing and helped developers understand APIs 50% faster.
What Problem Does This Paper Address?
Modern software runs on REST APIs, usually described by OpenAPI specifications. These specs are supposed to be the "source of truth," but they have a major flaw: they rarely include concrete examples of what the data should look like. In a dataset of over 13,000 API parameters, the authors found that only about 15% had human-written examples.
This "example gap" creates a domino effect of problems:
- Testing (Fuzzing): Automated tools that hammer APIs to find bugs often fail because they generate "garbage" data that gets rejected by basic input validation, never reaching the deeper, more complex code.
- Documentation: Developers struggle to understand how to use an endpoint without seeing a sample request.
- AI Chatbots: LLM-based assistants need examples to learn how to map natural language (e.g., "Check my balance") to specific API parameters.
Methods
The authors introduce ICICL, which moves beyond simple "one-shot" prompting. The workflow consists of three main stages:
- Retrieval-Based Prompting: Instead of asking an LLM to "guess" an example, the system uses a "Parameter Bank" (mined from thousands of existing public APIs). It uses the BM25 algorithm to find parameters that are semantically similar to the one it's trying to solve.
- Iterated Calls (The Core Innovation):
- The Greedy Anchor: First, it calls the LLM (Falcon-40B) with the most similar matches using "greedy decoding" (temperature 0) to get one highly likely, correct example.
- The Ensemble Approach: Inspired by how "ensembles" of different models perform better than one, ICICL creates 10 different prompt contexts. Each context contains a different mix of retrieved examples. It calls the LLM 10 times at a higher temperature (0.5) to generate a diverse pool of candidates.
- Post-Processing: The system filters out candidates that don't match the required data type. It then uses BERT embeddings to select the top three examples that are diverse but still semantically close to the "Greedy Anchor" to ensure they aren't hallucinations.
Datasets
- Parameter Bank: 13,346 parameters extracted from 1,236 OpenAPI specifications mined from API Guru.
- Intrinsic Evaluation: A randomly sampled set of 1,000 parameters for testing correctness and diversity.
- Extrinsic Benchmarks:
- Fuzzing: 7 widely-used REST services (e.g., REST Countries, YouTube, FDIC).
- Chatbots: The MixATIS and Schema Guided Dialog (SGD) datasets.
- Human Study: A custom-built pilot study with 6 expert developers.
Key Insights and Ideas
- Diversity vs. Correctness: Standard In-Context Learning (ICL) is a trade-off. If you want correct answers, you get repetitive ones. If you want diverse ones, you get "hallucinations." ICICL solves this by using the "Greedy Anchor" as a safety rail while varying the context (the examples provided in the prompt) rather than just the model's randomness.
- Synthetic is Better than Nothing: Even though the LLM isn't 100% accurate, the "noisy" synthetic examples are still vastly more useful for automated tools than having no examples at all.
- The Power of Context Sampling: By slightly changing which "few-shot" examples the LLM sees in each iteration, the model explores different valid formats for the same parameter (e.g., different ways to represent a date or a currency code).
Key Results
- Correctness: 74.3% of the generated examples were manually verified as semantically correct.
- Fuzzing (Security Testing): Adding ICICL examples to four popular API fuzzers (like EvoMaster and MoREST) increased branch coverage by an average of 116%. It allowed the testers to bypass "4xx" errors and actually exercise the API logic.
- Chatbots: On the SeqATIS benchmark, the generated examples improved intent recognition by 3% and slot filling by 5%.
- Human Productivity: In a pilot study, developers were able to complete API-related tasks 50% faster when provided with ICICL-generated examples, though they tended to "blindly trust" the AI, sometimes copying incorrect values.
Limitations and Future Work
- The Trust Trap: The human study revealed that developers often treat AI-generated examples as "oracles." If the AI generates a subtly wrong example, the human might copy it into their code without checking.
- Cost: Making 11 calls to an LLM (1 greedy + 10 diverse) is more computationally expensive and slower than a single call.
- Hallucinations: While 74% correctness is high, the remaining 26% of incorrect examples could still cause issues in production-critical systems.
- Future Direction: The authors suggest exploring how to better integrate these synthetic examples into the developer's IDE to provide "just-in-time" documentation.
Summary updated