SpeCrawler: Generating OpenAPI Specifications from API Documentation Using Large Language Models

Koren Lazar, Matan Vetzler, Guy Uziel, David Boaz, Esther Goldbraich, David Amid, Ateret Anaby-TavorarXiv 2024 · arXiv

TL;DR

SpeCrawler is a novel, multi-stage framework that automatically converts unstructured web API documentation into machine-readable OpenAPI Specifications (OAS). By combining rule-based HTML scraping with a carefully orchestrated Large Language Model (LLM) pipeline, it successfully bridges the gap between human-centric documentation and the structured schemas required for automated testing, API orchestration, and LLM tool-use.

What Problem Does This Paper Address?

While APIs are the backbone of modern software integration, their documentation is notoriously fragmented. Most online API documentation is written for human developers, varying wildly in layout, language, and structure across different websites.

To automate API integration or allow LLM agents to use external tools effectively, we need machine-readable descriptions like the OpenAPI Specification (OAS). However, manually writing and maintaining these specs is labor-intensive and error-prone. Previous attempts to automate this process using rigid, rule-based scrapers or classic machine learning often fail because they cannot generalize to new, unseen documentation layouts.

Methods

SpeCrawler tackles this challenge by breaking down the generation process into a multi-stage pipeline that combines deterministic rules with generative LLMs:

  1. Scraping & Alignment: A rule-based scraper navigates the target API webpage, handles dynamic content (like button clicks), and extracts example-style components (e.g., cURL commands and JSON responses). It uses a DOM-tree heuristic to pair corresponding requests and responses.
  2. Base OAS Generation: Instead of asking an LLM to generate a massive, complex OAS file in a single shot—which often leads to failure—SpeCrawler divides the task. It first prompts the LLM to generate a structural "skeleton" containing metadata (servers, endpoints, parameters). Then, it generates JSON schemas from the request/response examples, breaking excessively long or nested objects into smaller segments before aggregating them back into the skeleton.
  3. Enrichment: Example code rarely contains full parameter descriptions or constraint details. SpeCrawler uses a semantic-based algorithm to isolate the "minimal ancestor" HTML element containing the reference documentation (parameter tables). It then uses an LLM via In-Context Learning (ICL) to parse this raw HTML into structured data (TSV tables for requests, nested JSON objects for responses).
Figure 2: SpeCrawler Architecture: This diagram demonstrates the carefully designed steps involved in transforming REST API documentation into an accurate OpenAPI specification (OAS). The process begins by extracting pairs of request and response elements from the API documentation webpage, which serve as the foundation for creating a skeletal OAS, as described in details in 3.2. Then, the descriptive section of the documentation is used to gather comprehensive details about the API, its request and response elements, and their parameters, as described in 3.3. Subsequently, the outputs from both processes are integrated to form a comprehensive OAS. Both procedures rely on large language models for generation.

To select the best examples for In-Context Learning, SpeCrawler computes the cosine similarity of HTML tag frequency distributions between the target page and a pool of labeled examples, ensuring the LLM receives formatting demonstrations from structurally similar websites.

Datasets

The authors evaluated SpeCrawler using several manually curated evaluation sets:

  • Syntactic Evaluation Corpus: 49 documentation pages covering approximately 189 endpoints, used to test the structural validity of the base OAS generation.
  • Enrichment Dataset: 15 challenging HTML scopes from diverse API documentation websites, manually tagged to evaluate parameter extraction.
  • End-to-End Evaluation Dataset: 30 complete API documentation pages mapped to ground-truth, human-authored OAS files, totaling around 100 endpoints.
  • Challenging URL Benchmark: 10 highly complex documentation URLs used to compare SpeCrawler against standard commercial solutions.

Key Insights and Ideas

  • Decomposition is Crucial: Expecting an LLM to output a fully formed, deeply nested OAS file directly from a raw webpage fails. Decomposing the task into skeleton generation, schema extraction, and text enrichment makes the problem tractable for standard foundation models.
  • HTML Tags as Structural Signatures: Using the frequency distribution of HTML tags acts as an effective proxy for layout similarity. This allows the system to dynamically select the most relevant few-shot examples for the LLM during the enrichment phase.
  • The Context Window Trap: Simply throwing a massive raw HTML page into a long-context model like GPT-4-Turbo yields poor results. Raw HTML contains too much noise, which dilutes the model's attention and leads to low recall of actual API parameters. Smart filtering and chunking remain essential.
Figure 4: Labeled data - This figure provides a visual representation of input-output pairs from the enrichment generation stage, sourced from PayPal Developer and Amplitude APIs. The top section displays examples of a request element, while the bottom section displays a response element example. On the left side of the figure, you’ll find input sources, which consist of scraped raw HTML scopes from API documentation websites. On the right side, the results of the enrichment generation process are presented. For request elements, the results are formatted as a TSV table, while for response elements, they are showcased as a response OpenAPI schema nested object.

Key Results

The authors evaluated SpeCrawler across multiple state-of-the-art LLMs, including Code Llama, Llama-2, Mistral, Mixtral, and IBM’s Granite model.

  • Base Generation: Code Llama-34b-Instruct achieved the highest ratio of valid OAS generation at 89%, while IBM's Granite model achieved 73% valid OAS but with the lowest rate of syntax warnings (0.48).
  • End-to-End Performance: Activating the enrichment module significantly boosted precision, recall, and F1 scores across all models. IBM's Granite model emerged as the top overall performer in end-to-end testing, excelling at parameter name extraction.
  • Comparison to Baselines: When tested against GPT-4-Turbo and OpenAI's ActionGPT on 10 challenging URLs, SpeCrawler dramatically outperformed both. GPT-4-Turbo achieved a dismal precision of 0.18 and recall of 0.05, primarily because the raw HTML of the documentation pages exceeded or overwhelmed its context window. SpeCrawler maintained a precision of 0.54 and a recall of 0.47 while ensuring 100% valid JSON and OAS structure.

Limitations and Future Work

The authors note several areas for improvement:

  • Multi-Scope Enrichment: Currently, the enrichment phase relies on a single isolated HTML scope. Allowing the model to aggregate information across multiple HTML blocks could improve description accuracy.
  • Deep Crawling: The scraper is currently limited to a single documentation page. Expanding it to follow associated links within the same domain would capture scattered API details.
  • Consolidation Strategies: Future work will explore alternative ways to split the generation tasks, such as processing requests and responses completely independently before merging.

Summary updated

Abstract

In the digital era, the widespread use of APIs is evident. However, scalable utilization of APIs poses a challenge due to structure divergence observed in online API documentation. This underscores the need for automatic tools to facilitate API consumption. A viable approach involves the conversion of documentation into an API Specification format. While previous attempts have been made using rule-based methods, these approaches encountered difficulties in generalizing across diverse documentation. In this paper we introduce SpeCrawler, a comprehensive system that utilizes large language models (LLMs) to generate OpenAPI Specifications from diverse API documentation through a carefully crafted pipeline. By creating a standardized format for numerous APIs, SpeCrawler aids in streamlining integration processes within API orchestrating systems and facilitating the incorporation of tools into LLMs. The paper explores SpeCrawler's methodology, supported by empirical evidence and case studies, demonstrating its efficacy through LLM capabilities.
BibTeX
@misc{lazar2024specrawler,
  author = {Koren Lazar and Matan Vetzler and Guy Uziel and David Boaz and Esther Goldbraich and David Amid and Ateret Anaby-Tavor},
  title = {{SpeCrawler: Generating OpenAPI Specifications from API Documentation Using Large Language Models}},
  year = {2024},
  eprint = {2402.11625},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.11625},
}

Loading related content...

Figure