TL;DR
SpeCrawler is a novel, multi-stage framework that automatically converts unstructured web API documentation into machine-readable OpenAPI Specifications (OAS). By combining rule-based HTML scraping with a carefully orchestrated Large Language Model (LLM) pipeline, it successfully bridges the gap between human-centric documentation and the structured schemas required for automated testing, API orchestration, and LLM tool-use.
What Problem Does This Paper Address?
While APIs are the backbone of modern software integration, their documentation is notoriously fragmented. Most online API documentation is written for human developers, varying wildly in layout, language, and structure across different websites.
To automate API integration or allow LLM agents to use external tools effectively, we need machine-readable descriptions like the OpenAPI Specification (OAS). However, manually writing and maintaining these specs is labor-intensive and error-prone. Previous attempts to automate this process using rigid, rule-based scrapers or classic machine learning often fail because they cannot generalize to new, unseen documentation layouts.
Methods
SpeCrawler tackles this challenge by breaking down the generation process into a multi-stage pipeline that combines deterministic rules with generative LLMs:
- Scraping & Alignment: A rule-based scraper navigates the target API webpage, handles dynamic content (like button clicks), and extracts example-style components (e.g., cURL commands and JSON responses). It uses a DOM-tree heuristic to pair corresponding requests and responses.
- Base OAS Generation: Instead of asking an LLM to generate a massive, complex OAS file in a single shot—which often leads to failure—SpeCrawler divides the task. It first prompts the LLM to generate a structural "skeleton" containing metadata (servers, endpoints, parameters). Then, it generates JSON schemas from the request/response examples, breaking excessively long or nested objects into smaller segments before aggregating them back into the skeleton.
- Enrichment: Example code rarely contains full parameter descriptions or constraint details. SpeCrawler uses a semantic-based algorithm to isolate the "minimal ancestor" HTML element containing the reference documentation (parameter tables). It then uses an LLM via In-Context Learning (ICL) to parse this raw HTML into structured data (TSV tables for requests, nested JSON objects for responses).
To select the best examples for In-Context Learning, SpeCrawler computes the cosine similarity of HTML tag frequency distributions between the target page and a pool of labeled examples, ensuring the LLM receives formatting demonstrations from structurally similar websites.
Datasets
The authors evaluated SpeCrawler using several manually curated evaluation sets:
- Syntactic Evaluation Corpus: 49 documentation pages covering approximately 189 endpoints, used to test the structural validity of the base OAS generation.
- Enrichment Dataset: 15 challenging HTML scopes from diverse API documentation websites, manually tagged to evaluate parameter extraction.
- End-to-End Evaluation Dataset: 30 complete API documentation pages mapped to ground-truth, human-authored OAS files, totaling around 100 endpoints.
- Challenging URL Benchmark: 10 highly complex documentation URLs used to compare SpeCrawler against standard commercial solutions.
Key Insights and Ideas
- Decomposition is Crucial: Expecting an LLM to output a fully formed, deeply nested OAS file directly from a raw webpage fails. Decomposing the task into skeleton generation, schema extraction, and text enrichment makes the problem tractable for standard foundation models.
- HTML Tags as Structural Signatures: Using the frequency distribution of HTML tags acts as an effective proxy for layout similarity. This allows the system to dynamically select the most relevant few-shot examples for the LLM during the enrichment phase.
- The Context Window Trap: Simply throwing a massive raw HTML page into a long-context model like GPT-4-Turbo yields poor results. Raw HTML contains too much noise, which dilutes the model's attention and leads to low recall of actual API parameters. Smart filtering and chunking remain essential.
Key Results
The authors evaluated SpeCrawler across multiple state-of-the-art LLMs, including Code Llama, Llama-2, Mistral, Mixtral, and IBM’s Granite model.
- Base Generation: Code Llama-34b-Instruct achieved the highest ratio of valid OAS generation at 89%, while IBM's Granite model achieved 73% valid OAS but with the lowest rate of syntax warnings (0.48).
- End-to-End Performance: Activating the enrichment module significantly boosted precision, recall, and F1 scores across all models. IBM's Granite model emerged as the top overall performer in end-to-end testing, excelling at parameter name extraction.
- Comparison to Baselines: When tested against GPT-4-Turbo and OpenAI's ActionGPT on 10 challenging URLs, SpeCrawler dramatically outperformed both. GPT-4-Turbo achieved a dismal precision of 0.18 and recall of 0.05, primarily because the raw HTML of the documentation pages exceeded or overwhelmed its context window. SpeCrawler maintained a precision of 0.54 and a recall of 0.47 while ensuring 100% valid JSON and OAS structure.
Limitations and Future Work
The authors note several areas for improvement:
- Multi-Scope Enrichment: Currently, the enrichment phase relies on a single isolated HTML scope. Allowing the model to aggregate information across multiple HTML blocks could improve description accuracy.
- Deep Crawling: The scraper is currently limited to a single documentation page. Expanding it to follow associated links within the same domain would capture scattered API details.
- Consolidation Strategies: Future work will explore alternative ways to split the generation tasks, such as processing requests and responses completely independently before merging.
Summary updated