Automating API Documentation from Crowdsourced Knowledge

Bonan Kou, Zijie Zhou, Muhao Chen, Tianyi ZhangarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces AutoDoc, a novel system that automatically generates comprehensive API documentation by extracting knowledge from Stack Overflow discussions. By combining a fine-tuned retrieval model with GPT-4o for extraction and validation, AutoDoc produces more accurate and informative documentation than existing methods, even enabling smaller open-source LLMs to achieve comparable results.

What Problem Does This Paper Address?

Developers rely heavily on API documentation to understand and use software libraries. However, official API documentation is often outdated, incomplete, and fails to cover the practical nuances that developers encounter. While platforms like Stack Overflow (SO) are rich sources of this practical, up-to-date knowledge, manually sifting through countless posts to find relevant information is time-consuming and inefficient. Existing automated approaches struggle with the complexity of natural language in SO posts or require extensive manual labeling. This paper aims to bridge this gap by creating a system that can automatically extract and synthesize this valuable, crowdsourced knowledge into usable API documentation.

Methods

AutoDoc employs a four-stage pipeline:

  1. Relevant Post Retrieval: To handle the vast amount of data on SO, AutoDoc uses a Dense Passage Retrieval (DPR) model. Crucially, this model is fine-tuned on SO data to better understand the unique linguistic patterns and technical jargon found in developer discussions, improving its ability to identify relevant answer posts for specific API knowledge types.
  2. Knowledge Extraction: Once relevant posts are retrieved, GPT-4o is prompted to extract specific types of API knowledge (seven defined types, including functionality, concept, and patterns).
  3. Knowledge Validation: To combat LLM hallucinations, AutoDoc incorporates a self-checking mechanism where GPT-4o is prompted again to verify the correctness of the extracted knowledge against the source post. Incorrect or unsupported information is discarded.
  4. Knowledge Summarization: To avoid redundancy, GPT-4o is used to summarize and synthesize the validated knowledge snippets from multiple posts into a concise and coherent API document.

Datasets

The study evaluated AutoDoc on a benchmark of 48 APIs from various popular libraries: Java SDK, TensorFlow, Android SDK, and Kotlin SDK. These APIs were selected across different popularity levels (high, medium, low), determined by their percentile ranking in GitHub code search. The system was trained and evaluated using data from the Stack Overflow data dump.

Key Insights and Ideas

  • Crowdsourced Knowledge is Gold: Stack Overflow contains a wealth of practical API knowledge that is often missing from official documentation. AutoDoc demonstrates that this knowledge can be effectively leveraged.
  • Fine-tuning Matters: Simply using a general-purpose retrieval model (like the original DPR) is insufficient. Fine-tuning the retrieval model on domain-specific data (SO posts) significantly improves its ability to find relevant information.
  • LLM Self-Correction is Powerful: The knowledge validation step, where the LLM checks its own output for correctness, is crucial for mitigating hallucinations and building trust in the generated documentation.
  • Smaller LLMs Can Shine: AutoDoc's architecture allows smaller, open-source LLMs to achieve performance comparable to larger proprietary models, democratizing the generation of high-quality documentation.
  • Beyond Official Docs: A significant portion of the generated documentation contains knowledge not found in official sources, highlighting AutoDoc's value in supplementing existing resources.

Key Results

  • Accuracy: AutoDoc generated API documents that were up to 77.7% more accurate than five comparison baselines.
  • Uniqueness: The generated documents were 9.5% less duplicated than baselines.
  • Novelty: AutoDoc uncovered 34.4% more knowledge that was not present in the official API documents.
  • LLM Sensitivity: While larger LLMs produced better results, AutoDoc enabled smaller models like Mistral-7B-v0.3 to achieve comparable accuracy and uniqueness.
  • User Study: 80% of participants preferred AutoDoc's generated API documents over those from baselines (GPT-4o ZSP and SISE) across helpfulness, comprehensiveness, conciseness, overview, and overall preference.

Limitations and Future Work

  • Less Popular APIs: AutoDoc's performance (correctness and uniqueness) declines for less popular APIs, as there are fewer SO discussions to draw from. This can lead to increased reliance on the LLM's internal knowledge, potentially increasing hallucinations.
  • Traceability: The summarization process can erase the direct link to the original SO post from which a knowledge snippet was extracted. Future work aims to develop a component for traceability to boost user trust.
  • Overlap with Official Docs: While AutoDoc aims to provide novel information, some overlap with official documentation is unavoidable. Future work could involve a component to explicitly mark or filter shared knowledge, allowing users to focus on the unique insights.
  • User Feedback: Users expressed a desire to see the original source post for generated snippets and for better handling of shared knowledge.

Summary updated

Abstract

API documentation is crucial for developers to learn and use APIs. However, it is known that many official API documents are obsolete and incomplete. To address this challenge, we propose a new approach called AutoDoc that generates API documents with API knowledge extracted from online discussions on Stack Overflow (SO). AutoDoc leverages a fine-tuned dense retrieval model to identify seven types of API knowledge from SO posts. Then, it uses GPT-4o to summarize the API knowledge in these posts into concise text. Meanwhile, we designed two specific components to handle LLM hallucination and redundancy in generated content. We evaluated AutoDoc against five comparison baselines on 48 APIs of different popularity levels. Our results indicate that the API documents generated by AutoDoc are up to 77.7% more accurate, 9.5% less duplicated, and contain 34.4% knowledge uncovered by the official documents. We also measured the sensitivity of AutoDoc to the choice of different LLMs. We found that while larger LLMs produce higher-quality API documents, AutoDoc enables smaller open-source models (e.g., Mistral-7B-v0.3) to achieve comparable results. Finally, we conducted a user study to evaluate the usefulness of the API documents generated by AutoDoc. All participants found API documents generated by AutoDoc to be more comprehensive, concise, and helpful than the comparison baselines. This highlights the feasibility of utilizing LLMs for API documentation with careful design to counter LLM hallucination and information redundancy.
BibTeX
@misc{kou2026automating,
  author = {Bonan Kou and Zijie Zhou and Muhao Chen and Tianyi Zhang},
  title = {{Automating API Documentation from Crowdsourced Knowledge}},
  year = {2026},
  eprint = {2601.08036},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.08036},
}

Loading related content...

Figure