TL;DR
This paper introduces AutoDoc, a novel system that automatically generates comprehensive API documentation by extracting knowledge from Stack Overflow discussions. By combining a fine-tuned retrieval model with GPT-4o for extraction and validation, AutoDoc produces more accurate and informative documentation than existing methods, even enabling smaller open-source LLMs to achieve comparable results.
What Problem Does This Paper Address?
Developers rely heavily on API documentation to understand and use software libraries. However, official API documentation is often outdated, incomplete, and fails to cover the practical nuances that developers encounter. While platforms like Stack Overflow (SO) are rich sources of this practical, up-to-date knowledge, manually sifting through countless posts to find relevant information is time-consuming and inefficient. Existing automated approaches struggle with the complexity of natural language in SO posts or require extensive manual labeling. This paper aims to bridge this gap by creating a system that can automatically extract and synthesize this valuable, crowdsourced knowledge into usable API documentation.
Methods
AutoDoc employs a four-stage pipeline:
- Relevant Post Retrieval: To handle the vast amount of data on SO, AutoDoc uses a Dense Passage Retrieval (DPR) model. Crucially, this model is fine-tuned on SO data to better understand the unique linguistic patterns and technical jargon found in developer discussions, improving its ability to identify relevant answer posts for specific API knowledge types.
- Knowledge Extraction: Once relevant posts are retrieved, GPT-4o is prompted to extract specific types of API knowledge (seven defined types, including functionality, concept, and patterns).
- Knowledge Validation: To combat LLM hallucinations, AutoDoc incorporates a self-checking mechanism where GPT-4o is prompted again to verify the correctness of the extracted knowledge against the source post. Incorrect or unsupported information is discarded.
- Knowledge Summarization: To avoid redundancy, GPT-4o is used to summarize and synthesize the validated knowledge snippets from multiple posts into a concise and coherent API document.
Datasets
The study evaluated AutoDoc on a benchmark of 48 APIs from various popular libraries: Java SDK, TensorFlow, Android SDK, and Kotlin SDK. These APIs were selected across different popularity levels (high, medium, low), determined by their percentile ranking in GitHub code search. The system was trained and evaluated using data from the Stack Overflow data dump.
Key Insights and Ideas
- Crowdsourced Knowledge is Gold: Stack Overflow contains a wealth of practical API knowledge that is often missing from official documentation. AutoDoc demonstrates that this knowledge can be effectively leveraged.
- Fine-tuning Matters: Simply using a general-purpose retrieval model (like the original DPR) is insufficient. Fine-tuning the retrieval model on domain-specific data (SO posts) significantly improves its ability to find relevant information.
- LLM Self-Correction is Powerful: The knowledge validation step, where the LLM checks its own output for correctness, is crucial for mitigating hallucinations and building trust in the generated documentation.
- Smaller LLMs Can Shine: AutoDoc's architecture allows smaller, open-source LLMs to achieve performance comparable to larger proprietary models, democratizing the generation of high-quality documentation.
- Beyond Official Docs: A significant portion of the generated documentation contains knowledge not found in official sources, highlighting AutoDoc's value in supplementing existing resources.
Key Results
- Accuracy: AutoDoc generated API documents that were up to 77.7% more accurate than five comparison baselines.
- Uniqueness: The generated documents were 9.5% less duplicated than baselines.
- Novelty: AutoDoc uncovered 34.4% more knowledge that was not present in the official API documents.
- LLM Sensitivity: While larger LLMs produced better results, AutoDoc enabled smaller models like Mistral-7B-v0.3 to achieve comparable accuracy and uniqueness.
- User Study: 80% of participants preferred AutoDoc's generated API documents over those from baselines (GPT-4o ZSP and SISE) across helpfulness, comprehensiveness, conciseness, overview, and overall preference.
Limitations and Future Work
- Less Popular APIs: AutoDoc's performance (correctness and uniqueness) declines for less popular APIs, as there are fewer SO discussions to draw from. This can lead to increased reliance on the LLM's internal knowledge, potentially increasing hallucinations.
- Traceability: The summarization process can erase the direct link to the original SO post from which a knowledge snippet was extracted. Future work aims to develop a component for traceability to boost user trust.
- Overlap with Official Docs: While AutoDoc aims to provide novel information, some overlap with official documentation is unavoidable. Future work could involve a component to explicitly mark or filter shared knowledge, allowing users to focus on the unique insights.
- User Feedback: Users expressed a desire to see the original source post for generated snippets and for better handling of shared knowledge.
Summary updated