TL;DR
This paper introduces a new NLP task: extracting software functionalities directly from README files. They've created a new dataset, FuncRead, and demonstrated that smaller, fine-tuned code-centric language models significantly outperform large, general-purpose models like ChatGPT and Bard on this task.
What Problem Does This Paper Address?
As software development increasingly adopts microservices architecture, a key challenge is decomposing monolithic applications into smaller, independent services. This requires accurately identifying the distinct functionalities of the existing codebase. While README files often contain descriptions of these functionalities, extracting them automatically and reliably is difficult. Existing text summarization or question-answering models aren't well-suited for this specific task because functionalities can be scattered, inconsistently phrased, and intermingled with other information like licensing or usage instructions. The authors aim to bridge this gap by developing a method to automatically extract these functionalities, which can then inform microservice refactoring efforts.
Methods
The core of the paper is the introduction of a novel task: functionality extraction from Git README files. This is framed as a text-to-text generation problem. The authors developed a new dataset, FuncRead, by collecting README files from public GitHub repositories with permissible licenses. They then annotated these READMEs to extract functionalities in both extractive (direct text spans) and abstractive (rephrased in the annotator's own words) formats.
For evaluating models, they employed bipartite matching (one-to-one, many-to-one, and weighted many-to-one) to align generated functionalities with the gold-standard annotations. This allowed them to calculate precision, recall, and F1 scores. They also used ROUGE and BERTScore to assess lexical and semantic similarity, respectively.
The experimental setup involved comparing large, out-of-the-box models like ChatGPT and Bard against smaller, fine-tuned models, including variants of StarCoderbase, Phi-2, Llama2, and CodeLlama. The fine-tuning process involved pre-processing the README data and appending a task-specific prompt.
Datasets
The paper introduces a new dataset called FuncRead. This dataset comprises 2101 human-annotated GitHub README files. The annotations include both extractive (direct text segments) and abstractive (rephrased descriptions) functionalities. The dataset is designed to facilitate research in functionality extraction from READMEs and aid in microservice refactoring.
Key Insights and Ideas
- Fine-tuned code models excel: Smaller, fine-tuned language models, particularly those with a code-centric architecture (like CodeLlama), significantly outperform large, general-purpose models (ChatGPT, Bard) on this specific task. This suggests that specialized training on code-related data is crucial for understanding software documentation.
- READMEs are a rich source: Despite their often unstructured nature, README files contain valuable information about software functionalities that can be reliably extracted with the right approach.
- Abstractive vs. Extractive: The distinction between extractive and abstractive annotations is important. While extractive captures direct mentions, abstractive allows for more concise and human-readable summaries of functionalities, which is valuable for downstream applications.
- Bipartite matching is key for evaluation: The use of bipartite matching, especially weighted many-to-one, is a sophisticated way to evaluate the quality of generated functionalities, accounting for cases where multiple gold functionalities might map to a single generated one, or vice-versa.
Key Results
The paper reports that fine-tuned models consistently outperform ChatGPT and Bard across various metrics. For instance, the best fine-tuned 7 Billion CodeLlama model achieved a 70% gain in F₁ score compared to ChatGPT and a 20% gain compared to Bard in one-to-one bipartite matching. The fine-tuned models also showed higher ROUGE and BERTScore, indicating better lexical and semantic similarity with the ground truth. The study found that while fine-tuned models sometimes combine multiple functionalities into a single sentence, this did not significantly hinder their performance, especially in many-to-one matching scenarios.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Size: The current dataset of 2101 samples might not cover all possible ways functionalities can be described. They suggest increasing the dataset size for future work.
- Human Annotation Errors: Despite validation, some human errors (missed or unwanted functionalities) were present in the annotations. Expanding validation efforts could mitigate this.
- Long READMEs: The 2048-token limit of current models poses a challenge for processing very long README files.
- Ambiguity in Functionality Definition: Defining a "ground truth" set of functionalities can be subjective, as different annotators might perceive them differently. Educating annotators with more examples is proposed.
Future work could involve addressing these limitations by expanding the dataset, exploring models with longer context windows, and refining annotation guidelines.
Summary updated