Abstract
Software development decisions are critical pieces of project knowledge that are often lost due to lack of formal documentation, a phenomenon known as "knowledge vaporization." This research proposes an automated multiclass classification approach to categorize decisions into five types: Design, Requirement, Management, Construction, and Testing. By analyzing developer mailing lists using a combination of feature selection and soft voting ensembles (SVE), the study identifies an optimal configuration achieving an F1-score of 0.727. This method offers a scalable way for stakeholders to retrieve relevant decision-making history from informal communication channels.
Introduction
During the software development life cycle, stakeholders make numerous decisions that shape the final product. However, these are rarely documented formally due to time and budget constraints. While this information exists within informal channels like mailing lists or issue trackers, manually extracting it is prohibitively difficult.
Previous research has largely focused on binary identification (is it a decision or not?) or specifically on design decisions. This paper addresses the gap in multiclass classification, sorting decisions into categories relevant to different stakeholders. For instance, a project manager may prioritize "Management" decisions, while developers focus on "Construction" or "Design" choices.
Methodology
The researchers evaluated 270 different configurations to identify the optimal pipeline for decision classification. The process involved:
- Preprocessing: Standard text cleaning, lemmatization, and stop-word removal.
- Feature Selection: The Chi-Squared ($CHI^2$) algorithm was used to rank terms by information gain, testing thresholds from 10% to 100% of available features.
- Feature Extraction: Three methods were compared: Bag of Words (BoW), TF-IDF, and Word2Vec (CBoW model).
- Classification: Four base learners—Naïve Bayes (NB), Logistic Regression (LR), Support Vector Machines (SVM), and Random Forest (RF)—were tested individually and in ensembles.
- Ensemble Approach: The study utilized Soft Voting Ensembles (SVE), which average probability scores from multiple base models to determine the final class.
Dataset and Quality Control
The study utilized a dataset of 844 manually labeled decisions from the Hibernate developer mailing list. The distribution included Design (314), Requirement (299), Management (109), Construction (99), and Testing (23) decisions. To ensure data quality, two researchers labeled the data independently, achieving a high Cohen’s Kappa of 0.856, with a third researcher resolving any discrepancies.
Results and Discussion
The experimental results highlighted several key insights regarding the classification of informal developer communication:
The Impact of Feature Selection
Feature selection proved to be a critical performance booster. By filtering out "noise" such as email signatures and irrelevant technical jargon, models were able to focus on highly predictive domain-specific keywords (e.g., "session" for Design decisions or "release" for Management).
As shown in the results, applying feature selection improved F1-scores across all extraction methods, with a particularly dramatic 25.9% average improvement for Word2Vec.
Model Performance
The "winning" configuration was BoW + 50% Feature Selection + SVE2 (NB, LR, SVM), yielding a weighted F1-score of 0.727. Interestingly, simpler models like BoW and TF-IDF outperformed the more complex Word2Vec. This is likely due to the relatively small size of the labeled dataset (844 instances), as Word2Vec typically requires much larger corpora to learn effective semantic relationships.
Conclusion and Future Work
While the proposed ensemble method shows promise, it faces limitations regarding generalizability, as the specific keywords identified are tied to the Hibernate project's domain. Additionally, the data imbalance—specifically the scarcity of Testing decisions—presents a challenge for model training. Future work will focus on expanding the dataset across multiple open-source projects and exploring deep learning architectures like FastText and Text-CNN to further improve classification accuracy on larger datasets.
Summary updated