MLR-predictor: a versatile and efficient computational framework for multi-label requirements classification

Summra Saleem, Muhammad Nabeel Asim, Ludger van Elst, Markus Junker, Andreas DengelFrontiers Artif. Intell 2024 · DOI

TL;DR

This paper introduces MLR-Predictor, a novel computational framework designed for multi-label requirements classification. By combining the OkapiBM25 text representation method with a Label Powerset data transformation and a Logistic Regression classifier, MLR-Predictor significantly outperforms existing state-of-the-art methods on benchmark datasets, offering a more robust and efficient solution for understanding software requirements.

What Problem Does This Paper Address?

Developing successful software hinges on accurately understanding user needs, which is captured in software requirements. Classifying these requirements into predefined categories is crucial for identifying project risks and ensuring milestones are met. While many methods exist for binary or multi-class classification, multi-label classification (where a single requirement can belong to multiple categories simultaneously) has been a significant challenge. Existing multi-label predictors often suffer from poor performance, making them impractical. This paper aims to address this gap by developing a versatile and efficient framework for multi-label requirements classification that achieves superior predictive performance.

Methods

The core of MLR-Predictor lies in its innovative approach to transforming text into meaningful representations and then classifying them. Key components include:

  • Representation Learning: The framework utilizes the OkapiBM25 model, a technique traditionally used in information retrieval, to transform requirements text into statistical vectors. This method is novel in its application to text classification and is highlighted for its ability to handle variations in requirement length and assign appropriate weights to words. It also incorporates pre-trained word embeddings like word2vec and FastText.
  • Data Transformation: MLR-Predictor converts multi-label classification problems into more manageable multi-class or binary classification tasks. It employs three widely used methods: Binary Relevance (BR), Label Powerset (LP), and Classifier Chain (CC). The paper emphasizes LP's strength in preserving label correlations.
  • Classification: The framework integrates nine traditional machine learning classifiers, including Logistic Regression (LR), Support Vector Machines (SVM), and Random Forests (RF).
  • Algorithm Adaptation: For direct multi-label classification, MLR-Predictor incorporates four adapted algorithms: Multi-label k-Nearest Neighbors (MLkNN), Binary Relevance k-Nearest Neighbors (BRkNN), and Multi-Label Hierarchical ARAM Neural Network (MLARAM).
  • Optimization: A Particle Swarm Optimizer (PSO) is used to find optimal hyper-parameters for the various components, significantly enhancing predictive pipeline performance.

Datasets

The proposed MLR-Predictor framework was evaluated on three public benchmark datasets:

  • PROMISE: Contains 792 requirements annotated with 15 different classes.
  • EHR-Binary: A version of the Electronic Health Records dataset with 5,722 requirements annotated into three classes: functional, non-functional, and both.
  • EHR-Multiclass: Another version of the EHR dataset with 5,722 requirements annotated into 15 different classes.

The paper also mentions a case study using customer reviews for software products, demonstrating the framework's generalizability.

Key Insights and Ideas

  • OkapiBM25's Superiority: The OkapiBM25 model proves to be a more effective text representation technique than traditional TF-IDF and even pre-trained word embeddings (word2vec, FastText) for requirements classification, especially when dealing with varying requirement lengths.
  • Label Powerset's Effectiveness: The Label Powerset (LP) data transformation method is particularly effective for multi-label classification as it preserves label correlations by treating unique label combinations as distinct classes.
  • PSO for Hyper-parameter Tuning: Optimizing hyper-parameters using Particle Swarm Optimization (PSO) significantly boosts the performance of predictive pipelines, demonstrating the importance of fine-tuning.
  • Generic Predictors are Feasible: The research shows that it is possible to develop generic multi-label requirements classification predictive pipelines that perform well across different datasets.
  • Deep Learning Limitations: Despite the advancements in deep learning, generic pre-trained word embeddings often fail to significantly improve performance on requirements classification tasks, likely due to the specialized nature of requirements data and the limited size of available datasets.

Key Results

The MLR-Predictor framework demonstrated superior performance compared to a large number of existing methods:

  • Outperforms 123 ML and 9 DL pipelines: The proposed framework significantly outperformed 123 machine learning and 9 deep learning predictive pipelines.
  • State-of-the-Art Improvement:
    • Achieved a 13% improvement in macro F1-measure on the PROMISE dataset compared to the state-of-the-art.
  • Showed a 1% improvement on the EHR-binary dataset.
  • Achieved a 2.5% improvement on the EHR-multiclass dataset.
  • Case Study Success: In a case study classifying software customer reviews, MLR-Predictor outperformed the state-of-the-art BERT language model by a 1.4% F-1 score.
  • Best Performing Pipeline: The combination of OkapiBM25 representation, Label Powerset data transformation, and Logistic Regression classifier (OkapiBM25-LP-LR) emerged as the top-performing pipeline across all benchmark datasets.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Limited Training Data: The study's reliance on relatively small requirements datasets, especially for multi-label scenarios, can limit the performance of deep learning models.
  • Generic Word Embeddings: The use of pre-trained word embeddings generated from generic text corpora may not fully capture the nuances of requirements-specific language.
  • Exploration of Embeddings: Future work could explore other word embedding techniques beyond word2vec and FastText, such as deepwalk, graph representation, and node2vec.
  • Large Language Models (LLMs): The potential of LLMs for requirements classification remains largely unexplored and is suggested as a future research avenue.
  • Robustness and Adversarial Attacks: Investigating the framework's robustness against adversarial attacks is crucial for real-world applications.
  • Time and Space Complexity: Further analysis of the MLR framework's time and space complexity is recommended.

Summary updated

BibTeX
@article{saleem2024mlr,
  author = {Summra Saleem and Muhammad Nabeel Asim and Ludger van Elst and Markus Junker and Andreas Dengel},
  title = {{MLR-predictor: a versatile and efficient computational framework for multi-label requirements classification}},
  journal = {Frontiers Artif. Intell.},
  year = {2024},
  doi = {10.3389/frai.2024.1481581},
}
Figure