Using Frame Embeddings to Identify Semantically Related Software Requirements

Waad Alhoshan, Riza Batista-Navarro, Liping ZhaoREFSQ Workshops 2019 · OpenReview

Abstract

Identifying semantically related software requirements is a cornerstone of effective traceability and system organization. However, the inherent ambiguity of natural language often hinders automated approaches that rely on keyword matching. This paper presents a novel "frame-to-frame" (F2F) method that utilizes FrameNet semantic frames and domain-specific word embeddings to detect relatedness. By mapping requirements to underlying conceptual frames, the method achieves a peak F1 score of 86.36%, significantly outperforming standard baselines.

Introduction

In the early stages of software development, requirements are typically documented in unstructured natural language. As these documents scale, identifying related requirements—a process known as traceability—becomes a massive, error-prone manual task. The primary challenge lies in linguistic variability: two requirements might describe the same feature using entirely different vocabularies (e.g., "authorised users can view" vs. "accessing logs is restricted").

Standard Natural Language Processing (NLP) techniques often fail in this context because they rely on exact word matching or are trained on general-domain text like news articles. This research aims to bridge the gap by creating an automated way to measure semantic relatedness between requirements, even when the wording varies significantly.

Methodology

The authors propose a "frame-to-frame" (F2F) method leveraging FrameNet, a computational lexicon that groups words into "semantic frames" representing underlying concepts. The pipeline consists of three core components:

  1. Domain-Specific Word Embeddings: To capture software-specific context, a Word2Vec model was trained on a massive corpus of over 3 million mobile app reviews scraped from the Apple Store and Google Play.
  2. Frame Embeddings: Vector representations for entire frames were created by averaging the word vectors of all lexical units (words) that trigger a specific FrameNet frame.
  3. Requirement-to-Requirement Scoring: The system identifies all semantic frames evoked in a requirement statement. It then constructs a similarity matrix comparing every frame in Statement A to every frame in Statement B using cosine similarity. The average of this matrix provides the final relatedness score.
Examples of requirement pairs with their semantic frame annotations and calculated F2F relatedness scores.

Experimental Setup

The proposed method was evaluated using the SEM-REQ dataset, a newly introduced benchmark containing 1,770 pairs of software requirements. These pairs were drawn from the FN-RE corpus and manually labeled for semantic relatedness by three annotators, including a requirements engineer and an NLP expert, ensuring high inter-annotator agreement.

Key Results

The F2F method demonstrated superior performance compared to a baseline system (a standard Word2Vec model trained on 100 billion words from Google News).

  • Peak Performance: The F2F method achieved a peak F-score of 86.36% at a similarity threshold of 0.4.
  • Baseline Comparison: The baseline system only managed a peak F-score of 62.42% at a threshold of 0.1, with performance dropping off sharply as the threshold increased.
Performance comparison (F1 Score) between the F2F Method and the Baseline System across various thresholds.

Discussion and Insights

The success of the F2F method highlights several critical insights for automated requirements engineering:

  • Semantics over Syntax: By mapping words to FrameNet frames, the system successfully links requirements that share zero overlapping vocabulary but describe the same software event. For instance, it can link "registering a user" to "providing name and address."
  • Domain-Specific Training: General-purpose NLP models struggle with software engineering text. Training embeddings on app reviews provided the contextual nuance needed to understand software-specific terminology and user-centric actions.
  • Capturing Contextual Similarity: The method calculates the distance between different but related frames (e.g., Verification and Indicating). This allows the system to capture implicit dependencies between sequential system features, mimicking how a human engineer would group related tasks.

Conclusion and Future Work

While the approach is highly effective, it is currently limited by the coverage of the FrameNet lexicon; requirements using terminology not mapped to an existing frame may be missed. Future work will focus on integrating the F2F method into practical Requirements Engineering (RE) software tools. This integration would allow engineers to automatically analyze and trace semantic relatedness in real-world documents, facilitating the effortless organization of complex system features.

Summary updated

BibTeX
@inproceedings{alhoshan2019using,
  author = {Waad Alhoshan and Riza Batista-Navarro and Liping Zhao},
  title = {{Using Frame Embeddings to Identify Semantically Related Software Requirements}},
  booktitle = {REFSQ Workshops},
  year = {2019},
  url = {https://openreview.net/forum?id=Tdf2K33dyQ},
}
Figure