TL;DR
Automotive systems are complex, and ensuring that stakeholder requirements are correctly linked to system requirements is crucial for safety and compliance. This paper introduces TVR, a novel approach using Large Language Models (LLMs) enhanced with retrieval-augmented generation (RAG) to automatically validate existing traceability links and recover missing ones, achieving high accuracy on real-world automotive data.
What Problem Does This Paper Address?
In the development of complex automotive systems, maintaining accurate traceability between high-level stakeholder requirements (what the user/customer needs) and detailed system requirements (how the system will meet those needs) is paramount. This linkage is essential for ensuring consistency, correctness, and regulatory compliance, especially in safety-critical domains. However, this process is often manual, labor-intensive, and prone to errors due to:
- Improper propagation of changes: When stakeholder requirements are updated, corresponding system requirements may not be, leading to outdated or invalid links.
- Human error in mapping: Manually linking requirements can lead to incorrect associations or missed links.
- Variations in requirement expression: Stakeholder and system requirements are often expressed using different terminology, abstraction levels, and sentence structures, making automated analysis challenging.
These issues can lead to inconsistencies, increased maintenance costs, and potentially safety risks. Existing automated approaches often struggle with the nuances of industrial requirements, especially when dealing with variations and the need to validate existing links rather than just discovering new ones.
Methods
The paper proposes TVR (Traceability Validation and Recovery), a retrieval-augmented generation (RAG) approach that leverages Large Language Models (LLMs). Unlike traditional RAG systems that retrieve external documents for knowledge, TVR retrieves previously validated requirement pairs (both correct and incorrect links) from a dataset. These retrieved pairs serve as "in-context demonstrations" to guide the LLM.
The TVR framework has two main components:
- Retriever: This component identifies the most similar requirement pairs from a training dataset that closely match the input test pair (a stakeholder requirement and a system requirement). It uses embeddings generated from concatenated requirement texts and a similarity-based retrieval mechanism (FAISS library with cosine similarity) to select representative valid and invalid examples.
- Generator: This component is an LLM (e.g., Claude 3.5 Sonnet) that receives the input requirement pair along with the retrieved examples. It is prompted to specifically focus on whether the message or signal mentioned in the stakeholder requirement is correctly covered by the system requirement. The LLM then outputs a "Yes" or "No" prediction.
For traceability recovery, TVR employs a three-step preprocessing pipeline to filter down the vast number of potential missing links before applying the validation process.
Datasets
The study uses a dataset of Diagnostic Trouble Code (DTC) requirements from an automotive industry partner. This dataset includes:
- 1,320 stakeholder requirements.
- 48 system requirements.
- 2,132 existing traceability links, which were manually validated by system engineers.
The dataset was annotated by two co-authors following industrial guidelines, with disagreements resolved through discussion with system engineers. Cohen's kappa was computed to ensure high annotation reliability (κ ≈ 0.962). The dataset contains both valid (1,913) and invalid (219) traceability links.
Key Insights and Ideas
- RAG for Reasoning, Not Just Knowledge: TVR demonstrates that RAG can be effectively used to provide reasoning examples to LLMs, rather than just external factual knowledge, to improve performance on tasks like traceability validation.
- In-Context Demonstrations are Key: Providing LLMs with examples of correct and incorrect traceability links significantly improves their ability to understand and apply the validation criteria, especially when dealing with domain-specific language and variations.
- Domain-Specific Nuances Matter: Traditional NLP and similarity-based methods struggle with the subtle differences in terminology and structure found in automotive DTC requirements. LLMs, when guided appropriately, can better capture these nuances.
- Complementary Strengths: The study found that TVR and retrieval-based methods have complementary error patterns, suggesting that hybrid approaches could further enhance performance.
Key Results
- Traceability Validation: TVR achieved a high accuracy of 98.87% and a Macro-F1 score of 96.72% on the validation task.
- Robustness: TVR demonstrated strong robustness to unseen requirement variations, maintaining an accuracy of 97.13%.
- Traceability Recovery: TVR successfully identified 502 missing traceability links with a correctness of 85.50%.
- LLM Performance: Claude 3.5 Sonnet was identified as the best-performing LLM for this task, especially when used with the TVR approach.
Limitations and Future Work
- Dataset Specificity: The study used a dataset from a single automotive project. While the principles are expected to generalize, further evaluation on diverse projects and domains is needed.
- LLM Limitations: Despite high accuracy, TVR still made errors, primarily due to LLMs occasionally considering irrelevant parts of requirements or misinterpreting trigger conditions as target messages.
- Future Work:
- Explore more constrained prompting strategies or explicitly identifying requirement components to reduce errors.
- Integrate retrieval-based similarity with LLM-based reasoning for hybrid approaches.
- Expand the retrieval database with more validated data to improve performance.
- Adapt TVR to other types of requirements and domains.
- Investigate the use of open-source models and datasets.
Summary updated