TL;DR
This paper introduces two novel approaches, Kashif (a classifier using Sentence Transformers) and RICE_LRT (a prompt-based method for Large Language Models), to automate the challenging task of tracing software requirements to legal regulations. Their findings demonstrate that while Kashif offers significant improvements over existing methods on a standard dataset, RICE_LRT, powered by an engineered prompt for LLMs, shows superior performance on more complex, real-world legal datasets like GDPR, highlighting the potential of LLMs for this domain.
What Problem Does This Paper Address?
Ensuring software compliance with an ever-growing body of legal regulations (like GDPR) is crucial for public safety and ethical development. A fundamental step in this process is "requirements traceability," which involves linking software requirements to specific legal provisions. This task is currently manual, time-consuming, and prone to errors, especially for complex systems with hundreds of requirements. The inherent "language discrepancy" between technical software requirements and legal jargon, coupled with limited annotated data, makes automated solutions difficult. This paper aims to bridge this gap by developing and evaluating automated methods for legal requirements traceability (LRT).
Methods
The paper proposes two main approaches:
- Kashif: This is a classifier that leverages Sentence Transformers (STs). STs are pre-trained language models optimized for understanding sentence-level semantics. Kashif uses cosine similarity between the embeddings of requirements and legal provisions to predict trace links. It involves training and fine-tuning the ST model on a labeled dataset.
- RICE_LRT: This approach utilizes Large Language Models (LLMs), specifically GPT-40, with a structured prompt engineering framework called RICE. The prompt is carefully designed to guide the LLM to act as a requirements analyst, identify trace links, and provide rationales. This method focuses on few-shot learning, where a few examples are provided to the LLM to demonstrate the task.
Datasets
- HIPAA Dataset: A publicly available dataset used for initial evaluation. It contains 10 "shall" requirements documents from the healthcare domain, traced against 10 provisions of the Health Insurance Privacy and Portability Act (HIPAA). It includes 1,891 requirements with 243 trace links.
- GDPR Dataset: A newly created dataset for evaluating performance on more complex scenarios. It comprises 310 requirements covering diverse domains and types, traced against 26 provisions of the General Data Protection Regulation (GDPR). This dataset was manually curated by two annotators with extensive RE experience.
Key Insights and Ideas
- Language Discrepancy is a Major Hurdle: The significant difference in vocabulary, style, and abstraction between legal texts and software requirements is a primary challenge for automated traceability.
- LLMs with Engineered Prompts Shine in Complex Legal Domains: While Kashif (ST-based) performs well on simpler datasets, RICE_LRT, using a carefully crafted prompt for LLMs, significantly outperforms it and other baselines on the more complex GDPR dataset. This suggests that LLMs, when guided effectively, can better handle the nuances of legal language.
- Fine-tuning is Crucial for Classifier-Based Approaches: Kashif's performance on the HIPAA dataset demonstrates the benefit of fine-tuning ST models on domain-specific data.
- Prompt Engineering Matters: The RICE framework and the specific prompt design for RICE_LRT were key to its success, showing that how you ask an LLM is as important as the LLM itself.
- Recall Over Precision for Analysts: The study suggests that for human analysts, having a slightly higher number of false positives (requiring filtering) is more manageable than missing true trace links, especially in legal contexts.
Key Results
- Kashif on HIPAA: Achieved an F2 score of approximately 63%, outperforming the best baseline by 34 percentage points. It also achieved the highest MAP (0.81) and AUC (0.93) among classifiers.
- Kashif on GDPR: Performance dropped significantly, yielding only 15% recall, indicating limitations in generalizing to more complex legal domains without further adaptation.
- RICE_LRT on GDPR: Achieved an average recall of 84% and an F2 score of 61%, outperforming Kashif by 34 pp in F2 score and significantly improving over baseline prompts. It also demonstrated a good balance between recall (84%) and cost (9.5% of provisions retrieved).
- Prompt Baselines: Simpler prompts (like P1, P2, P3_1, P3_2) showed significantly lower performance compared to RICE_LRT on the GDPR dataset.
Limitations and Future Work
- External Validity: While evaluated on two datasets (HIPAA and GDPR) and different requirement types, further studies with more diverse requirements documents and other regulations are needed to improve external validity. The evaluation was also specific to GPT-40, and results with other LLMs might differ.
- Human-in-the-Loop Study: The authors plan to conduct a human-in-the-loop study with domain experts to investigate LLM applicability in the LRT context further.
- Domain-Specific Knowledge: Future work aims to enhance LLM performance by incorporating domain-specific knowledge to better address terminology and contextual gaps between regulatory texts and technical requirements.
- False Positives: While RICE_LRT reduces manual effort, the occurrence of false positives still requires investigation and potential filtering by human analysts.
Summary updated