An automated framework for the extraction of semantic legal metadata from legal texts

Amin Sleimi, Nicolas Sannier, Mehrdad Sabetzadeh, Lionel C. Briand, Marcello Ceci, John Dann

TL;DR

This paper introduces an automated framework for extracting semantic metadata from legal texts, a crucial step for understanding and analyzing legal requirements. By reconciling existing metadata typologies and developing NLP and ML-based extraction rules, the authors achieve high accuracy in identifying key legal concepts, paving the way for more efficient legal analysis and compliance management.

What Problem Does This Paper Address?

Manually annotating large legal corpora with semantic metadata is prohibitively expensive and time-consuming. While the Requirements Engineering (RE) community recognizes the importance of this metadata for understanding legal provisions, there's a lack of a harmonized view on useful metadata types. Furthermore, existing automated approaches for metadata extraction are scarce and don't fully leverage the potential of modern Natural Language Processing (NLP) and Machine Learning (ML) techniques, particularly for capturing the nuanced semantics of legal language. This paper aims to bridge this gap by proposing a unified conceptual model and an automated extraction framework.

Methods

The authors developed a multi-stage approach:

  1. Conceptual Model Development: They conducted a thorough review and reconciliation of semantic legal metadata types proposed in the RE literature to create a harmonized conceptual model. This model defines six statement-level concepts (e.g., Obligation, Permission) and 18 phrase-level concepts (e.g., Actor, Condition, Sanction).
  2. NLP-based Extraction Rules: Using a qualitative study on traffic laws, they devised NLP-based rules for extracting phrase-level metadata. These rules primarily leverage constituency and dependency parsing techniques.
  3. ML-based Actor Role Classification: Recognizing the difficulty in rule-based classification of actor roles (agent, target, auxiliary party) due to legal language subtleties, they developed an ML classifier. This classifier uses a combination of sentence-level, actor's neighborhood, and main-verb relationship features.
  4. Statement-level Extraction Rules: Rules were developed to classify entire statements based on identified phrase-level metadata and specific markers.

Datasets

  • Qualitative Study Data: 200 randomly selected statements from Luxembourgish traffic laws were used for the initial qualitative study and to derive extraction rules.
  • Case Study 1 (CS1): 150 randomly selected legal statements from Luxembourgish traffic laws were used for evaluation. This dataset builds upon the 200 statements from the qualitative study.
  • Case Study 2 (CS2): 200 randomly selected statements from five different Luxembourgish legislative domains (Code of Commerce, Penal Code, Health Code, Labor Code, and Code for the Environment) were used to evaluate the generalizability of the approach.

Key Insights and Ideas

  • Harmonization is Key: The paper highlights the significant challenge and importance of reconciling diverse metadata typologies from the RE literature into a single, comprehensive conceptual model.
  • NLP for Structure, ML for Nuance: While NLP techniques (parsing) are effective for identifying the structure and spans of legal concepts, ML is crucial for disambiguating nuanced roles like "agent," "target," and "auxiliary party," which are difficult to capture with explicit rules.
  • Context is King (and Hard to Automate): The study reveals that implicit information from preceding statements significantly impacts classification accuracy, especially for statement-level concepts like prohibition and obligation. This highlights a limitation of statement-isolation approaches.
  • Marker Saturation: The authors observe that with sufficient qualitative data and domain-specific lexicons, the set of markers for many metadata types can "saturate" quickly, meaning new data yields diminishing returns in discovering new markers.
  • Domain-Specific vs. General Markers: While general markers show good cross-domain applicability, the study indicates that domain-specific nuances and polysemous terms pose challenges, suggesting the need for domain-specific glossaries.

Key Results

The framework achieved high accuracy in its evaluations:

  • Case Study 1 (Traffic Laws):
    • Overall phrase-level metadata extraction: 97.2% precision and 94.9% recall.
  • Actor role classification: 90.2% accuracy for agent, 79.0% for target, and 78.3% for auxiliary party (using Random Forest).
  • Case Study 2 (Multi-domain):
    • Overall phrase-level metadata extraction: 82.4% precision and 92.4% recall. The decrease in precision is attributed to longer, more complex statements and domain differences.

Limitations and Future Work

  • Statement Isolation: The current approach treats each statement independently, missing contextual information from preceding statements, which impacts accuracy for certain classifications.
  • Parser Limitations: The accuracy of NLP parsers decreases with statement length, a common issue in legal texts. Training parsers specifically on legal corpora is suggested.
  • Ambiguity and Polysemy: Ambiguous markers and polysemous terms pose challenges, requiring domain-specific glossaries or more advanced disambiguation techniques.
  • Constraint and Result: The study did not yield enough data to derive extraction rules for "constraint" and "result" metadata types.
  • Future Work: The authors plan to explore relationships between metadata types, investigate the usefulness of the extracted metadata in practical use cases (like query systems), and improve cross-reference resolution. They also aim to develop more nuanced markers and potentially domain-specific glossaries.

Summary updated

BibTeX
@article{sleimi2021automated,
  author = {Amin Sleimi and Nicolas Sannier and Mehrdad Sabetzadeh and Lionel C. Briand and Marcello Ceci and John Dann},
  title = {{An automated framework for the extraction of semantic legal metadata from legal texts}},
  journal = {Empir. Softw. Eng.},
  year = {2021},
  doi = {10.1007/s10664-020-09933-5},
}
Figure