Identification of Regulatory Requirements Relevant to Business Processes: A Comparative Study on Generative AI, Embedding-based Ranking, Crowd and Expert-driven Methods

Catherine Sai, Shazia Sadiq, Lei Han, Gianluca Demartini, Stefanie Rinderle-MaarXiv 2024 · DOI · arXiv

Introduction

Ensuring that business processes—such as insurance claim processing or bank account opening—comply with an ever-evolving landscape of regulations is a significant operational challenge. Traditionally, this "compliance burden" requires legal experts to manually audit documents to determine which specific clauses apply to particular tasks. This study investigates the potential for automating this mapping using Generative AI (LLMs), state-of-the-art (SOTA) Natural Language Processing (NLP), and crowdsourcing, comparing them against the expert-driven gold standard.

The Challenge of Granularity

A core difficulty in regulatory mapping is the level of detail, or granularity. Organizations must identify relevance not just at a high business level, but down to specific sub-processes and individual tasks.

Mapping regulatory requirements to different levels of business process detail.

As shown in Figure 1, the study categorizes process detail into three levels:

  • Level 1 (Process): Broad end-to-end workflows.
  • Level 2 (Sub-process): Intermediate groupings of activities.
  • Level 3 (Task/Event): Individual atomic steps.

Furthermore, the research distinguishes between Compliance Relevance (mandatory actions) and Informative Relevance (contextual background), adding another layer of complexity to automated classification.

Methodology

The researchers benchmarked four distinct approaches to identifying regulatory relevance:

  1. Expert Analysis: Manual labeling by legal and domain experts, serving as the "Gold Standard."
  2. SOTA NLP (Legal Information Retrieval): Two pipelines were tested: a lexical approach (BM25 ranking with Cross-Encoder re-ranking) and a semantic approach (Bi-Encoder sentence embeddings with Cross-Encoder re-ranking).
  3. Generative AI (GPT-4): A zero-shot approach utilizing engineered prompts that provided business context and process descriptions alongside regulatory text.
  4. Crowd Study: Utilizing Amazon MTurk workers with rigorous quality filters, such as "attention traps" and semantic consistency checks.
Comparison of the four identification methods across automation and transparency axes.

Figure 3 illustrates the trade-offs between these methods. While Expert Analysis and Crowd Studies are manual, SOTA NLP and LLMs offer automation at the cost of varying levels of transparency and reproducibility.

Experimental Setup and Datasets

The study introduced a new benchmark dataset based on two real-world use cases:

  • Insurance: Travel insurance claims handling involving 7 Australian regulatory documents (489 passages).
  • Banking: A "Know Your Customer" (KYC) process involving 311 text passages.

Passages were categorized into three groups to test fine-grained discrimination: Group A (process-relevant), Group B (business-relevant but process-irrelevant), and Group C (entirely irrelevant).

Key Results and Insights

The comparative analysis yielded several critical findings:

  • GPT-4 Dominance in Recall: GPT-4 proved to be an exceptional "first-pass" filter. For Level 1 relevance, it achieved a recall of 1.0 in the insurance case and 0.97 in banking. However, its precision was lower as it frequently flagged "informative" or borderline texts as relevant.
  • The Granularity Penalty: All methods saw performance declines as granularity increased. Mapping a legal clause to a specific Level 3 task (e.g., "Verify identity") is significantly harder than mapping it to a general process due to reduced semantic overlap.
  • NLP Stability: The Bi-Encoder + Cross-Encoder method was the most reproducible and transparent, though its recall was lower (approximately 0.41 for Level 1) compared to the LLM.
  • Crowd Effectiveness: Interestingly, crowd workers were highly effective at Level 2 (sub-process) relevance, achieving 0.80 recall after quality filtering, outperforming traditional NLP methods.

Conclusion

The study concludes that while GPT-4 is a powerful tool for ensuring no regulatory requirements are missed (high recall), a hybrid approach remains necessary. The authors recommend a tiered strategy: use GPT-4 + Experts when speed and high recall are priorities, and SOTA NLP + Experts for high-impact scenarios where transparency and reproducibility are paramount. Future work should focus on integrating these automated links directly into Business Process Model and Notation (BPMN) tools to assist managers during the design phase.

Summary updated

Abstract

Organizations face the challenge of ensuring compliance with an increasing amount of requirements from various regulatory documents. Which requirements are relevant depends on aspects such as the geographic location of the organization, its domain, size, and business processes. Considering these contextual factors, as a first step, relevant documents (e.g., laws, regulations, directives, policies) are identified, followed by a more detailed analysis of which parts of the identified documents are relevant for which step of a given business process. Nowadays the identification of regulatory requirements relevant to business processes is mostly done manually by domain and legal experts, posing a tremendous effort on them, especially for a large number of regulatory documents which might frequently change. Hence, this work examines how legal and domain experts can be assisted in the assessment of relevant requirements. For this, we compare an embedding-based NLP ranking method, a generative AI method using GPT-4, and a crowdsourced method with the purely manual method of creating relevancy labels by experts. The proposed methods are evaluated based on two case studies: an Australian insurance case created with domain experts and a global banking use case, adapted from SAP Signavio's workflow example of an international guideline. A gold standard is created for both BPMN2.0 processes and matched to real-world textual requirements from multiple regulatory documents. The evaluation and discussion provide insights into strengths and weaknesses of each method regarding applicability, automation, transparency, and reproducibility and provide guidelines on which method combinations will maximize benefits for given characteristics such as process usage, impact, and dynamics of an application scenario.
BibTeX
@misc{sai2024identification,
  author = {Catherine Sai and Shazia Sadiq and Lei Han and Gianluca Demartini and Stefanie Rinderle-Ma},
  title = {{Identification of Regulatory Requirements Relevant to Business Processes: A Comparative Study on Generative AI, Embedding-based Ranking, Crowd and Expert-driven Methods}},
  year = {2024},
  eprint = {2401.02986},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.02986},
}
Figure