TL;DR
This paper explores how Natural Language Processing (NLP) and Large Language Models (LLMs) can automate parts of the complex and time-consuming legal due diligence process. They found that a hierarchical sentence extraction approach, combined with careful handling of imbalanced data, performs best, and that LLMs like GPT-4 show promise as helpful assistants for legal professionals.
What Problem Does This Paper Address?
Due diligence is a critical, yet notoriously tedious and expensive, legal process in mergers and acquisitions (M&A). It involves sifting through vast amounts of legal documents to identify potential risks. Currently, this is done manually by legal experts, which is time-consuming and prone to errors, especially given the sheer volume and length of legal documents. While NLP has seen significant advancements, its application to due diligence has been limited, primarily due to the lack of publicly available, large-scale datasets and the challenge of processing extremely long legal texts. This paper aims to bridge this gap by investigating efficient NLP architectures and the potential of LLMs for this specific legal task.
Methods
The researchers explored three main NLP architectures for due diligence:
- Single-Sentence Classification: This is the most basic approach, where each sentence is analyzed independently to determine if it's "relevant" or "non-relevant." They fine-tuned standard models like BERT and a legal-specific BERT (LegalBERT) for this.
- Context-Aware Sentence Classification: This method improves upon the first by considering a sentence's surrounding context (neighboring sentences) to better understand its relevance.
- Hierarchical Sentence Extraction: This is the most sophisticated approach, designed to handle the long nature of legal documents. It uses a two-stage process: a sentence-level encoder to convert sentences into vectors, followed by a document-level encoder that processes these vectors to sequentially classify sentences. This architecture is inspired by extractive summarization techniques but adapted for legal documents.
To address the significant class imbalance in legal documents (very few relevant sentences compared to many irrelevant ones), they developed an IMBALANCED SAMPLER strategy and used a weighted binary cross-entropy loss function. For LLM evaluation, they used GPT-4 in a few-shot learning setting, simplifying the task to paragraph classification.
Datasets
The primary dataset used is the KIRA dataset (Roegiest et al., 2018). This dataset contains real-world legal documents across 50 topics, with sentences annotated for relevance. A key characteristic of this dataset is its extreme imbalance: documents are very long (averaging over 3,000 sentences), but only a handful (around 4.8 on average) are marked as relevant. Due to computational constraints, the study focused on five selected topics from the KIRA dataset.
Key Insights and Ideas
- Length Matters: The inherent lengthiness of legal documents poses a significant challenge for standard NLP models, often exceeding their token limits and hindering performance.
- Context is Tricky: While considering sentence context can be beneficial, it doesn't automatically guarantee better performance, especially if the context window itself is limited by model architecture.
- Legal-Specific Models Aren't Always Superior: Legal-specific PLMs (like LegalBERT) did not consistently outperform general-purpose PLMs (like BERT) on this task, suggesting that domain-specific pre-training doesn't automatically translate to better performance on all downstream legal tasks.
- Hierarchical Approach Wins: The hierarchical sentence extraction architecture proved to be the most effective and practical for due diligence, outperforming simpler models and even the KIRA baseline.
- LLMs as Assistants: GPT-4 shows promise as a valuable tool for lawyers, demonstrating high recall rates that can significantly reduce the manual review workload, even with a simplified task.
- Don't Underestimate Traditional Methods: The study found that traditional models like Bi-LSTM, when properly tuned and combined with effective data handling strategies, can still achieve competitive results, highlighting the value of established techniques.
Key Results
- The hierarchical sentence extraction model consistently achieved better or comparable performance (especially in recall) compared to the KIRA baseline and other tested architectures across the evaluated topics.
- For the single-sentence classification task, both BERT and LegalBERT performed similarly or worse than the KIRA baseline, indicating that sentence-level context is crucial.
- In context-aware classification, performance was similar or worse than single-sentence classification, likely due to length limitations.
- When evaluating GPT-4 on topic 1243 with a simplified task, it achieved comparable F1 scores to the KIRA baseline, but with a notably high recall rate, suggesting it can effectively identify most relevant information, thereby reducing lawyer effort. The performance improved with more few-shot examples.
Limitations and Future Work
- Limited Scope: Due to computational constraints, the study only analyzed 10% of the KIRA dataset (five topics). Investigating a broader range of topics could provide more robust evidence.
- Dataset Size: The KIRA dataset, while valuable, is limited in size and scope.
- Future Work: The authors suggest combining LLMs for initial relevant sentence identification with high-precision models for final filtering. Further research could explore more advanced LLM prompting strategies and investigate broader applications of NLP in due diligence. They also note that traditional approaches still hold significant value and can be practically employed.
Summary updated