TL;DR
This paper explores how well large language models (LLMs) can understand and reason about complex legal texts in civil procedure. They found that while fine-tuned models like Legal-BERT showed promise, zero-shot LLMs, particularly Bing's Copilot, demonstrated surprisingly strong performance on this challenging task, highlighting the potential of LLMs for legal reasoning.
What Problem Does This Paper Address?
Aspiring lawyers need to master complex legal reasoning, including analogy and identifying contradictions. While NLP models have advanced, there's a lack of challenging tasks that specifically test argumentation in legal contexts. This paper tackles the difficulty of processing lengthy and intricate legal data, aiming to assess LLMs' ability to determine if a given answer accurately addresses a legal question based on provided explanations.
Methods
The researchers employed a multi-phase experimental approach:
- Long-Context Models: They first evaluated Longformer and Big Bird, models known for their ability to handle significantly larger input token sizes (up to 4096 tokens) compared to standard BERT models.
- Legal-Specific Pre-trained Models: Next, they assessed the performance of Legal-RoBERTa and Legal-XLM-ROBERTa, models pre-trained on extensive legal corpora, to gauge their efficacy in legal text analysis.
- Zero-Shot LLMs: Finally, they tested three popular large language models – GPT 3.5, Gemini, and Bing's Copilot – using a zero-shot methodology. This involved crafting prompts to elicit a "yes" or "no" answer regarding the correctness of a proposed legal answer based on an explanation. Prompt engineering was crucial here to refine the models' outputs.
To address class imbalance in the dataset, they utilized the Focal Loss function, which down-weights well-classified examples and focuses training on harder instances.
Datasets
The primary dataset for this study was sourced from "The Glannon Guide To Civil Procedure" by Joseph Glannon. Each sample in the dataset consists of a question, a proposed answer, and a detailed explanation. The dataset was split into 80% for training, 10% for validation, and 10% for testing, resulting in a total of 848 entries. Notably, the "Analysis" and "Complete Analysis" columns were absent in the test data.
Key Insights and Ideas
- Zero-Shot LLMs Show Remarkable Potential: The zero-shot performance of LLMs, especially Bing's Copilot, was a significant finding. This suggests that even without explicit task-specific fine-tuning, these models can leverage their broad knowledge and reasoning capabilities to tackle complex legal questions.
- Prompt Engineering is Crucial: The effectiveness of the zero-shot approach was heavily influenced by prompt engineering. Techniques like asking for "step-by-step" reasoning or detailed explanations of the inference process significantly improved model performance.
- Token Size Limitations Persist: Despite advancements, even models designed for longer contexts (like Copilot's 4000-token limit) can struggle with extremely lengthy legal documents, indicating ongoing challenges in processing the full scope of legal texts.
- Focal Loss Aids Imbalanced Data: The use of Focal Loss proved effective in mitigating class imbalance, a common issue in legal datasets, and improved the performance of models on minority classes.
Key Results
The study achieved a highest F₁ score of 64% with the zero-shot approach.
- Pre-trained Models: In the initial phases, fine-tuned models like Legal-BERT served as a baseline. Table 2 shows that Longformer and Big Bird achieved an accuracy of 0.79 and an F₁ of 0.44, while Legal-RoBERTa and Legal-XLM-ROBERTa had similar accuracies (0.79 and 0.78) but slightly lower F₁ scores (0.44 and 0.43). Legal-BERT itself had an accuracy of 0.75 and an F₁ of 0.58.
- Zero-Shot LLMs (Test Set): Bing's Copilot emerged as the top performer among the zero-shot models, surpassing expectations with an accuracy of 0.67 and an F₁ of 0.64. GPT 3.5 achieved an accuracy of 0.69 and an F₁ of 0.59, while Gemini had an accuracy of 0.44 and an F₁ of 0.44.
Limitations and Future Work
Limitations:
- Token Size Constraints: Even advanced models like Bing's Copilot had input token limitations that hindered processing of the longest legal samples.
- Prompt Completeness: Achieving optimal results was challenging due to the need for comprehensive prompts, which were sometimes constrained by token limits.
- Underlying Reasons for Performance: The paper notes that while Legal-BERT showed the most promising result among fine-tuned models, the exact reasons for this, and why other models struggled, require further analysis.
Future Work:
- Explore More LLMs: Investigating a wider range of LLMs could reveal further capabilities, especially in summarizing lengthy datasets.
- Data Summarization: Developing techniques to summarize legal texts could help overcome token limitations and improve model performance.
- Collaborative Approaches: Exploring methods like "round table" techniques, where multiple LLM agents discuss and reach a consensus, could enhance reasoning.
- Hyperparameter Tuning and Prompt Refinement: Further optimization of hyperparameters and iterative refinement of prompts are suggested to boost performance.
- Multi-Model Approaches: Investigating the effectiveness of combining multiple models could lead to more robust solutions.
Summary updated