TL;DR
This paper introduces LEXTIME, the first benchmark dataset specifically designed to evaluate how well Large Language Models (LLMs) can understand temporal ordering of events in legal texts. They found that LLMs perform better on legal texts than narrative ones, but still struggle with the unique linguistic complexities of law, suggesting a need for domain-specific modeling.
What Problem Does This Paper Address?
Understanding the sequence of events is crucial for many legal tasks, from analyzing case law to ensuring compliance. While LLMs have shown promise in various NLP tasks, their ability to accurately reconstruct event timelines in specialized domains like law remains largely unexplored. Existing benchmarks for temporal reasoning often focus on general narrative texts, leaving a gap in understanding how LLMs handle the intricate language, complex sentence structures, and specialized vocabulary found in legal documents. This paper aims to bridge that gap by creating a dedicated benchmark and evaluating LLMs' performance on this challenging task.
Methods
The authors developed LEXTIME, a dataset of 512 instances extracted from U.S. Federal Complaints. Each instance includes a context paragraph, a pair of events (either explicit-explicit or explicit-implicit), and a query asking about their temporal relationship, with a binary "yes/no" answer. They employed a prompt-chaining approach with Mistral-Large-Instruct-2407 to pre-label instances, followed by manual review and correction. For evaluation, they used standard prompting (zero-shot, one-shot, few-shot) and Chain-of-Thought (CoT) prompting. They also compared LEXTIME's performance against the TRACIE dataset, a benchmark for temporal ordering in narrative texts.
Datasets
- LEXTIME: A newly introduced benchmark dataset consisting of 512 instances derived from U.S. Federal Complaints. It focuses on temporal event ordering in legal language, featuring explicit and implicit events.
- TRACIE: Used as a comparison dataset, TRACIE consists of short stories and implicit events, curated for temporal ordering tasks. The authors adapted TRACIE instances to match LEXTIME's query format for fair comparison.
Key Insights and Ideas
- Legal Language is a Distinct Genre: Legal texts exhibit unique lexical, syntactic, and discourse features that differentiate them from narrative texts, posing specific challenges for temporal reasoning.
- LLMs Adapt to Legal Texts (to an extent): LLMs show higher accuracy on legal event ordering compared to narrative texts, suggesting that legal language might offer clearer temporal structures for models to leverage.
- Implicit Events are Key: Pairs involving one implicit and one explicit event yield the highest accuracy, indicating that LLMs are better at inferring temporal relationships when one event is not directly stated but implied.
- CoT Prompting Doesn't Always Help: Contrary to expectations, Chain-of-Thought prompting did not improve performance for temporal event ordering on LEXTIME or TRACIE, and in some cases, even decreased accuracy.
- Linguistic Complexities are Bottlenecks: Features common in legal writing, such as paraphrases, subordinate clauses, passive voice, and non-past tenses, disproportionately contribute to model errors, highlighting the need for domain-specific modeling.
Key Results
- LLMs achieved higher accuracy on LEXTIME than on TRACIE, with improvements up to +10.5% (GPT-4o, few-shot).
- The highest accuracy recorded was 80.8% for explicit-implicit event pairs using GPT-4 Turbo in a few-shot setting.
- Longer input contexts generally improved performance for larger models like GPT-4 Turbo (+2.3%), while shorter contexts were better for smaller models.
- CoT prompting led to a 3.4% accuracy drop for LEXTIME compared to standard few-shot prompting.
- Errors were significantly associated with linguistic features like paraphrases (+89% increase in error samples) and subordinate clauses (+84%).
Limitations and Future Work
The study acknowledges several limitations:
- The generalizability of findings is constrained by the specific area of law covered and the size of LEXTIME (512 samples).
- The temporal relation scheme is limited to three categories (precedes, simultaneous, follows), whereas a more granular approach might be needed for richer temporal distinctions.
- Ambiguous cases were resolved during annotation, meaning the dataset doesn't fully represent these challenging scenarios.
Future work could extend the evaluation to other temporal reasoning tasks (event frequency, duration), explore more granular temporal relation schemes, and develop domain-sensitive modeling approaches to better handle the linguistic complexities of legal texts.
Summary updated