Abstract: The effectiveness of Large Language Models (LLMs) in legal reasoning is often limited due to the unique legal terminologies and the necessity for highly specialized knowledge. These limitations highlight the need for high-quality data tailored for complex legal reasoning tasks. This paper introduces LEGALSEMI, a benchmark specifically curated for legal scenario analysis. LEGALSEMI comprises 54 legal scenarios, each rigorously annotated by legal experts, based on the comprehensive IRAC (Issue, Rule, Application, Conclusion) framework. In addition, LEGALSEMI is accompanied by a structured knowledge graph (SKG). A series of experiments were conducted to assess the usefulness of LEGALSEMI for IRAC analysis. The experimental results demonstrate the effectiveness of incorporating the SKG for issue identification, rule retrieval, application and conclusion generation using four different LLMs. LEGALSEMI will be publicly available upon acceptance of this paper.
Introduction
While Large Language Models (LLMs) like GPT-4 have demonstrated remarkable linguistic capabilities, their performance in specialized domains like law remains inconsistent. Legal reasoning requires more than just fluency; it demands precision, adherence to logical frameworks, and the ability to bridge the gap between everyday language and technical "legalese." Current models frequently encounter three primary hurdles: hallucination (citing non-existent precedents), reasoning gaps (jumping to conclusions without intermediate logic), and the linguistic gap (failing to map layperson descriptions to specific legal rules).
To address these challenges, researchers have introduced LEGALSEMI, a new benchmark and semi-structured knowledge base (SKE) designed to help LLMs master the IRAC (Issue, Rule, Application, Conclusion) method of legal reasoning. By grounding models in a structured graph of Malaysian Contract Law, this approach significantly improves the accuracy of legal issue identification and rule retrieval.
The LEGALSEMI Framework and IRAC Methodology
In professional legal practice, the IRAC framework is the standard for solving complex problems. Most existing AI benchmarks focus solely on the final "Conclusion," neglecting the critical intermediate steps. LEGALSEMI provides a specialized dataset consisting of 54 legal scenarios, rigorously annotated by legal experts and law students. Each scenario includes decomposed questions (Issues), relevant sections of the Contracts Act 1950, pertinent court cases (Rules), and step-by-step reasoning (Application).
LEGALSEMI is unique in its depth of "Application" annotations and its integration with a structured knowledge base. Compared to earlier datasets like SIRAC or SARA (which focuses on US tax law), LEGALSEMI offers a more comprehensive map of how legal concepts, statutes, and court cases relate to one another.
Methodology: A Neuro-Symbolic Approach
The researchers employed a neuro-symbolic approach, combining the statistical power of LLMs with the hard logic of a structured database.
- The SKE (Semi-structured Knowledge Base): A graph database (Neo4j) was constructed containing 3,207 nodes and 2,441 edges. This graph maps the relationships between legal concepts (e.g., "invitation to treat"), statutes (Contracts Act 1950), and historical court cases.
- Enriched Representations: To bridge the linguistic gap, LLMs were used to generate "layman" explanations and real-life examples for complex statutes. These were stored in the SKE to help the model identify the correct law when presented with a casual, non-technical scenario.
- Prompt Engineering: Various prompting strategies were tested, including self-evaluation and in-context learning, to guide models through the distinct IRAC stages.
Key Insights and Experimental Results
The experiments revealed that structure is more important than raw model power for legal tasks. Even high-performing LLMs saw significant gains when "hinted" with legal concepts from the SKE. For example, providing legal concepts improved the quality of issue generation by over 21.4%.
Performance Gains
- Issue Identification: GPT-3.5 Turbo experienced a 28% performance increase when leveraging legal concepts from the SKE.
- Rule Retrieval: By incorporating the SKE, retrieval performance for relevant statutes and cases jumped from 6.32% to 12.93%.
- Application Generation: When provided with the correct issues and rules, the models' ability to generate logical "Application" sections improved by 18.9% for GPT-3.5.
Critical Observations
- The Subordinate Concept Struggle: LLMs are proficient at identifying broad concepts (like "Offer") but struggle with granular sub-concepts (like "Audition" as a form of invitation to treat), suggesting a lack of detail in standard pre-training data.
- Retrieval Nuances: Interestingly, traditional keyword-based methods like "Smooth TF-IDF" outperformed sophisticated neural embeddings for rule retrieval. This suggests that embeddings may inadvertently widen the linguistic gap in highly specialized domains.
- Defeasible Reasoning: Legal reasoning is rarely "black and white." The "Application" phase is critical because it justifies assumptions, and LEGALSEMI's focus on this stage aligns well with human expert judgment (showing a 0.89 Spearman’s rank correlation).
Conclusion and Future Work
LEGALSEMI demonstrates that augmenting LLMs with structured legal knowledge is a viable path toward more reliable automated legal reasoning. However, limitations remain. With only 54 scenarios, the dataset is high-quality but small. Models also still struggle with "association errors"—citing the correct law but applying it for the wrong reason—and complex multi-hop reasoning across disparate cases. Future work will focus on scaling the dataset and testing the SKE approach across different legal jurisdictions beyond Malaysian Contract Law.
Summary updated