TL;DR
This paper introduces the TRISM framework and its practical pipeline, RASOR, which combine the natural language fluency of Large Language Models (LLMs) with the logical rigor of symbolic reasoning (Knowledge Graphs). By generating explicit rationales, dynamically selecting document chunks, and verifying outputs against legal constraints, RASOR slashes LLM hallucination rates in contract analysis from 75% to under 40%.
What Problem Does This Paper Address?
Large Language Models (LLMs) show immense promise in automating legal document analysis, but they suffer from a fatal flaw: hallucinations. In a high-stakes field like law, a single fabricated citation or missed precedent can jeopardize an entire case and lead to severe court sanctions.
Standard Retrieval-Augmented Generation (RAG) systems attempt to ground LLMs in authoritative texts, but commercial legal RAG tools still maintain error rates between 17% and 34%. This is because traditional RAG relies on opaque chunking and ranking heuristics that often strip away vital context. Furthermore, standard LLMs struggle with the complex, hierarchical nature of legal reasoning—such as understanding when a local municipal regulation overrides a general state principle, or identifying nuanced contextual exceptions within contract clauses.
Methods
The authors propose TRISM (Trustworthy, Reliable, Interpretable, Safe Models), a framework rooted in NeuroSymbolic AI. This approach fuses neural networks (excellent at pattern recognition and language generation) with symbolic systems (excellent at rule-based logic and structured knowledge representation).
To operationalize TRISM, the authors develop RASOR (Rationalize, Select, and Refine), a modular RAG pipeline that breaks down legal text generation into five distinct steps:
- Rationalize: A legal LLM (SaulLM) is fine-tuned to generate explicit, interpretable legal rationales based on a user's query.
- Select: A hyperparameter-free Neural Rationale-based Chunk Selector (NRCS) uses a Dense Passage Retriever to match these generated rationales with the most relevant document chunks.
- Refine: A Refiner module applies preference learning inspired by Direct Preference Optimization (DPO) to optimize the rationales, bypassing the need for complex reinforcement learning (RLHF).
- Verify: An unsupervised Verifier filters out irrelevant or contradictory document chunks to mitigate hallucination risks.
- Inference: The final LLM generates a response structured with explicit citations and clear logical pathways.
Additionally, the paper outlines a formal framework for constructing and automatically updating Legal Knowledge Graphs (KGs). These KGs explicitly map legal entities (statutes, cases, principles) as nodes and their hierarchical relationships as edges, ensuring the AI respects jurisdictional and temporal boundaries.
Datasets
The framework was evaluated using the Contract Understanding Atticus Dataset (CUAD), a prominent legal benchmark containing over 13,000 expert annotations across 510 contracts. The evaluation was performed on the CUAD test set, which consists of 400 complex legal queries.
Key Insights and Ideas
- "Show Your Work" Wins: Forcing the system to generate explicit, structured rationales before retrieving documents significantly improves both transparency and raw generation accuracy.
- Dynamic Context Filtering: Instead of using a fixed number of retrieved document chunks ($K$), RASOR dynamically adjusts $K$ based on query complexity. This minimizes background noise for straightforward queries while capturing necessary context for intricate legal issues.
- An Evolutionary Pathway: RASOR bridges pure neural approaches and full neurosymbolic systems. Because its pipeline is modular, each neural component has a clear symbolic counterpart that can be progressively swapped out for formal legal ontologies and strict rule-based verifiers.
Key Results
- Massive Hallucination Drop: RASOR successfully reduced hallucination rates from roughly 75% (with SaulLM-52B) to under 40%.
- Superior Accuracy: Evaluated using strict exact string matching (which heavily penalizes hallucinations), RASOR outperformed the state-of-the-art Contriever re-ranker by up to 17%. It achieved 52% accuracy at 128-token chunks and extended to 64% accuracy at 512-token chunks.
- Robust Retrieval: In information retrieval tests, RASOR achieved a precision-recall balance with a robust recall rate of approximately 93% across varying document chunk sizes.
Limitations and Future Work
While RASOR establishes a strong foundation, its current implementation still relies heavily on neural pathways for rationale generation and verification.
The authors outline several immediate directions for future work to complete the neurosymbolic transition:
- Replacing the neural chunk selector with a Knowledge Graph-enhanced selector using legal ontologies (e.g., GraphRAG).
- Upgrading the unsupervised verifier to a Rule-based Logical Verifier that enforces formal legal constraints and constitutional AI principles.
- Implementing multi-agent legal reasoning systems where specialized agents handle distinct legal domains (such as statutory interpretation vs. case precedents) and coordinate through structured legal protocols.
Summary updated