Abstract
Large Language Models (LLMs) often struggle in specialized domains like law because they rely on user-provided narratives that frequently omit critical factual details. This paper presents D3LM (Diagnostic Legal Large Language Model), a framework designed to bridge this information gap by shifting AI from a passive responder to an active, diagnostic interrogator. By leveraging a novel Positive-Unlabeled Reinforcement Learning (PURL) algorithm and legal knowledge graphs, D3LM identifies missing "load-bearing" facts and proactively solicits them from the user. Experimental results on the newly curated US-CVG dataset demonstrate that D3LM significantly outperforms state-of-the-art models like GPT-4 in generating accurate legal court views and earns higher preference from legal professionals.
Introduction: The Information Gap in Legal AI
While LLMs like GPT-4 demonstrate remarkable general-purpose reasoning, their application in the legal domain is hindered by the "user-led" nature of current interactions. Non-expert users typically provide vague or incomplete descriptions of legal incidents, missing the specific elements—such as intent or state of intoxication—that determine judicial outcomes. In a professional setting, a lawyer does not simply answer a client's initial query; they perform a diagnostic analysis, asking targeted questions to fill factual voids.
The D3LM framework addresses this by mimicking the professional diagnostic process, ensuring that the final "Court View" is based on a complete and legally relevant factual foundation.
The D3LM Framework: Iterative Diagnostics and PURL
The core of D3LM is its ability to "know what it doesn't know." This is achieved through three primary technical innovations:
- Iterative Diagnostic Loop: Instead of a single-turn generation, the model outputs a specialized "completeness token." If the model identifies that the current fact set is insufficient to provide a reliable legal opinion, it triggers a diagnostic phase to ask the user for more information.
- Positive-Unlabeled Reinforcement Learning (PURL): Selecting the right question from thousands of potential legal facts is a complex search problem. D3LM utilizes a legal knowledge graph where nodes represent facts and rules. Since the model knows which facts are present (Positive) but cannot easily label all other facts as irrelevant (Unlabeled), it employs a Positive-Unlabeled (PU) risk estimator. This is integrated with NeuralUCB, a bandit-based reinforcement learning algorithm, to select the most informative fact to query.
- IRAC Structuring: To process complex case law into a format navigable by RL algorithms, the model uses the IRAC (Issue, Rule, Analysis, Conclusion) framework. This standardizes legal documents into structured data, allowing the model to map narratives to specific legal rules.
The US-CVG Dataset
To train and evaluate D3LM, the authors introduced the US-CVG dataset, the first English-language benchmark for Court Views Generation based on US criminal law. Derived from the Caselaw Access Project, the dataset includes 19,491 training cases. The authors utilized GPT-4 to condense massive legal documents into structured IRAC formats and fact-rule graphs, which were subsequently validated by legal experts to ensure high fidelity.
Experimental Results and Human Evaluation
D3LM was benchmarked against GPT-4 and fine-tuned Llama2-13B. In automated metrics, D3LM achieved a ROUGE-L of 59.2 and a BLEU-N of 26.9, notably higher than GPT-4’s 57.1 and 14.1 respectively.
Beyond automated scores, human evaluation by legal professionals confirmed the model's superiority. D3LM received higher ratings for Accuracy (4.48 vs. 4.39 for GPT-4) and the Adoptability of its Rationales (4.19 vs. 4.06). Furthermore, in usability tests with 69 participants, over 62% preferred D3LM, citing its proactive questioning as a key factor in their perceived reliability and satisfaction with the AI's legal advice.
Conclusion and Future Directions
D3LM represents a significant step toward "lawyer-like" AI by prioritizing proactive information gathering over passive response. While currently focused on US criminal law due to the requirements of specialized knowledge graphs, the framework's success suggests broad applicability. Future work will focus on reducing the latency of the iterative loop and expanding the system's reach into civil law, medical diagnostics, and other expert-driven consultancy domains.
Summary updated