Abstract
The integration of generative Large Language Models (LLMs) into the legal domain has been accelerated by their versatile nature. However, users without legal backgrounds often struggle to formulate professional queries, inadvertently overlooking critical legal factors in their case narratives. To address this, we propose the Diagnostic Legal Large Language Model (D3LM). D3LM utilizes adaptive lawyer-like diagnostic questions to collect missing case information before providing feedback. It incorporates an innovative graph-based Positive-Unlabeled Reinforcement Learning (PURL) algorithm and an LLM-based stopping criterion for precise Court Views Generation (CVG). Our research also introduces a new English-language CVG dataset based on US case law. D3LM surpasses classical LLMs by delivering superior performance and a more reliable user experience.
Introduction: The Challenge of Passive Legal AI
Most non-lawyers seeking legal advice from Large Language Models (LLMs) provide "vague narratives" that omit critical legal factors—such as the presence of surveillance footage or the defendant's state of intoxication. Standard LLMs are typically passive responders; they process whatever text is provided and generate a response. If the input is incomplete, the output often results in "hallucinated" or overly generalized "Court Views" (judicial rationales).
As illustrated above, while a standard LLM might provide an insufficient response based on a partial story, a lawyer—and now D3LM—proactively asks professional questions to uncover the facts necessary for a culpable determination.
The D3LM Framework
D3LM transitions AI from a passive responder to an active investigator through three main pillars:
- The Diagnostic Loop: When a user submits a query, the model evaluates information completeness. If critical facts are missing, it triggers a specific token to activate a question-generation phase rather than providing a premature answer.
- Positive-Unlabeled Reinforcement Learning (PURL): This core innovation uses a Fact-Rule Graph—a structured map of legal concepts. PURL treats known facts as "positive" and potential missing facts as "unlabeled." It employs a NeuralUCB bandit algorithm to select the most legally relevant missing facts and transforms them into natural language questions.
- IRAC Structural Convention: To handle the extreme length of legal documents, the model uses the IRAC (Issue, Rule, Analysis, Conclusion) framework. This allows the LLM to distill complex legal texts into a format optimized for reasoning.
Data Scarcity and the US-CVG Dataset
To train and validate D3LM, we introduced the first English-language dataset for Court View Generation based on US case law. Sourced from the Caselaw Access Project, the dataset focuses on criminal cases and includes approximately 19,500 training samples. Because US cases often exceed standard LLM context windows (averaging over 7,600 tokens), we utilized GPT-4 to distill these into structured IRAC summaries, which were then validated by legal professionals.
Experimental Results
D3LM was tested against several baselines, including GPT-4 and fine-tuned versions of Llama-2.
Automatic Metrics
D3LM achieved a ROUGE-L score of 59.2 and a BLEU-N score of 26.9, significantly outperforming GPT-4 (which scored 57.1 and 14.1, respectively). The substantial jump in BLEU scores suggests that D3LM’s outputs are much closer to actual judicial rationales than standard generative models.
Human Evaluation and User Preference
Three legal professionals evaluated the models on fluency, accuracy, and adoptability. D3LM consistently outperformed the baselines in the categories most critical for legal reliability.
In a usability study with 69 participants, 62.3% preferred D3LM over standard LLMs, citing higher perceived reliability and satisfaction with the interactive questioning process.
Conclusion and Future Outlook
D3LM demonstrates that anchoring reinforcement learning to a Fact-Rule Graph ensures AI asks legally meaningful questions. While currently optimized for criminal law, this "diagnostic" framework could be extended to other high-stakes fields like medical consultations or business strategy, where missing facts are just as costly as they are in the courtroom. Future work will focus on reducing the latency inherent in multi-turn diagnostic interactions and expanding the knowledge graphs to civil and contract law.
Summary updated