Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learning

Yang Wu, Chenghao Wang, Ece Gumusel, Xiaozhong Liu

Abstract

The integration of generative Large Language Models (LLMs) into the legal domain has been accelerated by their versatile nature. However, users without legal backgrounds often struggle to formulate professional queries, inadvertently overlooking critical legal factors in their case narratives. To address this, we propose the Diagnostic Legal Large Language Model (D3LM). D3LM utilizes adaptive lawyer-like diagnostic questions to collect missing case information before providing feedback. It incorporates an innovative graph-based Positive-Unlabeled Reinforcement Learning (PURL) algorithm and an LLM-based stopping criterion for precise Court Views Generation (CVG). Our research also introduces a new English-language CVG dataset based on US case law. D3LM surpasses classical LLMs by delivering superior performance and a more reliable user experience.

Most non-lawyers seeking legal advice from Large Language Models (LLMs) provide "vague narratives" that omit critical legal factors—such as the presence of surveillance footage or the defendant's state of intoxication. Standard LLMs are typically passive responders; they process whatever text is provided and generate a response. If the input is incomplete, the output often results in "hallucinated" or overly generalized "Court Views" (judicial rationales).

Comparison of standard LLM, human lawyer, and the D3LM diagnostic approach.

As illustrated above, while a standard LLM might provide an insufficient response based on a partial story, a lawyer—and now D3LM—proactively asks professional questions to uncover the facts necessary for a culpable determination.

The D3LM Framework

D3LM transitions AI from a passive responder to an active investigator through three main pillars:

  1. The Diagnostic Loop: When a user submits a query, the model evaluates information completeness. If critical facts are missing, it triggers a specific token to activate a question-generation phase rather than providing a premature answer.
  2. Positive-Unlabeled Reinforcement Learning (PURL): This core innovation uses a Fact-Rule Graph—a structured map of legal concepts. PURL treats known facts as "positive" and potential missing facts as "unlabeled." It employs a NeuralUCB bandit algorithm to select the most legally relevant missing facts and transforms them into natural language questions.
  3. IRAC Structural Convention: To handle the extreme length of legal documents, the model uses the IRAC (Issue, Rule, Analysis, Conclusion) framework. This allows the LLM to distill complex legal texts into a format optimized for reasoning.

Data Scarcity and the US-CVG Dataset

To train and validate D3LM, we introduced the first English-language dataset for Court View Generation based on US case law. Sourced from the Caselaw Access Project, the dataset focuses on criminal cases and includes approximately 19,500 training samples. Because US cases often exceed standard LLM context windows (averaging over 7,600 tokens), we utilized GPT-4 to distill these into structured IRAC summaries, which were then validated by legal professionals.

Experimental Results

D3LM was tested against several baselines, including GPT-4 and fine-tuned versions of Llama-2.

Automatic Metrics

D3LM achieved a ROUGE-L score of 59.2 and a BLEU-N score of 26.9, significantly outperforming GPT-4 (which scored 57.1 and 14.1, respectively). The substantial jump in BLEU scores suggests that D3LM’s outputs are much closer to actual judicial rationales than standard generative models.

Human Evaluation and User Preference

Three legal professionals evaluated the models on fluency, accuracy, and adoptability. D3LM consistently outperformed the baselines in the categories most critical for legal reliability.

Human evaluation scores comparing D3LM against Llama2 and GPT-4 baselines.

In a usability study with 69 participants, 62.3% preferred D3LM over standard LLMs, citing higher perceived reliability and satisfaction with the interactive questioning process.

Conclusion and Future Outlook

D3LM demonstrates that anchoring reinforcement learning to a Fact-Rule Graph ensures AI asks legally meaningful questions. While currently optimized for criminal law, this "diagnostic" framework could be extended to other high-stakes fields like medical consultations or business strategy, where missing facts are just as costly as they are in the courtroom. Future work will focus on reducing the latency inherent in multi-turn diagnostic interactions and expanding the knowledge graphs to civil and contract law.

Summary updated

Abstract

The integration of generative Large Language Models (LLMs) into various applications, including the legal domain, has been accelerated by their expansive and versatile nature. However, when facing a legal case, users without a legal background often struggle to formulate professional queries and may inadvertently overlook critical legal factors when presenting their case narrative to LLMs. To address this issue, we propose the Diagnostic Legal Large Language Model (D3LM), which utilizes adaptive lawyer-like diagnostic questions to collect additional case information and then provides high-quality feedback. D3LM incorporates an innovative graph-based Positive-Unlabeled Reinforcement Learning (PURL) algorithm, enabling the generation of critical questions and enhancing user-LLM interactions. Moreover, an integrated LLM-based stopping criterion facilitates precise Court Views Generation (CVG). Our research also introduces a new English-language CVG dataset based on the US case law database, enriching the realm of LLM research and deployment with a vital dimension. D3LM surpasses classical LLMs by delivering outstanding performance and a remarkable user experience in the legal domain.
BibTeX
@inproceedings{wu2024knowledge,
  author = {Yang Wu and Chenghao Wang and Ece Gumusel and Xiaozhong Liu},
  title = {{Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.918},
}
Figure