Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models

Antoine Louis, Gijs van Dijck, Gerasimos SpanakisAAAI 2024 · DOI

TL;DR

This paper introduces LLeQA, the first expert-annotated dataset for long-form legal question answering in French, addressing the critical need for accessible legal information. They propose a "retrieve-then-read" framework using retrieval-augmented LLMs to generate comprehensive, interpretable answers to statutory law questions, demonstrating promising but imperfect results.

What Problem Does This Paper Address?

Navigating legal disputes is a daunting task for most individuals due to a lack of legal knowledge, often leaving them vulnerable. While automated legal aid systems offer a promising solution, existing approaches are limited by narrow scopes or provide uninformative answers. This paper tackles the challenge of generating detailed, interpretable answers to broad statutory law questions, aiming to bridge the gap between complex legal texts and the general public. The authors highlight the significant global issue of billions of people facing unresolved legal challenges, underscoring the importance of improving access to legal information.

Methods

The core of the proposed solution is a "retrieve-then-read" pipeline.

  • Retriever: A lightweight bi-encoder model is employed to efficiently search a large corpus of legal articles and retrieve a small set of the most relevant provisions for a given question. This retriever is trained using a contrastive loss function to ensure relevant question-provision pairs are closer in the embedding space than irrelevant ones. They utilize both in-batch negatives and BM25-based hard negatives for training.
  • Generator (Reader): An instruction-tuned Large Language Model (LLM) is used to read the retrieved provisions and formulate a comprehensive answer. Two learning strategies are explored for the generator:
  • In-context learning: This involves providing the LLM with instructions and a few examples (zero-shot, one-shot, few-shot) to guide its response without updating its weights.
  • Parameter-efficient finetuning (QLORA): This technique fine-tunes a small number of adapter weights while keeping the base LLM's parameters quantized and frozen, making the process computationally efficient.
  • Interpretability: To enhance trustworthiness, the system aims to provide rationales by extracting relevant paragraph markers from the retrieved articles, rather than generating free-form explanations, which can be prone to hallucination.

Datasets

The paper introduces the Long-form Legal Question Answering (LLeQA) dataset.

  • LLeQA: This dataset is built upon the BSARD dataset and enhanced with 760 new legal questions and 5,308 additional statutory articles, bringing the total to 1,868 expert-annotated French legal questions and 27,942 Belgian statutory articles. It includes detailed answers written by legal professionals and paragraph-level references to relevant legal provisions. The dataset is designed to be multifaceted, supporting tasks beyond LQA, such as legal inquiry classification and topic modeling.

Key Insights and Ideas

  • The critical need for long-form, interpretable legal answers: The paper strongly emphasizes that brief or uninformative answers are insufficient for complex legal questions. Users need detailed explanations rooted in specific legal provisions to truly understand their rights and obligations.
  • LLMs can be adapted for specialized legal domains: By leveraging retrieval-augmented LLMs and exploring different finetuning strategies, the authors demonstrate the potential of these models to handle the complexities of legal language and reasoning.
  • Synthetic rationale generation is a viable, cost-effective alternative: Generating paragraph-level rationales using LLMs offers a scalable solution to enhance interpretability, though the quality is not yet at expert level.
  • Automatic metrics are insufficient for evaluating legal QA: The authors highlight that standard metrics like ROUGE are inadequate for assessing the factual correctness and query relevance of long-form legal answers, pointing to the need for more nuanced evaluation methods.
  • The "retrieve-then-read" pipeline is effective but not perfect: While the framework shows promise, the qualitative analysis reveals that LLMs can still hallucinate facts, underscoring the challenges in ensuring factual accuracy in legal contexts.

Key Results

  • Retriever Performance: The CamemBERT retriever, fine-tuned on LLeQA, significantly outperforms strong baselines like BM25 and mE5, achieving an R@5 score of 48.6% and R@10 of 60.6%.
  • Generator Performance (Automatic Evaluation):
    • METEOR scores: For question answering, Vicuna and WizardLM achieved the highest METEOR scores (19.7 and 20.4 respectively) when fine-tuned, outperforming zero-shot and few-shot in-context learning. TULU showed the lowest scores.
  • Rationale Extraction F1: Fine-tuned models generally performed poorly on rationale extraction, with the highest F1 score being 3.5% for TULU.
  • Qualitative Analysis:
    • WizardLM and Vicuna produced answers with high overlap with ground truth and were generally factually correct, though prone to hallucination.
  • TULU tended to produce concise answers, likely due to its training data.
  • Guanaco and WizardLM sometimes exhibited repetitiveness.
  • A significant finding was that LLMs could fabricate facts, even when seemingly addressing the question, highlighting the risk of misleading users.

Limitations and Future Work

  • Hallucinations: The primary limitation acknowledged is the propensity for LLMs to hallucinate facts in both answers and rationales, making them potentially unreliable for critical legal advice.
  • Automatic Metric Limitations: The authors note that current automatic metrics do not fully capture the quality of long-form legal answers, suggesting the need for more robust evaluation methods.
  • Human Evaluation: Due to resource constraints, a comprehensive human evaluation was not conducted, which is crucial for a thorough assessment of answer quality.
  • Future Work: The authors suggest exploring methods to further ground LLM responses in legal provisions, mitigate hallucinations, and develop more accurate evaluation metrics. They also emphasize the ethical consideration of not prematurely deploying these systems due to the risk of misinformation.

Summary updated

BibTeX
@inproceedings{louis2024interpretable,
  author = {Antoine Louis and Gijs van Dijck and Gerasimos Spanakis},
  title = {{Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30232},
}
Figure