MAINDZ at SemEval-2024 Task 5: CLUEDO - Choosing Legal oUtcome by Explaining Decision through Oversight

Irene Benedetto, Alkis Koudounas, Lorenzo Vaiani, Eliana Pastor, Luca Cagliero, Francesco TarasconiNAACL 2024

TL;DR

This paper introduces CLUEDO, a novel framework for legal reasoning using Large Language Models (LLMs). CLUEDO leverages an ensemble of LLMs to predict legal outcomes and explain their reasoning, achieving strong performance on a new legal task and outperforming individual models while offering more stable predictions.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown impressive capabilities in general reasoning, but their application in specialized domains like law remains less explored. While LLMs hold promise for transforming legal practices, concerns about their reliability, such as generating misleading or incorrect information, are significant. Existing legal benchmarks are often limited and don't fully capture the nuances of legal tasks. This paper addresses the challenge of evaluating and enhancing LLMs' legal reasoning abilities, specifically focusing on determining the correctness of a proposed legal argument for a given case and question.

Methods

The authors tackle this problem by first reformulating the legal reasoning task into a multiple-choice question answering (MCQA) format. They employ a technique called Multiple-Choice Prompting (MCP), where LLMs are presented with a question and several candidate answers, allowing them to compare options directly.

The core of their contribution is the CLUEDO framework, which stands for "Choosing Legal Outcome by Explaining Decisions through Oversight." CLUEDO works in two main stages:

  1. Collaborative Models: Several LLMs are trained to predict the correct answer to a legal question and, crucially, to generate an explanation for their choice. These models are trained using the MCQA format.
  2. Detective Model: A final "detective" LLM (in this case, GPT-4) operates in a zero-shot manner. It receives the outputs (answers and explanations) from the collaborative models and uses this collective intelligence to make the final decision on the most accurate answer. This "oversight" mechanism aims to improve accuracy and stability.

Datasets

The paper utilizes a dataset introduced by Bongard et al. (2022) for the SemEval-2024 Task 5. This dataset is derived from U.S. civil procedure cases and includes:

  • General case introductions.
  • Specific legal questions.
  • Potential solution arguments.
  • Annotated labels indicating correctness (1 or 0).
  • Detailed analyses explaining the arguments' applicability.

The dataset is adapted for the MCQA task by presenting candidate answers, and in cases with only one provided answer, the system automatically generates a "None of the above" option.

Key Insights and Ideas

  • MCQA is Superior: The multiple-choice question answering format consistently leads to better performance (higher F1 Macro scores) compared to a single-choice approach for zero-shot models. This suggests that providing LLMs with explicit options to compare aids their reasoning.
  • Explanation Matters: Including the analysis/explanation generation during the training process significantly improves performance for multiple-choice tasks, leading to more balanced precision and recall. This highlights the importance of not just predicting an answer but also understanding why it's correct.
  • Ensemble Power: The CLUEDO framework, by aggregating the outputs of multiple LLMs and using a detective model for oversight, outperforms individual models. This ensemble approach mitigates the weaknesses of any single model.
  • Stability is Key: While individual powerful models like GPT-4 can achieve high scores, they can also exhibit significant variability in their predictions. CLUEDO, by averaging and synthesizing multiple predictions, offers more stable and reliable results, even if its absolute top score on test data is slightly lower than the best individual model.

Key Results

  • The CLUEDO system achieved an F1 macro score of 0.74 on the development set and 0.77 on the test set, securing second place in the public SemEval-2024 Task 5 competition.
  • In zero-shot evaluations (Table 1), the multiple-choice approach generally led to higher F1 Macro scores across various model families (Flan T5 XXL, GPT-4, Llama 2, Mistral) compared to single-choice.
  • Training with analysis generation improved performance for Llama 2 models in multiple-choice tasks, yielding an F1 Macro score increase.
  • CLUEDO demonstrated reduced prediction variability compared to individual models, particularly GPT-4, even when temperature was set to zero.

Limitations and Future Work

  • Model Selection: The selection of collaborative models was based on their performance on the development set. The paper notes that for the "detective" model, GPT-4 was chosen due to its strong zero-shot performance.
  • GPT-4 Variability: The paper acknowledges that GPT-4, despite its power, can exhibit significant output variations, a known issue that CLUEDO aims to mitigate through its ensemble approach.
  • Future Directions: The authors suggest that their framework's success indicates the promise of tailored approaches for enhancing LLM legal reasoning. Future work could explore more sophisticated ensemble strategies or investigate the impact of different collaborative model combinations. The paper also implicitly points towards the need for more robust legal benchmarks that capture complex reasoning.

Summary updated

BibTeX
@inproceedings{benedetto2024maindz,
  author = {Irene Benedetto and Alkis Koudounas and Lorenzo Vaiani and Eliana Pastor and Luca Cagliero and Francesco Tarasconi},
  title = {{MAINDZ at SemEval-2024 Task 5: CLUEDO - Choosing Legal oUtcome by Explaining Decision through Oversight}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure