FZI-WIM at SemEval-2024 Task 2: Self-Consistent CoT for Complex NLI in Biomedical Domain

Jin Liu, Steffen ThomaNAACL 2024

TL;DR

This paper introduces a "Self-Consistent Chain-of-Thought" (CoT) approach for complex Natural Language Inference (NLI) in the biomedical domain. By generating multiple reasoning paths and using majority voting, their system achieves state-of-the-art results on the NLI4CT task, demonstrating improved faithfulness and consistency compared to simpler methods.

What Problem Does This Paper Address?

The paper tackles the challenge of ensuring reliable and trustworthy Natural Language Inference (NLI) in the critical domain of clinical trials. While NLI models are advancing, they often struggle with consistency and faithfulness, meaning they might produce contradictory or fabricated conclusions. This is particularly problematic in healthcare, where errors can have serious consequences. The NLI4CT task specifically aims to evaluate NLI models on their ability to infer relationships between premises (clinical trial reports) and statements, with a focus on safety and trust. Existing methods, even those using large language models (LLMs) with Chain-of-Thought (CoT) prompting, still exhibit these inconsistencies, necessitating a more robust approach.

Methods

The core of the FZI-WIM system is an enhancement of the Chain-of-Thought (CoT) paradigm, known for eliciting reasoning capabilities in LLMs. Instead of relying on a single reasoning path generated by greedy decoding, the authors employ Self-Consistency. This involves:

  1. CoT Generation with GPT-4: For training and validation, GPT-4 is used to generate reasoning chains (rationales) for NLI tasks. This distilled knowledge is then used to instruction-tune an open-source LLM.
  2. LoRA Instruction-Tuning: A parameter-efficient fine-tuning method called Low-Rank Adaptation (LoRA) is used to adapt the Mixtral-8x7B-Instruct LLM with the generated CoT data. This allows the model to learn domain-specific reasoning.
  3. Self-Consistent Inference: During inference, the instruction-tuned model generates multiple CoT reasoning chains for each premise-statement pair. These multiple chains are then aggregated using majority voting to determine the final verification label (Entailment or Contradiction). This self-consistency approach aims to mitigate errors that might occur in a single reasoning path.

Datasets

The primary dataset used is the NLI4CT (Safe Biomedical Natural Language Inference for Clinical Trials) dataset. This dataset consists of premises from clinical trial reports and statements that require entailment or contradiction judgments. The dataset is split into training (1700 examples), validation (200 examples), and test sets (5500 examples). Notably, the test set includes 500 "anchor" examples and 5000 examples created with semantic-preserving and semantic-altering interventions to evaluate consistency and faithfulness.

Key Insights and Ideas

  • Self-Consistency is Key for Trust: The paper strongly advocates for self-consistency as a crucial step beyond basic CoT. By sampling multiple reasoning paths, the system becomes more robust to individual errors, leading to more reliable NLI.
  • LLMs Can Be Biased Towards Statement Information: The authors observed that their instruction-tuned LLM sometimes contradicts the premise if information is only in the statement, even if factually correct. This highlights a subtle but important failure mode.
  • Domain-Specific Tuning Matters: LoRA instruction-tuning with distilled CoT rationales from GPT-4 significantly improved the LLM's ability to handle biomedical NLI tasks.
  • Majority Voting is a Simple but Effective Aggregation: While not perfect, majority voting across multiple CoT chains proved effective in improving performance, especially for faithfulness.

Key Results

The FZI-WIM system achieved impressive results on the NLI4CT task:

  • 1st Place: Baseline F1 score (0.80) on the control set (no interventions).
  • 3rd Place: Faithfulness score (0.90).
  • 12th Place: Consistency score (0.73).

Compared to a greedy CoT approach, their self-consistent CoT system showed improvements of:

  • 1.31 percentage points in baseline F1.
  • 0.75 percentage points in consistency score.
  • 0.69 percentage points in faithfulness score.

The paper also highlights that while self-consistency offers significant gains, the improvement for binary classification tasks can be limited, suggesting room for further refinement.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Improvement for Binary Tasks: The performance gains from self-consistency were somewhat limited for the binary classification nature of the NLI4CT task.
  • Number of Generated Chains: Computational constraints limited the number of generated reasoning chains (average 10.36), which is fewer than in prior self-consistency work (around 40).
  • Aggregation Mechanism: The simple majority voting mechanism might not be sufficient for complex reasoning.

Future work could involve:

  • Developing more sophisticated aggregation mechanisms that go beyond simple majority voting.
  • Exploring more fine-grained evaluation of individual reasoning steps within the CoT paradigm.
  • Investigating how to better integrate domain-specific knowledge and common sense for more robust NLI.

Summary updated

BibTeX
@inproceedings{liu2024fzi,
  author = {Jin Liu and Steffen Thoma},
  title = {{FZI-WIM at SemEval-2024 Task 2: Self-Consistent CoT for Complex NLI in Biomedical Domain}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure