NootNoot At SemEval-2024 Task 6: Hallucinations and Related Observable Overgeneration Mistakes Detection

Sankalp Bahad, Yash Bhaskar, Parameswari KrishnamurthyNAACL 2024

TL;DR

This paper tackles the critical problem of semantic hallucinations in neural language generation (NLG) systems, where models produce fluent but factually incorrect text. They introduce the SHROOM dataset and evaluate three methods for detecting these hallucinations: fine-tuning on labeled data, fine-tuning on partially labeled data, and a zero-shot approach using a large language model (LLM). The zero-shot approach with Mixtral 8x7b showed competitive performance, highlighting its potential for hallucination detection without extensive training.

What Problem Does This Paper Address?

Modern neural language generation (NLG) models are incredibly good at producing fluent and coherent text. However, a significant challenge is that these models can also "hallucinate" – generate outputs that are grammatically sound but semantically incorrect or unsupported by the source input. This is a major roadblock for deploying NLG in critical applications where accuracy is paramount, such as machine translation or information retrieval. Existing evaluation metrics often focus on fluency rather than correctness, making it difficult to automatically detect these hallucinations. This paper aims to address this gap by developing and evaluating methods for detecting semantic hallucinations in NLG outputs.

Methods

The authors explored three primary methodologies for detecting semantic hallucinations:

  1. Fine-tuning Roberta-base on Mixtral-labeled training data (2-Class): This involved taking a pre-trained model (Roberta-base) and fine-tuning it on a dataset where each instance was labeled as either a hallucination or not. The task was framed as binary classification.
  2. Fine-tuning Roberta-base on pre-annotated validation data (6-Class): Here, they used a development dataset with more granular, fine-grained annotations (probability labels from 0 to 1 in 0.2 increments). This allowed for a multi-class classification approach, aiming to capture more nuanced levels of hallucination.
  3. Zero-shot Mixtral 8x7b: This approach involved directly using a powerful LLM (Mixtral 8x7b instruct) without any further fine-tuning. The model was prompted to classify outputs based on their semantic consistency with the source context.

Datasets

The primary dataset used in this work is SHROOM (Semantic Hallucinations Real Or Mistakes), a newly introduced benchmark for detecting semantic hallucinations. SHROOM encompasses data from three diverse NLG tasks:

  • Definition Modeling (DM): Generating definitions for words.
  • Machine Translation (MT): Translating text from one language to another.
  • Paraphrase Generation (PG): Rewriting sentences while preserving their meaning.

The dataset includes source text, target text (for DM and MT), and generated hypotheses. Annotations were performed by at least five annotators, with a majority vote determining the gold label. The paper also distinguishes between "model-aware" (where model information is available) and "model-agnostic" (where it is not) settings for the data.

Key Insights and Ideas

  • The power of LLMs in a zero-shot setting: The most striking insight is that a large, pre-trained LLM like Mixtral 8x7b can achieve competitive performance in detecting semantic hallucinations without any task-specific fine-tuning. This suggests that these models have internalized a strong understanding of semantic consistency.
  • Fine-tuning benefits, but with caveats: Fine-tuning Roberta-base on labeled data did lead to improvements in detection accuracy. However, the paper hints at potential overfitting, especially when fine-tuning on validation data over many epochs.
  • Granularity matters (to an extent): Using a 6-class classification approach (based on probability labels) for fine-tuning showed some promise, but the zero-shot LLM still outperformed it. This suggests that while fine-grained labels can be informative, the inherent capabilities of large models might be more impactful.
  • Model-agnostic vs. Model-aware: The paper explores both settings, indicating that hallucination detection is a challenge regardless of whether the model that generated the output is known.

Key Results

  • Methodology 1 (Fine-tune on labeled training data): Achieved agnostic accuracy of 76.47% and aware accuracy of 61.27% by the third epoch. The Matthews correlation coefficient (rho) peaked at 0.58 (agnostic) and 0.38 (aware).
  • Methodology 2 (Fine-tune on labeled validation data): Agnostic accuracy fluctuated around 45-51%, and aware accuracy around 47-58%. Rho varied between 0.43-0.52 (agnostic) and 0.48-0.52 (aware).
  • Methodology 3 (Zero-shot Mixtral 8x7b): Achieved agnostic accuracy of 78.73% and aware accuracy of 77.73%. The Matthews correlation coefficient (rho) was 0.50 (agnostic) and 0.48 (aware).

The zero-shot Mixtral 8x7b approach demonstrated the strongest overall performance, particularly in agnostic accuracy.

Limitations and Future Work

  • Potential Overfitting: The paper notes that fine-tuning on validation data seemed to diminish in effectiveness as epochs increased, suggesting potential overfitting.
  • Limited Fine-tuning Data: While the zero-shot approach performed well, it's possible that with more extensive and diverse labeled data, fine-tuning could yield even better results.
  • Future Work: The authors suggest exploring further optimization of fine-tuning strategies and investigating the generalizability of pre-trained models across different domains and tasks. They also highlight the need for robust detection mechanisms to ensure the accuracy and reliability of NLG systems.

Summary updated

BibTeX
@inproceedings{bahad2024nootnoot,
  author = {Sankalp Bahad and Yash Bhaskar and Parameswari Krishnamurthy},
  title = {{NootNoot At SemEval-2024 Task 6: Hallucinations and Related Observable Overgeneration Mistakes Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure