UMUTeam at SemEval-2024 Task 6: Leveraging Zero-Shot Learning for Detecting Hallucinations and Related Observable Overgeneration Mistakes

Ronghao Pan, José Antonio García-díaz, Tomás Bernal-beltrán, Rafael Valencia-garcíaNAACL 2024

TL;DR

This paper tackles the critical challenge of automatically detecting "hallucinations" – grammatically correct but semantically incorrect outputs from Large Language Models (LLMs). The UMUTeam leveraged zero-shot learning with LLMs like Tulu, Mistral, and LLaMa-2, achieving competitive results in the SemEval-2024 SHROOM task by framing hallucination detection as a binary classification problem.

What Problem Does This Paper Address?

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP), demonstrating remarkable capabilities in understanding and generating human-like text. However, a significant issue with current LLMs is their tendency to produce outputs that are fluent and grammatically sound but factually incorrect or unsupported by the source material. This phenomenon, known as "hallucination," is particularly problematic in applications where accuracy is paramount, such as machine translation or information retrieval. Existing evaluation metrics often prioritize fluency over correctness, exacerbating this problem. The SemEval-2024 Task 6 (SHROOM) specifically aims to address this by developing systems to automatically detect these hallucinations and related overgeneration errors in LLM outputs.

Methods

The UMUTeam's approach is rooted in zero-shot learning (ZSL), a technique that allows LLMs to perform tasks without explicit training on them. They utilized prompt engineering to guide pre-trained LLMs – specifically LLaMa-2, Mistral, and Tulu – to classify whether a generated sentence is supported by a given context. The core idea is to craft prompts that instruct the LLM to output a simple "yes" or "no" answer indicating support.

The system architecture involves:

  1. Prompting: A carefully designed prompt is fed to the LLM, containing the context and the sentence to be evaluated. The prompt explicitly asks the LLM to answer with "yes" or "no" at the beginning of its response.
  2. Filtering: The LLM's response is then processed by a "filter" module. This module extracts the first word of the response and classifies it as either "Hallucination" (if the answer is "no") or "Not Hallucination" (if the answer is "yes").
  3. Correlation Value: To further evaluate the LLM's confidence, the log probability of the first token of the generated response is extracted.

This approach was applied in two configurations:

  • Model-aware: The system has access to the model that produced the text.
  • Model-agnostic: The system only has access to the input text and the generated output, without knowledge of the generating model.

Datasets

The authors utilized the dataset provided by the SHROOM task organizers, which includes unlabeled train data, trial data, and validation data for both model-aware and model-agnostic setups. For evaluation, they exclusively used the provided validation data. The paper does not introduce any new datasets.

Key Insights and Ideas

  • Zero-Shot Learning is Viable for Hallucination Detection: The study demonstrates that LLMs, even without specific fine-tuning for hallucination detection, can effectively identify semantically incorrect outputs through well-crafted prompts.
  • Tulu Shines: The Tulu LLM consistently outperformed Mistral and LLaMa-2 in both model-aware and model-agnostic evaluations, suggesting its architecture or training data is particularly well-suited for this task.
  • Prompt Engineering is Crucial: The effectiveness of the ZSL approach heavily relies on the design and wording of the prompts to elicit accurate binary classifications from the LLMs.
  • Model-Agnostic vs. Model-Aware: While both setups yielded competitive results, the model-aware setup generally saw slightly higher accuracy, which is expected given the additional information available.

Key Results

The UMUTeam achieved the following results in the SemEval-2024 SHROOM task:

  • Model-aware setup: Ranked 18th out of 46 teams with an accuracy of 78.4% and a rho correlation of 0.506895.
  • Model-agnostic setup: Ranked 29th out of 49 participants with an accuracy of 76.9333% and a rho correlation of 0.560945.

In their internal validation experiments, the Tulu model achieved:

  • Model-aware: Accuracy of 76.6467% and rho of 0.521104.
  • Model-agnostic: Accuracy of 73.7475% and rho of 0.553962.

The confusion matrices revealed that while the Tulu model was good at identifying "not hallucinations" (around 80-90% accuracy), it still confused hallucinations with non-hallucinations with a probability of around 30-40% in the model-agnostic setup.

Limitations and Future Work

  • Limitations: The paper acknowledges that their system, while competitive, is not perfect. The confusion matrices highlight that misclassifications still occur, particularly in distinguishing between hallucinations and non-hallucinations. The authors also note that their system is 2.866% worse in precision and 19.25% worse in rho compared to the best team in the model-aware setup.
  • Future Work: The authors propose investigating hallucination detection in the political domain. They aim to generate political discourse that imitates politicians from different political wings and then identify the generated hallucinations. This suggests a future direction towards domain-specific hallucination detection and generation.

Summary updated

BibTeX
@inproceedings{pan2024umuteam,
  author = {Ronghao Pan and José Antonio García-díaz and Tomás Bernal-beltrán and Rafael Valencia-garcía},
  title = {{UMUTeam at SemEval-2024 Task 6: Leveraging Zero-Shot Learning for Detecting Hallucinations and Related Observable Overgeneration Mistakes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure