TU Wien at SemEval-2024 Task 6: Unifying Model-Agnostic and Model-Aware Techniques for Hallucination Detection

Varvara Arzt, Mohammad Mahdi Azarbeik, Ilya Lasy, Tilman Kerl, Gábor RecskiNAACL 2024

TL;DR

This paper tackles the critical problem of "hallucinations" in neural text generation, where models produce fluent but factually incorrect outputs. The authors propose a unified approach combining model-agnostic (e.g., word alignment, BERT fine-tuning) and model-aware (e.g., analyzing internal LLM states) techniques, achieving strong results in the SemEval-2024 SHROOM shared task.

What Problem Does This Paper Address?

Natural Language Generation (NLG) models, particularly Large Language Models (LLMs), are increasingly capable of producing human-like text. However, a significant challenge is their tendency to "hallucinate" – generating outputs that are grammatically correct and fluent but factually inaccurate or inconsistent with the source information. This phenomenon severely limits the practical utility of these models in real-world applications where factual correctness is paramount. Identifying and mitigating these hallucinations automatically is a substantial research challenge. The SemEval-2024 SHROOM shared task specifically aimed to address this by evaluating systems on their ability to detect hallucinations in LLM outputs across various tasks like definition modeling, machine translation, and paraphrase generation. A key hurdle in this domain is the scarcity of high-quality labeled data for training and evaluating hallucination detection models.

Methods

The authors employed a dual strategy, combining model-agnostic and model-aware techniques.

Model-Agnostic Methods: * Word Alignment: This approach leverages semantic similarity between the generated text and the reference text. They developed methods that align words and calculate sentence similarity based on word-level similarity metrics (e.g., Levenshtein distance, BERTScore). The core idea is that a higher semantic similarity between the generated output and the reference implies a lower probability of hallucination. * BERT-based Fine-tuning: They fine-tuned a pre-trained BERT model (specifically, the Vectara hallucination detection model) on hallucination detection tasks. This model was trained on a cross-encoder architecture and leveraged existing datasets. * Automatic Data Annotation: To overcome the lack of labeled training data, they used GPT-3.5 Turbo for zero-shot prompting and BERTScore to generate probabilistic labels for hallucination.

Model-Aware Methods: * Hidden States Analysis: This involved analyzing the internal representations (activations) of the LLM's hidden layers. They used a feedforward neural network trained on these activations to predict hallucination probabilities, inspired by methods like SAPLMA. * Attention Flow Analysis: This technique examines the attention weights within the LLM to understand which parts of the input text are most influential for generating the output. By analyzing these attention patterns, they aimed to identify potential indicators of hallucination.

Ensemble Methods: To further improve performance, they combined the outputs of individual systems using various ensemble techniques, including logistic regression, linear regression, simple voting, and probability fusion.

Datasets

The primary dataset used was the SHROOM validation and test sets. This dataset includes LLM outputs for definition modeling (DM), machine translation (MT), and paraphrase generation (PG) tasks, paired with input source text and 'gold' reference text. The validation set provided binary labels (Hallucination/Not Hallucination) and hallucination probabilities. The authors also generated their own labels using GPT-3.5 and BERTScore for training purposes.

Key Insights and Ideas

  • Unified Approach is Key: The paper demonstrates that combining model-agnostic and model-aware techniques, especially through ensembling, leads to superior performance compared to relying on a single approach.
  • Automatic Labeling is Crucial: The scarcity of labeled data is a major bottleneck. Their work highlights the effectiveness of using LLMs like GPT-3.5 and metrics like BERTScore for generating training labels, even with potential biases.
  • Subjectivity in Hallucination: The qualitative analysis reveals that defining and annotating hallucinations can be subjective, especially for tasks like paraphrase generation where multiple correct outputs are possible. This ambiguity poses a challenge for both human annotators and automated systems.
  • Model-Aware Challenges: Model-aware methods, while promising in theory, faced practical challenges, including difficulties in reproducing model outputs and potential issues with the quality of automatically generated labels.

Key Results

The authors' best-performing systems achieved:

  • Model-Agnostic Track: An accuracy of 81.7% with the mm-linreg ensemble model.
  • Model-Aware Track: An accuracy of 80.6% with the vectara-val fine-tuned model.

These results placed their team 3rd in the model-aware track and 8th in the model-agnostic track among all participants in the SemEval-2024 SHROOM shared task. The finetuning of the Vectara model on the validation set showed significant improvements in accuracy and correlation.

Limitations and Future Work

  • Reproducibility Issues: The authors encountered difficulties in reproducing the exact outputs of provided Huggingface models for MT and DM tasks, which impacted the alignment of labels for model-aware methods.
  • Annotation Subjectivity: The inherent subjectivity in defining and annotating hallucinations, particularly for tasks with multiple valid outputs, complicates evaluation and model development. A clearer definition of the annotation task could enhance inter-annotator agreement and model performance.
  • Language Specificity: Their models were primarily trained on English, which limited their ability to fully grasp grammatical nuances in other languages (e.g., Russian) without explicit training on those languages or their source texts.
  • Future Work: The paper suggests that incorporating source text ('src') into the training and evaluation pipeline, especially for MT tasks, could improve performance. Exploring more sophisticated ensemble techniques and further refining automatic data annotation strategies are also potential avenues.

Summary updated

BibTeX
@inproceedings{arzt2024tu,
  author = {Varvara Arzt and Mohammad Mahdi Azarbeik and Ilya Lasy and Tilman Kerl and Gábor Recski},
  title = {{TU Wien at SemEval-2024 Task 6: Unifying Model-Agnostic and Model-Aware Techniques for Hallucination Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure