MALTO at SemEval-2024 Task 6: Leveraging Synthetic Data for LLM Hallucination Detection

Federico Borra, Claudio Savelli, Giacomo Rosso, Alkis Koudounas, Flavio GiobergiaNAACL 2024

TL;DR

This paper tackles the critical problem of "hallucinations" in Large Language Models (LLMs) – instances where models generate fluent but factually incorrect text. The authors introduce a novel pipeline that leverages LLM-generated synthetic data and a weighted ensemble of three distinct models to improve hallucination detection accuracy, achieving 80.07% on the SemEval-2024 SHROOM task.

What Problem Does This Paper Address?

Natural Language Generation (NLG) models, especially advanced LLMs, have become incredibly adept at producing human-like text. However, a significant challenge remains: these models can "hallucinate," meaning they generate outputs that are semantically inaccurate or factually incorrect, despite being grammatically sound and fluent. This phenomenon can lead to the dissemination of misinformation, offensive content, or misleading statements, posing serious risks, particularly for vulnerable communities. The SemEval-2024 SHROOM challenge specifically aims to address the gap in automatically assessing the semantic correctness and meaningfulness of NLG outputs, highlighting the urgent need for robust hallucination detection methods.

Methods

The authors propose a two-pronged approach:

  1. Data Augmentation Pipeline: To overcome the scarcity of labeled data, they employ two LLM-driven techniques:
  2. LLM-aided Pseudo-Labeling: They use the SOLAR LLM in a few-shot setting to generate synthetic labels for a large corpus of unlabeled data (30,000 instances).
  3. Sentence Rephrasing: They leverage GPT-4 to rephrase both the model's output and the target "gold" reference text, creating a more diverse dataset while aiming to preserve label reliability.
  4. Weighted Ensemble Model: They combine the predictions of three distinct models, all based on the DeBERTa architecture, which has shown strong performance in Natural Language Inference (NLI) tasks.
  5. Baseline Model: A DeBERTa encoder fine-tuned on gold-labeled data.
  6. C-RLFT (Conditioned Reinforcement Learning Fine Tuning): This model is fine-tuned using a mix of gold, pseudo-labeled, and rephrased data, with different weights assigned to each data type based on their perceived reliability (gold data receives the highest weight).
  7. Sequential Model: This model undergoes a three-step fine-tuning process, starting with pseudo-labeled data, then rephrased data, and finally gold-labeled data, mimicking a curriculum learning approach.

The outputs of these three models are then fed into a single-layer network (an ensemble layer) that is trained to produce a final prediction.

Datasets

The primary dataset used is from the SemEval-2024 Task 6: SHROOM challenge. This dataset consists of:

  • Gold-labeled data (Dg): 500 instances for training and 1,500 for validation/testing, covering Definition Modeling (DM), Machine Translation (MT), and Paraphrase Generation (PG) tasks.
  • Unlabeled data (Du): 30,000 instances (10,000 per task) used for augmentation.

The authors also created augmented datasets:

  • Pseudo-labeled data (Dpl): Derived from Du using LLM-generated labels.
  • Rephrased data (Dr): Created by rephrasing sentences from Dg.

Key Insights and Ideas

  • Leveraging LLMs for Data Augmentation is Crucial: The paper strongly suggests that LLMs can be effectively used not only to generate text but also to create synthetic labels and augment existing datasets, which is vital for tasks with limited labeled data.
  • Ensembling Diverse Models Improves Robustness: Combining models trained with different strategies (baseline, C-RLFT, sequential) leads to better overall performance, particularly in improving recall. This indicates that different approaches capture different aspects of hallucination detection.
  • NLI Models are Strong Backbones for Hallucination Detection: Using DeBERTa models pre-trained on Natural Language Inference (NLI) tasks significantly boosts performance, highlighting the close relationship between NLI and hallucination detection.
  • Curriculum Learning (Sequential Fine-tuning) Shows Promise: The sequential fine-tuning approach, starting with noisier data and progressing to cleaner data, appears to be a highly effective strategy for training robust hallucination detectors.
  • Recall is More Important Than Precision for User Warnings: In a practical scenario where warnings are issued to users, a higher recall (fewer missed hallucinations) is prioritized over precision (fewer false alarms), as false negatives can be more harmful.

Key Results

The proposed ensemble method achieved an accuracy of 80.07% on the SemEval-Task 6 SHROOM leaderboard.

  • The DeBERTa + MNLI backbone generally outperformed the base DeBERTa, with the ensemble achieving the best F1 score (0.7371) and accuracy (0.8017).
  • The sequential model also showed strong performance, outperforming the baseline and C-RLFT individually.
  • The ensemble layer learned weights that favored the C-RLFT and sequential models over the baseline, indicating their greater contribution to the final prediction.

Limitations and Future Work

  • LLM Limitations: The authors acknowledge that the LLMs used for data augmentation (SOLAR and GPT-4) can themselves be prone to hallucinations, which might introduce noise into the augmented datasets. They mitigate this through weighted loss and sequential training.
  • Potential for Incorrectly Labeled Ground Truth: One qualitative example suggests that the "gold" labels themselves might occasionally be incorrect, which can impact model training and evaluation.
  • Future Work: While not explicitly detailed, the success of the ensemble and sequential training suggests that further exploration of these techniques, potentially with even more diverse model architectures and augmentation strategies, could lead to further improvements. The paper also implicitly points to the need for more robust and reliable evaluation metrics for NLG.

Summary updated

BibTeX
@inproceedings{borra2024malto,
  author = {Federico Borra and Claudio Savelli and Giacomo Rosso and Alkis Koudounas and Flavio Giobergia},
  title = {{MALTO at SemEval-2024 Task 6: Leveraging Synthetic Data for LLM Hallucination Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure