TL;DR
This paper introduces HaRMoNEE, a system that achieved top rankings in SemEval-2024 Task 6 for detecting hallucinations in Large Language Model (LLM) outputs. They explored two main approaches: fine-tuning existing transformer models and prompt-tuning large LLMs, demonstrating that careful data preprocessing and prompt engineering are crucial for effective hallucination recognition.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly powerful, but they have a tendency to "hallucinate" – generating outputs that are factually incorrect, nonsensical, or not supported by the input. This is a significant problem for deploying LLMs in real-world applications where accuracy and trustworthiness are paramount. SemEval-2024 Task 6, specifically the SHROOM task, aimed to address this by challenging researchers to develop systems that can automatically detect these hallucinations in LLM-generated text. The task was divided into two subtasks: model-aware (where information about the model that generated the output is available) and model-agnostic (where it is not).
Methods
The HaRMoNEE team explored two primary methods:
- Fine-tuning Transformer Models: They adapted existing transformer-based models, pre-trained on tasks like Natural Language Inference (NLI) and hallucination detection, for the SHROOM task. This involved preparing the input data by concatenating relevant fields (like source text and hypothesis) and then fine-tuning the model on the provided labeled datasets. They experimented with different training strategies, including varying the number of training epochs and the combination of datasets used.
- LLM Prompt-Tuning: This approach treated large LLMs (like GPT-3.5 and GPT-4) as black boxes. Instead of fine-tuning the model's weights, they focused on crafting effective prompts to elicit accurate hallucination detection. They experimented with different prompt structures, including varying the number of input fields provided (e.g., source, target, hypothesis) and the level of detail in the instructions. A key aspect was outputting responses in JSON format to ensure structured and interpretable results.
Both approaches involved significant pre-processing of the input data and post-processing of the model outputs to align with the task's evaluation metrics.
Datasets
The primary dataset used was the SHROOM dataset, specifically designed for the SemEval-2024 Task 6. This dataset is split into two subtasks: model-aware and model-agnostic, each with its own training, validation, and test sets. The dataset contains examples from three different tasks: Definition Modeling (DM), Machine Translation (MT), and Paraphrase Generation (PG).
For pre-training and fine-tuning, the authors also leveraged established NLI datasets like SNLI (Stanford Natural Language Inference) and MultiNLI (Multi-Genre Natural Language Inference), as well as a paraphrase identification dataset called PAWS (Paraphrase Adversaries from Word Scrambling).
Key Insights and Ideas
- Data Pruning is Crucial: A surprising finding was that including fewer, more relevant fields in the input to the models often led to better performance. For instance, in fine-tuning, ignoring the "source" field when the "target" was available improved results. Similarly, for prompt-tuning, less verbose prompts that omitted task definitions or less critical fields performed better.
- Fine-tuning Outperforms Prompt-Tuning for Model-Aware: The HaRMONEE team's winning submission for the model-aware subtask was achieved through fine-tuning a transformer model. This suggests that when specific model information is available, fine-tuning can leverage it more effectively than prompt-based approaches.
- Prompt Engineering Matters for Model-Agnostic: For the model-agnostic subtask, prompt-tuning with GPT-4 yielded the best results. This highlights the power of carefully crafting prompts to guide LLMs in complex reasoning tasks like hallucination detection, especially when model-specific context is absent.
- Task-Specific Nuances: The performance varied across the different example tasks (DM, MT, PG). The authors noted that Definition Modeling (DM) was particularly challenging, potentially due to greater variability in the semantic richness of the target, source, and hypothesis fields compared to other tasks.
Key Results
- Model-Aware Subtask (#1 Ranking): The HaRMoNEE fine-tuning approach achieved the top rank for the model-aware subtask, reaching an accuracy of 0.813.
- Model-Agnostic Subtask (#10 Ranking): The HaRMoNEE prompt-tuning approach with GPT-4 achieved the 10th rank for the model-agnostic subtask, with an accuracy of 0.782.
- Baseline Improvements: Both approaches significantly outperformed the provided baselines. The fine-tuning approach improved accuracy by 0.068 points over the baseline for the model-aware subtask, while the prompt-tuning approach improved accuracy by 0.133 points over the baseline for the model-agnostic subtask.
Limitations and Future Work
The authors acknowledge that they did not fully explore certain avenues, including:
- Pseudo-labeling: Training on automatically generated labels for unlabeled data.
- Additional Datasets: Experimenting with more diverse datasets beyond SNLI, MultiNLI, and PAWS, such as adversarial translation or word disambiguation datasets.
- Advanced Techniques: Exploring methods like dense paraphrasing and frame saturation, as proposed in prior work.
They suggest that future work could benefit from harmonizing approaches by focusing on fewer, clearer "voices" or strategies, implying that simplifying the problem space can lead to more robust solutions.
Summary updated