TL;DR
This paper explores how well sentence embeddings can detect "hallucinations" (generated text that is inconsistent with the source) in machine translation, paraphrasing, and definition generation tasks. They found that sentence embeddings, when combined with a cosine similarity threshold, surprisingly outperformed large language models (LLMs) for this specific task, achieving up to 78% accuracy.
What Problem Does This Paper Address?
The increasing sophistication of text generation models, like GPT-4 and Llama 2, has brought about a significant challenge: hallucinations. These are segments of text that sound natural but contain information that contradicts or is absent from the original context. This issue makes it difficult to deploy these powerful models in real-world applications where factual accuracy is paramount. The SHROOM shared task at SemEval-2024 specifically aims to tackle this problem by developing methods to detect these hallucinations in generated text.
Methods
The authors proposed two main approaches for hallucination detection:
- Embedding-Based Approach: This method leverages sentence embeddings to capture the semantic meaning of both the generated text (hypothesis) and the original context (reference). They then calculate the cosine similarity between these embeddings. A predefined threshold is used: if the similarity falls below this threshold, the hypothesis is flagged as a hallucination. They experimented with several pre-trained sentence embedding models, including Sentence-T5 XL, BGE-base/large, and E5-base/large.
- LLM-Based Approach: This approach utilizes large language models (LLMs) like Llama 2 and Mistral. They employed two distinct prompts, inspired by the SelfCheckGPT framework, to guide the LLMs in classifying whether a generated sentence is a hallucination.
Datasets
The SHROOM task provided datasets for three NLG tasks: Machine Translation (MT), Paraphrasing Generation (PG), and Definition Modeling (DM). These datasets were divided into model-aware and model-agnostic tracks. The authors used trial, development (dev), and test datasets for their experiments. The trial dataset contained 501 samples, while the dev and test datasets had 499 and 1500 samples, respectively. Each sample included a generated hypothesis, a reference (source or target depending on the task), and annotations from multiple human annotators indicating whether the hypothesis was a hallucination.
Key Insights and Ideas
- Embeddings Shine: The most striking finding is that sentence embeddings, a seemingly simpler approach, outperformed LLMs in detecting hallucinations for this task. This suggests that for short, concise text segments, capturing semantic similarity directly might be more robust than relying on LLMs' more complex reasoning.
- Thresholding is Key: The effectiveness of the embedding-based approach hinges on finding the right cosine similarity threshold. The authors demonstrated that this threshold can be optimized to maximize classification accuracy, and that this optimization is relatively consistent across different embedding models.
- LLMs Struggle with Simple Prompts: While LLMs are powerful, their performance on this specific hallucination detection task, using the tested prompts, was not as strong as the embedding-based methods. This indicates that designing effective prompts for LLMs to reliably detect subtle inconsistencies remains a challenge.
Key Results
The authors' best performing system, using the Sentence-T5 XL embedding model with an optimized threshold, achieved an accuracy of 78%. This placed them at position 22 out of 48 in the model-agnostic track and 22 out of 45 in the model-aware track. Their embedding-based approaches consistently outperformed the baseline LLM system. The LLM-based approaches, while tested with different prompts, did not achieve comparable results and did not exceed the baseline.
Limitations and Future Work
The authors acknowledge that their LLM-based approach did not yield conclusive results, and further experiments are needed to draw firm conclusions about prompt effectiveness. They also mention that the ranking in the SemEval task is very tight, with small differences in accuracy separating many participants. For future work, they suggest exploring fine-tuning the embedding models further to potentially improve performance.
Summary updated