TL;DR
This paper presents the SHROOM-INDElab system for detecting hallucinations in Large Language Model (LLM) outputs. Their approach leverages prompt engineering with context-specific definitions and a two-stage zero-shot/few-shot classification strategy, achieving competitive results in the SemEval-2024 Task 6. They found that explicit definitions of hallucination were crucial, and surprisingly, a zero-shot approach often outperformed few-shot methods with automatically generated examples.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly used for complex natural language tasks, but they are prone to "hallucinations" – generating outputs that are factually incorrect, nonsensical, or not faithful to the input. Detecting these hallucinations is crucial for building reliable AI systems. This paper tackles the challenge of automatically identifying these hallucinations, particularly in the context of LLMs generating rationales for other tasks, which can themselves be prone to hallucination. They aim to evaluate how prompt engineering techniques, specifically zero-shot and few-shot learning, can be effectively applied to this hallucination detection problem.
Methods
The SHROOM-INDElab system employs a two-stage workflow for hallucination detection:
- Stage 1 (Zero-Shot Classification): The system uses prompt engineering to instruct an LLM to classify a given generated text as "Hallucination" or "Not Hallucination." This is done in a zero-shot manner, meaning no examples are provided in the prompt. Crucially, the prompt includes context-specific definitions of the task, the role the LLM should assume (e.g., a lexicographer for definition modeling), and a general definition of hallucination.
- Stage 2 (Few-Shot Classification): The classifications generated in Stage 1 are used as examples for a few-shot classifier. This stage aims to improve performance by providing the LLM with a small number of labeled examples.
Key to their approach is the careful design of prompts. They incorporate task-specific definitions, role-playing instructions for the LLM, and a consistent definition of hallucination across tasks. For probability estimation, they use temperature sampling and majority voting over multiple LLM inferences. Example selection for Stage 2 is guided by maximizing negative entropy and a diversity-consistency trade-off.
Datasets
The system was evaluated on the SemEval-2024 Task 6 dataset. This dataset consists of data points containing:
- The specific task the LLM was performing.
- The input text given to the LLM.
- A target text (an acceptable output).
- The generated text produced by the LLM.
The competition provided unlabeled training data and labeled validation and test datasets. The task featured two tracks: model-agnostic and model-aware.
Key Insights and Ideas
- The Power of Explicit Definitions: The paper strongly suggests that providing clear, explicit definitions of what constitutes a "hallucination" is vital for accurate detection. Removing this definition significantly degraded performance.
- Zero-Shot Outperforms Few-Shot (Surprisingly): In their experiments, a zero-shot approach often yielded better accuracy than a few-shot approach that used automatically generated examples. This challenges the common assumption that more examples always lead to better performance, especially when example generation is imperfect.
- Role-Playing Enhances Performance: Instructing the LLM to adopt specific personas (e.g., a lexicographer) for classification tasks improved performance, highlighting the effectiveness of role-play prompting.
- Consistency with Human Judgments: The system's classifications showed a moderate to strong correlation with human judgments, indicating its reliability.
Key Results
- The SHROOM-INDElab system achieved fourth-best performance in the model-agnostic track and sixth-best in the model-aware track of SemEval-2024 Task 6.
- Their best-performing submission used
gpt-4-0125-previewwith one example per label, 20 samples for majority voting, and a temperature of 1.2. - The ablation study revealed that removing selected examples led to better accuracy, while removing the concept definition of hallucination led to poorer accuracy and Spearman's ρ.
- The system's agreement with human labeling increased as the human labeling consistency (certainty) increased.
Limitations and Future Work
- Example Selection Nuances: The paper notes that the exclusion of selected examples led to better accuracy, suggesting that the way examples are selected and included in prompts warrants further investigation. The current method might not be optimal.
- Future Directions: The authors plan to explore this approach for evaluating natural language rationale generation and for knowledge graph evaluation and refinement. They also intend to investigate the impact of different prompt engineering strategies and LLM architectures on hallucination detection.
Summary updated