TL;DR
This paper presents a novel three-stage framework for automatically detecting and localizing hallucinations in LLM outputs, achieving top performance in the SemEval-2025 Mu-SHROOM challenge. The system leverages context retrieval, hallucination detection, and span mapping, enhanced by prompt optimization, demonstrating that even simple text extraction methods can be highly effective when paired with good context.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly used for knowledge-intensive tasks like question answering. However, they often "hallucinate," generating false or unverified information while maintaining a coherent output. Detecting these hallucinations is crucial for building trust in LLMs, but existing methods often provide only binary or scalar outputs, lacking the ability to pinpoint where in the text the hallucination occurs. This fine-grained localization is essential for fact-checking and improving LLM performance. The SemEval-2025 Task 3 (Mu-SHROOM) specifically addresses this by challenging participants to identify hallucinated text spans and their associated confidence across 14 languages.
Methods
The UCSC team developed a three-stage pipeline:
- Context Retrieval: The system first retrieves relevant external knowledge sources to provide context for verifying the LLM's answer. This can involve querying with the original question or claims from the answer.
- Hallucinated Content Detection: This stage identifies segments of the LLM's answer that are not supported by the retrieved context. The paper explores three implementations:
- Direct Text Extraction: Prompting an LLM to directly identify and extract segments not verifiable by the context.
- Verification with Knowledge Graph: Parsing context into a knowledge graph and using an LLM to verify individual facts from the answer against it.
- Minimal Cost Revision: Using a reasoning LLM to correct the answer with the fewest possible changes, with the differences indicating hallucinations.
- Span Mapping: The identified hallucinated content segments are then mapped back to precise character-level spans in the original LLM output using methods like substring matching, fact-to-span mapping, or edit distance.
A key innovation is the use of Prompt Optimization with MiPROv2, a Bayesian search framework, to automatically find optimal prompts for the hallucination detection stage, significantly improving performance. They also explored a Multi-System Combination approach, aggregating outputs from multiple systems to generate more robust "hard labels."
Datasets
The primary dataset used is the SemEval-2025 Task 3 (Mu-SHROOM) dataset, which provides multilingual question-answer pairs with human-annotated ground truth labels for hallucinated spans. The authors also created their own internal annotations for the English validation set to guide system development.
Key Insights and Ideas
- Context is King: Retrieving relevant external context is paramount for effective hallucination detection, significantly boosting performance.
- Simplicity Wins (Sometimes): Direct text extraction, a seemingly simpler method, often outperforms more complex approaches like knowledge graph verification when paired with good context.
- Prompt Optimization is Powerful: Automatically optimizing prompts for the hallucination detection step yields substantial performance gains, improving reliability and accuracy.
- Human Annotation Variability: The study highlights significant inconsistencies in human annotation of hallucinated spans, even when agreeing on underlying facts. This suggests that well-defined annotation frameworks are crucial for both human and automatic labeling.
- Multilingual Challenges: While the system performs well across many languages, performance in Chinese was notably lower, indicating potential challenges with complex datasets and model familiarity.
Key Results
The UCSC system achieved top performance in the SemEval-2025 Mu-SHROOM challenge, ranking #1 in average position across all 14 languages. Specifically, they achieved:
- Top two positions in 11 out of 14 languages for the IoU metric.
- Top two positions in 10 out of 14 languages for the Corr metric.
- The system combination approach improved the correlation score by an average of 5% across languages, though it slightly decreased IoU.
- Prompt optimization led to significant performance gains, with GPT-40 models showing strong results.
Limitations and Future Work
The paper acknowledges several limitations:
- Chinese Performance: The system underperforms in Chinese, suggesting a need for better handling of complex languages and datasets.
- Cost-Effectiveness: Achieving the highest performance relies on sophisticated LLMs and extensive prompt optimization, which may not be cost-effective for real-world applications.
- Knowledge Graph Unreliability: The knowledge graph verification step, while conceptually sound, proved unreliable for span mapping, leading to noise.
Future work could explore more robust knowledge graph integration, investigate cost-effective prompt optimization strategies, and develop better methods for handling language-specific nuances. The authors also suggest that a more well-defined annotation framework could benefit both automatic and human labeling of hallucinations.
Summary updated