TL;DR
This paper tackles the challenge of detecting hallucinations in large language models (LLMs) at a character level, a more granular approach than typical sentence-level detection. The YNU-HPCC team achieved top rankings by combining Retrieval-Augmented Generation (RAG) with zero-shot learning, enabling LLMs to leverage external knowledge for more accurate hallucination identification.
What Problem Does This Paper Address?
Large language models (LLMs) are prone to "hallucinations" – generating plausible-sounding but factually incorrect or fabricated information. This is a significant issue, especially in open-domain tasks like question answering and summarization, where LLMs might produce answers that contradict provided context or external knowledge. Hallucinations can be categorized as intrinsic (conflicting with source content) or extrinsic (unverifiable from source content). While many detection methods exist, they often focus on entire responses or sentences, or struggle when factual documents are hard to retrieve. This paper addresses the specific challenge of detecting hallucinations at the character level and assigning a likelihood score to each segment, a more nuanced requirement than binary classification.
Methods
The YNU-HPCC team employed a two-pronged approach:
- MRC Combined with RAG for Hallucination Detection: Initially, they combined Machine Reading Comprehension (MRC) with a Retrieval-Augmented Generation (RAG) strategy. This involved manually querying LLMs and then filtering responses for conciseness and directness. They used LLaMA3 for its cross-linguistic capabilities and employed BIO tagging for character-level interval detection. To address the need for probability scores, they used the softmax value of the hallucination end token as a proxy.
- Zero-shot Combined with RAG for Hallucination Detection: This evolved into their primary method. They created question-answer pair documents and used prompt engineering to guide LLMs (specifically OpenAI-01, Claude-3.5, Gemini, and DeepSeek V3) in directly categorizing model output text. The LLMs were instructed to output both hard (binary) and soft (probability) labels. To align with the task's requirement for probability values (1/12 to 1.0), they instructed the models to select values within this range, reflecting the 12 annotators' judgments. This approach leverages external knowledge through RAG and the generalizability of zero-shot learning.
Datasets
- RAGTruth: Used for fine-tuning the LLaMA model in the MRC approach. This dataset explicitly labels hallucination intervals numerically.
- HaluEVAL: Also used for fine-tuning the LLaMA model. This dataset identifies text segments containing hallucinations but does not specify precise intervals.
- SemEval-2025 Task 3 Test Set: The primary dataset for evaluation, consisting of multilingual instruction-tuned LLM outputs. The paper notes the Chinese test set had 48040 characters, and the English test set had 36745 characters.
Key Insights and Ideas
- Character-level detection is crucial: The paper highlights that focusing on character-level detection and probability scores provides a more nuanced and accurate understanding of hallucinations than sentence-level binary classification.
- Zero-shot learning is powerful for this task: By carefully crafting prompts, LLMs can effectively leverage external knowledge (via RAG) to detect and score hallucinations without task-specific fine-tuning, making the approach flexible and generalizable.
- Prompt engineering is key: The success of the zero-shot approach heavily relies on how prompts are designed to guide the LLM's reasoning and output format, especially for probability assignments.
- LLM performance varies significantly: Different LLMs exhibit distinct strengths and weaknesses in hallucination detection, with OpenAI-01 showing particularly strong performance.
- Multilingual challenges persist: While the approach is designed to be multilingual, differences in text structure (e.g., Markdown syntax in Chinese outputs) and data retrieval quality can impact performance across languages.
Key Results
The YNU-HPCC team achieved impressive results:
- First place in Chinese for Task 3, with OpenAI-01 achieving an IoU score of 0.5540 and a Cor score of 0.3518.
- Fifteenth place in English for Task 3, with an IoU score of 0.4807 and a Cor score of 0.4075.
- Figure 3 and 4 show that while Claude-3.5 performed well at sentence-level detection, OpenAI-01 achieved higher IoU scores due to fewer false positives.
- The paper notes a disproportionate increase in undetected sentences for Gemini and DeepSeek V3 in the Chinese track compared to the English track.
Limitations and Future Work
- Granularity mismatch: The paper acknowledges that while they aimed for character-level detection, their approach (and the LLMs' natural output) often aligns with sentence or clause levels, which is a slight deviation from the "minimum" span requirement.
- Probability assignment challenges: Fitting binary classification task probabilities to 12 individual votes proved challenging. Exploring multiple rounds of sampling was considered but deemed too costly.
- Data quality impact: Discrepancies in the quality of augmented documents and retrieval issues in the English test set potentially impacted performance compared to the Chinese track.
- Future Work: The team plans to explore a knowledge graph-based approach for self-checking LLM-generated answers, aiming for more intelligent models that can mitigate the societal and economic harm caused by incorrect outputs.
Summary updated