TL;DR
This paper introduces CrossCheckGPT, a novel reference-free method for universally ranking multimodal large language models (MLLMs) based on their tendency to hallucinate. By comparing outputs from multiple independent models, CrossCheckGPT provides more reliable and comparable hallucination scores across various modalities (text, image, audio-visual), outperforming existing methods. The work also introduces AVHalluBench, the first benchmark for audio-visual hallucination.
What Problem Does This Paper Address?
Generative AI models, especially multimodal foundation models, often "hallucinate" – producing outputs that are factually incorrect or contradict the provided input. This is a significant problem for applications relying on factual accuracy, as it can lead to the spread of misinformation. Existing methods for assessing hallucinations are often task-specific, require gold-standard references, or only detect hallucinations without providing a comparative ranking across different models. This limits their applicability and the ability to reliably compare the trustworthiness of various MLLMs across different domains and tasks.
Methods
The core innovation is CrossCheckGPT, a reference-free approach that leverages the idea that different independent models are unlikely to generate the same hallucinated content, while factual content should be more consistent across models.
CrossCheckGPT works by:
- Using multiple "evidence models": A set of pre-trained MLLMs are used to generate outputs related to the same input.
- Measuring cross-system consistency: The output of a "target model" (the one being evaluated) is compared against the outputs from the evidence models.
- Two consistency measures:
- CrossCheck-explicit: Evidence models generate multiple text passages. The target model's output is compared to these passages to assess consistency.
- CrossCheck-implicit: Evidence models are prompted to directly identify factual errors in the target model's output.
- Confidence-based weighting: The reliability of evidence models is considered by weighting their contributions based on their own hallucination scores (measured by SelfCheckGPT).
This approach allows for universal application across different modalities and tasks, as long as an appropriate information consistency measure can be defined.
Datasets
- WikiBio: Used for text-to-text biography generation tasks.
- MHaluBench: An existing image-to-text hallucination dataset.
- AVHalluBench: A newly introduced benchmark for audio-visual hallucination. This dataset contains 175 videos and includes carefully written hallucination-free descriptions for audio and visual content to aid in evaluation.
Key Insights and Ideas
- Cross-system consistency is a powerful signal for hallucination: The core idea that different models are less likely to agree on fabricated information than on factual information is a robust principle for evaluating MLLMs.
- Reference-free evaluation unlocks broader applicability: By not requiring gold-standard answers, CrossCheckGPT can be applied to a wider range of tasks and domains where creating such references is difficult or impossible.
- Multimodality amplifies hallucination challenges but also offers solutions: While audio-visual inputs can introduce new hallucination opportunities (e.g., misinterpreting sounds), they can also help reduce hallucinations by providing complementary information.
- The choice of consistency measure matters: CrossCheck-explicit generally outperformed CrossCheck-implicit in challenging, open-ended tasks, suggesting that generating explicit evidence passages provides a more comprehensive assessment.
Key Results
- High correlation with human judgments: CrossCheckGPT achieved a 98% Spearman's Rank Correlation (SRC) on MHaluBench and 89% SRC on AVHalluBench, significantly outperforming baselines like SelfCheckGPT and UniHD.
- Superior performance across modalities: The method demonstrated effectiveness in text-to-text, image-to-text, and audio-visual domains.
- Robustness to manipulation: CrossCheckGPT showed more stable rankings compared to SelfCheckGPT when generation temperature was varied, indicating less susceptibility to manipulation.
- AVHalluBench is the first of its kind: The introduction of this dataset fills a critical gap in evaluating audio-visual hallucination.
Limitations and Future Work
- Hallucination is expansive: The paper acknowledges that hallucination is a broad area, and their study covers a subset of possible domains. They plan to expand this with a live hallucination leaderboard.
- Evidence model similarity: The confidence-based weighting mechanism doesn't account for similarities between evidence models, which could lead to them making similar mistakes. Future work could incorporate model correlation into the weighting.
- Limited audio-visual LLMs: The availability of audio-visual LLMs for evidence generation was a constraint.
- Potential for inherited bias: CrossCheckGPT may inherit biases from the evidence models used. Practitioners should use the rankings as a guide rather than an absolute truth.
- Choosing the right consistency measure: The paper suggests a heuristic: use CrossCheck-explicit for high SelfCheckGPT scores (indicating challenging tasks) and CrossCheck-implicit for low scores. Further research could refine this.
Summary updated