ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models

Yahan Tu, Rui Hu, Jitao Sang

Abstract

Hallucination poses a persistent challenge for multimodal large language models (MLLMs). However, existing benchmarks for evaluating hallucinations are generally static, which may overlook the potential risk of data contamination. To address this issue, we propose ODE, an open-set, dynamic protocol designed to evaluate object hallucinations in MLLMs at both the existence and attribute levels. ODE employs a graph-based structure to represent real-world object concepts, their attributes, and the distributional associations between them. This structure facilitates the extraction of concept combinations based on diverse distributional criteria, generating varied samples for structured queries that evaluate hallucinations in both generative and discriminative tasks. Through the generation of new samples, dynamic concept combinations, and varied distribution frequencies, ODE mitigates the risk of data contamination and broadens the scope of evaluation. This protocol is applicable to both general and specialized scenarios, including those with limited data. Experimental results demonstrate the effectiveness of our protocol, revealing that MLLMs exhibit higher hallucination rates when evaluated with ODE-generated samples, which indicates potential data contamination. Furthermore, these generated samples aid in analyzing hallucination patterns and fine-tuning models, offering an effective approach to mitigating hallucinations in MLLMs.

Introduction

Hallucination—where a model generates content that is not grounded in the provided visual input—remains a significant barrier to the deployment of Multimodal Large Language Models (MLLMs). While numerous benchmarks exist to quantify this phenomenon, most rely on static datasets like COCO. The long-term availability and widespread use of these datasets introduce a significant risk of data contamination. Models may achieve high scores not through genuine visual understanding, but by memorizing statistical priors or specific image-text pairs encountered during pre-training.

Data contamination in static benchmarks leads to inflated performance compared to open-set evaluations.

As illustrated in Figure 1, performance on "Close Set" (previous test sets) often exceeds that on "Open Set" (newly generated or out-of-distribution) samples. This suggests that current evaluations may overstate model reliability due to memorization rather than true visual grounding.

The ODE Protocol

To address the limitations of static benchmarks, we introduce ODE (Open-Set Evaluation), a dynamic protocol that generates novel, synthetic test cases. The ODE pipeline consists of four primary stages:

  1. Graph-Based Conceptual Elements Modeling: We construct a weighted graph where nodes represent object concepts (entities), environments, and attributes. Edges represent the distributional relationships (co-occurrence frequencies) between these elements in real-world scenarios.
  2. Semantic Scene Construction: Using the graph, we select concept combinations based on four distributional criteria: Standard (common), Long-tailed (rare), Random (arbitrary), and Fictional (impossible in reality). This allows for a comprehensive stress-test of model grounding versus statistical bias.
  3. Image Generation and Filtering: Text-to-image models (e.g., Stable Diffusion) generate images based on the constructed semantic scenes. To ensure the "ground truth" is accurate, an automated filtering process using object detection tools removes images where the intended concepts failed to render correctly.
  4. Inquiry Construction: Structured queries are automatically generated for both generative tasks (e.g., "Describe this image") and discriminative tasks (e.g., "Is there a [Object] in the picture?").
The ODE pipeline: from graph-based concept modeling to synthetic image generation and inquiry construction.

Experimental Results and Analysis

We evaluated several state-of-the-art MLLMs, including LLaVA-1.5, MiniGPT-4, and InstructBLIP, using the ODE protocol.

Evidence of Data Contamination

Our results confirm a significant performance drop when models are tested on ODE-generated samples compared to traditional benchmarks. For instance, MiniGPT-4's F1 score on the discriminative existence task dropped from 70.3 on COCO2014 to 51.0 on ODE. This "performance cliff" strongly suggests that models rely heavily on memorized correlations from their training data.

Distributional Sensitivity

Models exhibit varying degrees of hallucination across different distributional criteria. While performance is relatively stable on "Standard" pairings, it degrades sharply on "Random" and "Fictional" combinations. This indicates that MLLMs often default to statistical priors (e.g., assuming a chair exists in a room scene) rather than performing rigorous visual verification.

Mitigation through Fine-Tuning

Beyond evaluation, ODE-generated samples serve as high-quality training data. Fine-tuning LLaVA-1.5 on these diverse, out-of-distribution cases significantly improved its robustness. Specifically, the model's recall on existence tasks increased from 71.0% to 96.0%, demonstrating that exposure to varied synthetic distributions can effectively mitigate hallucination.

Conclusion

The ODE protocol provides a robust, dynamic framework for evaluating and mitigating hallucinations in MLLMs. By moving beyond static, potentially contaminated benchmarks, ODE reveals the true extent of model reliance on statistical priors. Our findings highlight the necessity of open-set evaluation and the potential of synthetic data to enhance the visual grounding of multimodal models.

Summary updated

Abstract

Hallucination poses a persistent challenge for multimodal large language models (MLLMs). However, existing benchmarks for evaluating hallucinations are generally static, which may overlook the potential risk of data contamination. To address this issue, we propose ODE, an open-set, dynamic protocol designed to evaluate object hallucinations in MLLMs at both the existence and attribute levels. ODE employs a graph-based structure to represent real-world object concepts, their attributes, and the distributional associations between them. This structure facilitates the extraction of concept combinations based on diverse distributional criteria, generating varied samples for structured queries that evaluate hallucinations in both generative and discriminative tasks. Through the generation of new samples, dynamic concept combinations, and varied distribution frequencies, ODE mitigates the risk of data contamination and broadens the scope of evaluation. This protocol is applicable to both general and specialized scenarios, including those with limited data. Experimental results demonstrate the effectiveness of our protocol, revealing that MLLMs exhibit higher hallucination rates when evaluated with ODE-generated samples, which indicates potential data contamination. Furthermore, these generated samples aid in analyzing hallucination patterns and fine-tuning models, offering an effective approach to mitigating hallucinations in MLLMs.
BibTeX
@inproceedings{tu2025ode,
  author = {Yahan Tu and Rui Hu and Jitao Sang},
  title = {{ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01847},
}
Figure