TL;DR
DreamRealm is a novel post-training framework that mitigates visual hallucinations in Multimodal Large Language Models (MLLMs) by "materializing" the model's own errors. By mining a model's specific perceptual confusions and editing images to make those hallucinations factually true, DreamRealm teaches models to precisely match responses to their supporting visual evidence, leading to significant gains in fine-grained visual perception.
What Problem Does This Paper Address?
Despite their impressive capabilities, MLLMs frequently suffer from visual hallucinations—generating descriptions that contradict the image or lack visual support. This is particularly problematic when a correct answer hinges on subtle details, like the direction a giraffe's tail is swinging or the exact number of propellers on a plane.
Current solutions typically fall into two camps:
- Text-side correction: They keep the image fixed and teach the model to prefer a "correct" response over a "hallucinated" one. This ignores the potential for supervision on the image-input side.
- Generic visual contrast: They use image retrieval or random editing to create visual differences. However, these aren't tailored to a specific model's unique "blind spots," meaning the supervision might not target the model's actual perceptual confusions.
DreamRealm addresses this by asking: What if we show the model the exact world it is imagining when it hallucinates?
Methods
The authors introduce a two-part framework: RealmForgePipe for data construction and RealmMatch for training.
1. RealmForgePipe: Materializing the Hallucination World
Instead of just correcting an error, this pipeline explores the model's "latent" hallucinations.
- Exploration: It uses high-temperature sampling and attribute enumeration (e.g., changing "left" to "right" or "standing" to "sitting") to find incorrect responses the model is highly inclined to generate.
- Materialization: It uses an image-editing model to minimally alter the original image so that the hallucinated response becomes factually correct. For example, if a model insists a giraffe's tail is "hanging naturally downward" when it is actually "swung to the left," the pipeline edits the image to show the tail hanging down.
2. RealmMatch: Hallucination-Reality Correspondence Learning
Once they have a "family" of images (one original, several edited) and responses (one ground-truth, several hallucinations), they train the model using a likelihood score matrix.
- Bidirectional Contrastive Loss: The model is trained to not only pick the right text for an image (row-wise contrast) but also to pick the right image for a given text (column-wise contrast). This forces the model to understand the fine-grained relationship between specific visual features and linguistic descriptions.
Datasets
- RealmSet: The authors introduced this new dataset, containing 28,960 human-approved edited examples across 22 perceptual categories (e.g., action, color, position, counting). It was built using hallucinations mined from the Qwen3.5-9B model.
- Evaluation Benchmarks: The model was tested on six major benchmarks: HallusionBench, MMVP, AMBER, MMHal-Bench, CV-Bench-2D, and ObjectHal.
Key Insights and Ideas
- Hallucination Assignment: The core "aha moment" is shifting from suppressing hallucinations to assigning them. By showing the model the visual conditions under which its hallucination would be true, the model learns to distinguish subtle visual cues more effectively.
- Model-Specific Supervision: Training on a model's own hallucinations is significantly more effective than training on generic errors. The authors found that while DreamRealm helps other models, the gains are highest when the mined hallucinations come from the target model itself.
- Minimal Editing is Key: By only changing the specific attribute the model is confused about, the supervision remains highly focused on the perceptual gap.
Key Results
DreamRealm achieved state-of-the-art results across multiple benchmarks, significantly outperforming specialized hallucination-mitigation methods like VCD, RLAIF-V, and various DPO-based approaches.
On the Qwen3.5-9B backbone, DreamRealm delivered:
- A 7.26 percentage point gain in HallusionBench accuracy.
- An 11.34 percentage point boost in MMVP pair accuracy.
- Significant reductions in hallucination rates on MMHal-Bench (improving from 47.92% to 27.08%).
Ablation studies confirmed that the "Full RealmForgePipe" (including latent hallucinations) and the "RealmMatch" bidirectional objective were both essential for these results.
Limitations and Future Work
- Construction Cost: The pipeline requires significant compute for image generation and, crucially, human review to ensure the edited images actually match the target responses.
- Editing Capabilities: The framework is limited by the state of current image-editing models. Complex scenes or very subtle physical interactions remain difficult to edit convincingly.
- Future Directions: The authors suggest exploring more automated verification methods to reduce human-in-the-loop costs and applying this "materialization" logic to video-language models where temporal hallucinations are a major hurdle.
Summary updated