3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding

Makanjuola Adekunmi Ogunleye, Eman Abdelrahman, Ismini Lourentzou

TL;DR

3D-VCD is the first training-free, inference-time framework designed to mitigate hallucinations in 3D embodied Large Multimodal Models (MLLMs). By contrastively decoding predictions between an original 3D scene graph and a structurally or semantically distorted "negative" context, it suppresses language-prior biases and significantly improves spatial grounding. This lightweight method requires no retraining, making it highly practical for deploying safer and more reliable embodied agents.

What Problem Does This Paper Address?

When Multimodal Large Language Models (MLLMs) are integrated into embodied agents acting in 3D environments, reliability is paramount. A single hallucinated object can derail a physical task or lead to unsafe behavior. Unfortunately, 3D MLLMs frequently suffer from "over-affirmation" biases, relying on textually plausible language priors (e.g., assuming a microwave is in a kitchen) rather than actual 3D visual evidence.

While Visual Contrastive Decoding (VCD) has successfully mitigated hallucinations in 2D vision-language models, existing 2D methods rely on pixel-level perturbations like blurring or masking. These do not translate to 3D embodied settings, where hallucinations stem from failures in geometric grounding, object presence, occlusion, and spatial layouts rather than pixel inconsistencies. Furthermore, training-based mitigation strategies struggle to generalize to the infinite combinatorial diversity of real-world 3D environments.

Methods

The authors introduce 3D-VCD (3D Visual Contrastive Decoding), a training-free inference framework that operates on structured, object-centric 3D scene graphs.

3D-VCD Overview: The framework generates a distorted scene graph and contrastively fuses logits to eliminate hallucinated tokens.

As illustrated in Figure 2, the method follows a clear pipeline:

  1. Scene Graph Construction: The environment is represented as a structured 3D scene graph containing object categories, centroids, and spatial extents.
  2. Counterfactual Distortions: 3D-VCD applies lightweight perturbations to create a corrupted "negative" context ($\hat{\mathcal{G}}$). These include semantic perturbations (shuffling object category labels) and geometric perturbations (adding zero-mean Gaussian noise to 3D centroids and spatial extents).
  3. Dual-Context Inference: The MLLM processes the textual query under both the original scene graph ($\mathcal{G}$) and the distorted scene graph ($\hat{\mathcal{G}}$) in parallel.
  4. Contrastive Logit Fusion: If a token's probability remains high even when the underlying 3D geometry or semantics are corrupted, it indicates the token is driven by language priors rather than grounded evidence. The framework contrastively fuses the logits to penalize these insensitive tokens: $$z_t^{vcd} = (1 + \alpha) z_t^{(o)} - \alpha z_t^{(d)}$$ where $z_t^{(o)}$ and $z_t^{(d)}$ are the original and distorted logits, and $\alpha$ controls the penalty strength.

To ensure real-time viability, the authors implement batched dual forward passes and key-value (KV) caching, minimizing computational overhead.

Datasets

The framework was evaluated across two prominent embodied 3D grounding benchmarks:

  • 3D-POPE: A benchmark probing object hallucinations via binary visual questions, divided into Random, Popular, and Adversarial splits to target different bias types.
  • HEAL: A comprehensive benchmark testing an agent's grounding against scene-task inconsistencies (e.g., distractor injection, object removal).

The authors evaluated 3D-VCD using baseline architectures finetuned on the 3D-GRAND dataset (including 3D-LLM, 3D-VisTA, and LEO), as well as off-the-shelf instruction models like Llama-3-8B-Instruct and Qwen-14B-Instruct.

Key Insights and Ideas

  • Hallucination as Context-Insensitivity: The core philosophy of 3D-VCD is that a grounded token should be highly sensitive to changes in the visual scene. If corrupting the environment doesn't alter the model's confidence in a word, that word is a hallucination born from language priors.
  • Graph-Space vs. Pixel-Space: Moving contrastive decoding from 2D pixel manipulations to 3D graph-level manipulations allows the framework to probe structural and geometric reasoning directly, which is exactly where 3D MLLMs fail.
  • Implicit Distortions via Tasks: In benchmarks like HEAL, where explicit graph corruption isn't ideal, the authors successfully reinterpreted adversarial text prompts as the "distorted context," proving that contrastive decoding can generalize to task-level inconsistencies without changing the underlying 3D data.

Key Results

3D-VCD consistently outperformed existing 3D language models across all metrics without any additional training.

Figure 1. Split-wise results on 3D-POPE benchmark. Each split (Random, Popular, Adversarial) shows Precision, F1, and Accuracy. Our 3D-VCD training-free hallucination mitigation method con- sistently improves precision, F1, and accuracy while substantially reducing over-affirmation rates across all splits.
  • 3D-POPE: As shown in Figure 1, 3D-VCD achieved substantial gains. In the Random split, it boosted precision from 50.03% to 62.16% and accuracy from 50.07% to 67.99% compared to 3D-LLM, while drastically dropping the over-affirmation (Yes-rate) from 99.81% to 75.15%.
  • HEAL: When applied to Qwen-14B-Instruct, 3D-VCD achieved a 3.3× reduction in state hallucinations (CHAIR-$C_S$), cutting the error rate from 16.45% down to 5.0%.
  • Efficiency: Thanks to parallel batched inference and KV caching, 3D-VCD only increased end-to-end latency by roughly 0.25× (from ~2.0 seconds to 2.5 seconds per query), demonstrating its readiness for real-time robotics loops.
Inference time scales smoothly as scene complexity (number of objects) increases.

Limitations and Future Work

While highly effective, 3D-VCD relies on the availability of a structured 3D scene representation or paired task formulations to generate its counterfactual contexts. If the initial scene graph generation is severely corrupted or missing, the quality of the negative context could degrade.

For future work, the authors suggest extending this contrastive paradigm to temporal reasoning in dynamic 3D scenes, allowing embodied agents to maintain grounding and mitigate hallucinations over time as environments change.

Summary updated

Abstract

Large Language Models are increasingly integrated as the cognitive core of 3D embodied agents to enable complex environmental reasoning. However, these agents tend to inherit the critical flaw of hallucination, often failing to ground their responses to their 3D view. While Visual Contrastive Decoding (VCD) is a powerful training-free method for mitigating hallucinations in 2D image-based models, it has not been adapted to the complex 3D embodied environment. In this paper, we embarked on the ambitious goal of being the very first to bridge this gap by introducing a VCD framework for 3D embodied agents. Our method operates at inference time by generating a "negative" 3D context, not by blurring an image, but by applying novel distortions directly to a 3D scene graph, such as swapping object category labels or noising positional coordinates. We evaluate our approach on standard evaluation benchmarks and find that it consistently outperforms existing models. For example, in the random category of 3D-POPE, our 3D-VCD method reduced Yes-rate from 99.9\% to 75.1\% while simultaneously increasing precision from 50.0\% to 62.2\%. These results demonstrate that our training-free approach effectively curbs hallucination, yielding 3D agents that are significantly more reliable and grounded.
BibTeX
@inproceedings{ogunleye2026d,
  author = {Makanjuola Adekunmi Ogunleye and Eman Abdelrahman and Ismini Lourentzou},
  title = {{3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=R1d8Ms7HzA},
}
Figure