TL;DR
3D-VCD is the first training-free, inference-time framework designed to mitigate hallucinations in 3D embodied Large Multimodal Models (MLLMs). By contrastively decoding predictions between an original 3D scene graph and a structurally or semantically distorted "negative" context, it suppresses language-prior biases and significantly improves spatial grounding. This lightweight method requires no retraining, making it highly practical for deploying safer and more reliable embodied agents.
What Problem Does This Paper Address?
When Multimodal Large Language Models (MLLMs) are integrated into embodied agents acting in 3D environments, reliability is paramount. A single hallucinated object can derail a physical task or lead to unsafe behavior. Unfortunately, 3D MLLMs frequently suffer from "over-affirmation" biases, relying on textually plausible language priors (e.g., assuming a microwave is in a kitchen) rather than actual 3D visual evidence.
While Visual Contrastive Decoding (VCD) has successfully mitigated hallucinations in 2D vision-language models, existing 2D methods rely on pixel-level perturbations like blurring or masking. These do not translate to 3D embodied settings, where hallucinations stem from failures in geometric grounding, object presence, occlusion, and spatial layouts rather than pixel inconsistencies. Furthermore, training-based mitigation strategies struggle to generalize to the infinite combinatorial diversity of real-world 3D environments.
Methods
The authors introduce 3D-VCD (3D Visual Contrastive Decoding), a training-free inference framework that operates on structured, object-centric 3D scene graphs.
As illustrated in Figure 2, the method follows a clear pipeline:
- Scene Graph Construction: The environment is represented as a structured 3D scene graph containing object categories, centroids, and spatial extents.
- Counterfactual Distortions: 3D-VCD applies lightweight perturbations to create a corrupted "negative" context ($\hat{\mathcal{G}}$). These include semantic perturbations (shuffling object category labels) and geometric perturbations (adding zero-mean Gaussian noise to 3D centroids and spatial extents).
- Dual-Context Inference: The MLLM processes the textual query under both the original scene graph ($\mathcal{G}$) and the distorted scene graph ($\hat{\mathcal{G}}$) in parallel.
- Contrastive Logit Fusion: If a token's probability remains high even when the underlying 3D geometry or semantics are corrupted, it indicates the token is driven by language priors rather than grounded evidence. The framework contrastively fuses the logits to penalize these insensitive tokens: $$z_t^{vcd} = (1 + \alpha) z_t^{(o)} - \alpha z_t^{(d)}$$ where $z_t^{(o)}$ and $z_t^{(d)}$ are the original and distorted logits, and $\alpha$ controls the penalty strength.
To ensure real-time viability, the authors implement batched dual forward passes and key-value (KV) caching, minimizing computational overhead.
Datasets
The framework was evaluated across two prominent embodied 3D grounding benchmarks:
- 3D-POPE: A benchmark probing object hallucinations via binary visual questions, divided into Random, Popular, and Adversarial splits to target different bias types.
- HEAL: A comprehensive benchmark testing an agent's grounding against scene-task inconsistencies (e.g., distractor injection, object removal).
The authors evaluated 3D-VCD using baseline architectures finetuned on the 3D-GRAND dataset (including 3D-LLM, 3D-VisTA, and LEO), as well as off-the-shelf instruction models like Llama-3-8B-Instruct and Qwen-14B-Instruct.
Key Insights and Ideas
- Hallucination as Context-Insensitivity: The core philosophy of 3D-VCD is that a grounded token should be highly sensitive to changes in the visual scene. If corrupting the environment doesn't alter the model's confidence in a word, that word is a hallucination born from language priors.
- Graph-Space vs. Pixel-Space: Moving contrastive decoding from 2D pixel manipulations to 3D graph-level manipulations allows the framework to probe structural and geometric reasoning directly, which is exactly where 3D MLLMs fail.
- Implicit Distortions via Tasks: In benchmarks like HEAL, where explicit graph corruption isn't ideal, the authors successfully reinterpreted adversarial text prompts as the "distorted context," proving that contrastive decoding can generalize to task-level inconsistencies without changing the underlying 3D data.
Key Results
3D-VCD consistently outperformed existing 3D language models across all metrics without any additional training.
- 3D-POPE: As shown in Figure 1, 3D-VCD achieved substantial gains. In the Random split, it boosted precision from 50.03% to 62.16% and accuracy from 50.07% to 67.99% compared to 3D-LLM, while drastically dropping the over-affirmation (Yes-rate) from 99.81% to 75.15%.
- HEAL: When applied to Qwen-14B-Instruct, 3D-VCD achieved a 3.3× reduction in state hallucinations (CHAIR-$C_S$), cutting the error rate from 16.45% down to 5.0%.
- Efficiency: Thanks to parallel batched inference and KV caching, 3D-VCD only increased end-to-end latency by roughly 0.25× (from ~2.0 seconds to 2.5 seconds per query), demonstrating its readiness for real-time robotics loops.
Limitations and Future Work
While highly effective, 3D-VCD relies on the availability of a structured 3D scene representation or paired task formulations to generate its counterfactual contexts. If the initial scene graph generation is severely corrupted or missing, the quality of the negative context could degrade.
For future work, the authors suggest extending this contrastive paradigm to temporal reasoning in dynamic 3D scenes, allowing embodied agents to maintain grounding and mitigate hallucinations over time as environments change.
Summary updated