VGMamba: Attribute-to-Location Clue Reasoning for Quantity-Agnostic 3D Visual Grounding

Yihang Zhu, Jinhao Zhang, Yuxuan Wang, Aming Wu, Cheng DengICCV 2025

TL;DR

This paper introduces VGMamba, a novel framework for 3D visual grounding that tackles the challenge of accurately identifying objects in 3D scenes based on natural language descriptions, especially in complex scenarios with multiple similar objects. By proposing an "attribute-to-location clue reasoning" mechanism and leveraging the efficiency of Mamba-based models, VGMamba achieves state-of-the-art performance while significantly reducing computational costs.

What Problem Does This Paper Address?

3D Visual Grounding, the task of identifying and localizing specific 3D objects within a scene based on natural language descriptions, is crucial for applications like robotics and embodied AI. However, existing methods often struggle with complex queries, especially when multiple objects share similar attributes or when the scene contains many distractors. Many approaches follow a two-stage process: first detecting potential objects and then ranking them based on relevance to the query. This can be inefficient and inaccurate when language descriptions are nuanced or when distinguishing between similar objects is difficult. The paper highlights that human reasoning often involves a sequential process of considering object attributes (like color or shape) and then using location information to refine the search, a strategy that current models haven't effectively replicated. Furthermore, the quadratic complexity of traditional Transformer models limits their scalability for handling long-range dependencies in complex 3D scenes.

Methods

The core innovation of VGMamba lies in its attribute-to-location clue reasoning mechanism, implemented through a novel architecture composed of three key Mamba-based modules:

  • SVD-based Attribute Mamba (SAM): This module first uses Singular Value Decomposition (SVD) to decompose visual features into different attribute scales. It then employs a sliding window approach to recombine these singular vectors, capturing both coarse-grained and fine-grained attribute information. Finally, Mamba is used to sequentially model these multi-scale attribute features, effectively capturing long-range dependencies among attributes.
  • Location Mamba (LM): This module focuses on modeling spatial relationships. It incorporates object position and bounding box information, along with visual features, and uses Mamba for both global (cross-object) and local (neighboring object) communication to understand spatial context.
  • Instructive Dual-Mamba block (IDM): This module fuses object-related features and text features. It uses the language description to guide the extraction of object-related features by adapting the state transition matrices of the Mamba modules based on the text features, enabling cross-modal guidance.

By replacing computationally expensive attention mechanisms with Mamba's efficient state-space modeling, VGMamba aims to improve both accuracy and computational efficiency.

Datasets

The paper evaluates VGMamba on several established 3D visual grounding datasets:

  • ScanRefer: A dataset with natural language descriptions for objects in 3D scenes from ScanNet.
  • Nr3D and Sr3D: Datasets built on ScanNet, focusing on natural and template-based referring expressions, respectively, with an emphasis on spatial relationships.
  • Multi3DRefer: A dataset specifically designed for multi-object grounding, derived from ScanRefer, which includes scenarios with zero, one, or multiple target objects and distractors.

The paper does not introduce new datasets.

Key Insights and Ideas

  • Attribute-to-Location Reasoning is Key: The paper argues that mimicking human reasoning by first identifying attributes and then using location to refine the search is a more effective strategy for complex 3D visual grounding than treating these cues in isolation.
  • Mamba's Efficiency for Long Sequences: The adoption of Mamba is crucial for efficiently modeling long-range dependencies in both attribute features and spatial relationships, overcoming the limitations of quadratic attention mechanisms in Transformers.
  • Disentangling Attributes with SVD: Using SVD to decompose visual features allows the model to better isolate and leverage different attribute scales, which is essential for distinguishing objects with specific characteristics.
  • Cross-Modal Guidance for Fusion: The Instructive Dual-Mamba block's ability to use text features to guide the Mamba's state transitions for object feature extraction is a novel way to achieve more accurate multi-modal alignment.
  • Attribute Priors are More Discriminative: The ablation study suggests that prioritizing attribute information first leads to better performance, implying that attributes are often more distinguishing than location alone for initial object candidate selection.

Key Results

VGMamba achieves state-of-the-art performance on multiple benchmarks:

  • ScanRefer: VGMamba achieves an overall accuracy of 60.0% (IoU 0.25) and 53.9% (IoU 0.5), surpassing the previous best by a significant margin.
  • Nr3D/Sr3D: The model achieves state-of-the-art results with 68.3% and 81.3% accuracy, respectively, demonstrating robustness in challenging scenarios.
  • Multi3DRefer: VGMamba shows strong performance, particularly in scenarios with distractors (ST w/D), outperforming competitors by 3.1% and highlighting its ability to manage complex scenes. It also excels in the multi-object (MT) case, improving by 4.1%.
  • Efficiency: The paper demonstrates a favorable efficiency-performance trade-off, with VGMamba achieving high accuracy at significantly lower computational costs (FLOPS) compared to other methods.

Limitations and Future Work

  • Limitations: While the paper highlights the effectiveness of the attribute-first approach, it acknowledges that location information can sometimes be less specific and potentially misleading. The paper also notes that the model's performance on the Nr3D dataset is challenged by its intricate and free-form textual descriptions.
  • Future Work: The authors suggest that further exploration of Mamba-based models for 3D visual grounding is warranted. The success of the attribute-to-location reasoning mechanism also opens avenues for investigating more sophisticated reasoning strategies and cross-modal fusion techniques. The paper's focus on efficiency suggests potential for deployment in resource-constrained environments.

Summary updated

BibTeX
@inproceedings{zhu2025vgmamba,
  author = {Yihang Zhu and Jinhao Zhang and Yuxuan Wang and Aming Wu and Cheng Deng},
  title = {{VGMamba: Attribute-to-Location Clue Reasoning for Quantity-Agnostic 3D Visual Grounding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure