Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding

Ozan Unal, Christos Sakaridis, Suman Saha, Luc Van Gool

TL;DR

ConcreteNet shifts 3D visual grounding from coarse bounding boxes to precise point-level masks, enabling AI agents to better understand and interact with objects described in natural language. By introducing four targeted modules—including localized attention masking and viewpoint awareness—it achieves state-of-the-art performance on challenging benchmarks containing repetitive objects.

What Problem Does This Paper Address?

3D visual grounding is the task of localizing a specific object within a 3D scene based on a natural language description (e.g., "the office chair pulled out from the desk"). This capability is essential for embodied AI, indoor robotics, and AR/VR applications.

Traditionally, this problem is framed as "grounding-by-detection," where the model outputs a 3D bounding box. However, bounding boxes lack the geometric detail required for real-world physical interactions, such as a robotic arm attempting to grasp an oddly shaped object.

To fix this, the authors tackle dense 3D visual grounding (referral-based 3D instance segmentation), which outputs exact point-level masks. The core challenge here is separability: when a room contains multiple identical objects (like several chairs or monitors), standard segmentation models struggle to distinguish between them in the latent space, leading to high false-positive rates.

Fig. 1: ConcreteNet localizes referred ob- jects via dense masks rather than boxes.

Methods

The authors introduce ConcreteNet, a kernel-based 3D instance segmentation network augmented with four novel, stand-alone modules designed to improve verbo-visual fusion:

  1. Bottom-Up Attentive Fusion (BAF): Instead of using global attention—where every object candidate looks at every other object across the entire room—BAF enforces spatial locality. It applies a spherical attention mask that progressively expands its radius across transformer layers, forcing the model to resolve local, inter-instance relationships first (e.g., focusing on the desk right next to the target chair).
  2. Contrastive Learning Scheme: To prevent embeddings of identical semantic classes from clustering too closely together, the authors implement a cross-modal contrastive loss. It pulls matching language embeddings and 3D instance vectors together while pushing non-matching pairs apart.
  3. Global Camera Token (GCT): Human descriptions are inherently view-dependent ("the chair on the left"). GCT introduces a learned camera viewpoint token into the transformer decoder. This token is directly supervised by the actual camera positions used during data annotation, helping the model resolve perspective-dependent language.
  4. Multi-View Ensembling (MVE): A test-time augmentation technique. The point cloud is evaluated under multiple yaw rotations. A two-stage algorithm then determines the most consistent target object across views and refines the final mask using pointwise majority voting.
Fig. 2: Illustration of our ConcreteNet dense 3D visual grounding pipeline (left). Given a point cloud and a natural language prompt, we first generate instance candidates (blue) and word embeddings (pink). We then fuse these to densely ground the verbal description to the 3D scene. We improve performance by localizing attention via a bottom-up attentive fusion module (right), utilizing contrastive learning to promote better feature separability, and learning the camera position to disambiguate view- dependent descriptions. Our final prediction is generated by merging the token of the best-fitting instance with its predicted mask.

Datasets

The model was trained and evaluated on two primary datasets:

  • ScanRefer: Contains 51,583 descriptions of 11,046 objects from 800 ScanNet scenes. This dataset includes view-dependent descriptions, making it ideal for testing the GCT and BAF modules.
  • Nr3D (Noisy Reference 3D): Used to evaluate the model's generalization capabilities on referral-based 3D object localization.

Key Insights and Ideas

  • Locality Rules in Human Language: When humans differentiate between repetitive objects, they naturally rely on nearby landmarks. Restricting the model's attention mechanism to a "bottom-up" spatial neighborhood mirrors this human trait and prevents distant distractors from muddying the fusion process.
  • Late Multi-View Aggregation: Instead of performing computationally heavy multi-view feature fusion early in the encoder network, ensembling at the output mask level (MVE) successfully reduces epistemic uncertainty with minimal additional memory overhead.
  • The Power of Perspective: Explicitly supervising a global camera token provides an elegant way to inject viewpoint sensitivity into a 3D model without needing complex 2D image rendering pipelines.

Key Results

ConcreteNet achieved 1st place on the challenging ScanRefer online benchmark and won the 3D Object Localization challenge at the ICCV 3rd Workshop on Language for 3D Scenes.

  • ScanRefer Validation Split: At the strict 50% Intersection-over-Union (IoU) threshold, ConcreteNet achieved an overall accuracy of 46.53%, significantly outperforming previous state-of-the-art methods like EDA (42.26%) and HAM (40.60%).
  • Handling Repetitive Objects: On the "Multiple" subset of ScanRefer (scenes with identical distractors), the combination of BAF and contrastive learning boosted accuracy at IoU@50 from a baseline of 27.86% to 38.39%.
  • Nr3D Dataset: ConcreteNet outperformed existing methods by +6.55% in overall accuracy at the 50% IoU threshold, demonstrating strong generalization even without using the GCT module (as Nr3D lacks camera position data).

Limitations and Future Work

  • Dependency on Camera Metadata: To leverage the Global Camera Token (GCT), the dataset must provide ground-truth camera positions. While common in datasets like ScanRefer, this data isn't always available in the wild.
  • Inference Latency: Multi-View Ensembling (MVE) requires multiple forward passes through the decoder. While the authors note the memory cost is minimal because the heavy encoder isn't duplicated, it still introduces a trade-off between peak accuracy and real-time execution speed.
  • Future Directions: Promising next steps include implicitly predicting the camera viewpoint when annotations are missing, and optimizing the multi-view pipeline for real-time robotic deployment.

Summary updated

Abstract

3D visual grounding is the task of localizing the object in a 3D scene which is referred by a description in natural language. With a wide range of applications ranging from autonomous indoor robotics to AR/VR, the task has recently risen in popularity. A common formulation to tackle 3D visual grounding is grounding-by-detection, where localization is done via bounding boxes. However, for real-life applications that require physical interactions, a bounding box insufficiently describes the geometry of an object. We therefore tackle the problem of dense 3D visual grounding, i.e. referral-based 3D instance segmentation. We propose a dense 3D grounding network ConcreteNet, featuring four novel stand-alone modules that aim to improve grounding performance for challenging repetitive instances, i.e. instances with distractors of the same semantic class. First, we introduce a bottom-up attentive fusion module that aims to disambiguate inter-instance relational cues, next, we construct a contrastive training scheme to induce separation in the latent space, we then resolve view-dependent utterances via a learned global camera token, and finally we employ multi-view ensembling to improve referred mask quality. ConcreteNet ranks 1st on the challenging ScanRefer online benchmark and has won the ICCV 3rd Workshop on Language for 3D Scenes "3D Object Localization" challenge. Our code is available at ouenal.github.io/concretenet/.
BibTeX
@inproceedings{unal2024four,
  author = {Ozan Unal and Christos Sakaridis and Suman Saha and Luc Van Gool},
  title = {{Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73116-7\_12},
}
Figure