DACA: Dual-Anchor Collaborative Agent for Zero-Shot 3D Visual Grounding

Jiaming Liu, Linghe Kong, Guixu Zhang, Huaxi HuangarXiv 2026 · arXiv

TL;DR

DACA (Dual-Anchor Collaborative Agent) is a new zero-shot framework for 3D visual grounding that replaces traditional "one-way" pipelines with a bidirectional, collaborative reasoning loop. By allowing a semantic anchor (visual tracking) and a geometric anchor (3D reconstruction) to mutually correct each other, DACA achieves state-of-the-art performance on major benchmarks without requiring any expensive 3D-labeled training data.

What Problem Does This Paper Address?

3D visual grounding (3D-VG) is the task of finding a specific object in a 3D scene based on a natural language description (e.g., "the white pillow on the left side of the bed"). While humans do this effortlessly, it is incredibly difficult for AI.

Traditional supervised methods require massive amounts of 3D-text aligned data, which is 5–10 times more expensive to annotate than 2D data. To solve this, researchers have turned to zero-shot methods that leverage Large Language Models (LLMs) and Vision-Language Models (VLMs). However, existing zero-shot agents typically follow a "unidirectional" pipeline: they track the object semantically in 2D frames first, and then try to build a 3D model from those tracks.

The authors identify a fatal flaw in this approach: error accumulation. If the semantic tracker makes a mistake early on—perhaps due to a sudden viewpoint change or an occlusion—the error propagates, and the geometric reconstruction fails. There is no mechanism for the agent to "look back" and use geometric clues to fix its semantic understanding.

Figure 1. Comparison of four zero-shot 3D-VG paradigms. (a) LLM matches a query to 3D candidate descriptions, and (b) VLM uses rendered multi-view BEV images. (c) TAB passes semantic evidence to geometric extension through the coarse centroid. (d) DACA also makes geometrically retrieved views available for subsequent semantic acquisition. The arrows describe evidence flow between anchors.

Methods

DACA reinterprets 3D-VG as a semantic-geometric dual-anchor collaborative reasoning paradigm. It uses two specialized "tools" that talk to each other:

  1. Semantic Anchor Tool (MG-STE): This tool handles the "what" and "where" in 2D. It uses Motion-Guided Semantic Temporal Extension, which combines dense optical flow (to predict where an object moves between frames) with local VLM verification. This is much more efficient and robust than checking every full frame with a VLM.
  2. Geometric Anchor Tool (KA-MGE): This tool handles the 3D spatial reasoning. It uses Keypoint-Aware Multiview Geometric Extension. Instead of just tracking a single center point, it models the object using nine keypoints (the 8 corners of a bounding box plus the centroid). It also uses TSDF (Truncated Signed Distance Function) voxelization to perform ray-tracing visibility checks, ensuring it only considers views where the object isn't occluded.

The "secret sauce" is the Dual-Anchor Collaboration. The agent acts as a controller that orchestrates these tools. If the semantic tracker loses the target, the geometric anchor can propose new viewpoints based on the 3D model it has built so far. Conversely, semantic tracking helps refine the 3D geometry. This creates a closed-loop optimization process.

Figure 2. The pipeline of DACA. Given a language query and a sequence of images as input, the agent reasons about 3D-VG tasks under the guidance of 3D-VG skills, utilizing tools such as semantic and geometric anchors to complete the 3D reconstruction and grounding task.

Datasets

The authors evaluated DACA on two gold-standard benchmarks for 3D visual grounding:

  • ScanRefer: Contains over 51,000 queries for objects in ScanNet indoor scenes.
  • Nr3D (Natural Reference 3D): Part of the ReferIt3D dataset, featuring over 41,000 human-generated queries that are often more complex and viewpoint-dependent.

The experiments followed a standard zero-shot protocol using a 250-query validation subset for rigorous comparison with previous state-of-the-art methods like TAB.

Key Insights and Ideas

  • Bidirectionality Fixes Fragility: The most important insight is that geometry and semantics are two sides of the same coin. By feeding geometric observations back into the semantic acquisition phase, the agent can recover from tracking failures that would crash a standard pipeline.
  • Keypoints Over Centroids: Previous methods often relied on a single 3D centroid for projection. DACA’s use of nine keypoints allows it to capture the spatial extent and shape uncertainty of an object, making it far more resilient to noisy depth data or partial occlusions.
  • Optical Flow as a VLM Accelerator: Using optical flow to "crop" the VLM's attention to a local region significantly reduces false positives and computational overhead compared to global frame-by-frame matching.

Key Results

DACA sets a new high bar for zero-shot 3D visual grounding:

  • ScanRefer: Achieved an overall [email protected] of 73.6%, outperforming the previous best zero-shot method (TAB) by 2.4%. Remarkably, it even surpasses several fully supervised methods.
  • Nr3D: Achieved an overall accuracy of 70.4%, a 2.4% improvement over TAB. It showed particularly strong gains on "Hard" queries (+3.5%) and "View-Independent" queries (+2.6%).
  • Ablation Success: Removing either the semantic or geometric cross-validation branches led to significant performance drops, proving that the collaborative "handshake" between the two anchors is essential for accuracy.
Figure 4. Qualitative comparison of zero-shot 3D-VG prediction results on ScanRefer. GT bounding boxes are marked in green. The left shows the unique type, and the right shows the multiple type.

Limitations and Future Work

While DACA is a major step forward, the authors note a few areas for growth:

  • Static Assumption: The current framework assumes objects are stationary within a sequence. While the motion-guided tracking (MG-STE) provides a foundation for dynamic scenes, the 3D reconstruction currently expects a static environment.
  • Sensor Reliance: The method depends on high-quality RGB-D streams and accurate camera poses. In very messy or low-light real-world environments, these inputs might be degraded.
  • Future Directions: The authors suggest extending DACA to handle non-rigid or moving objects and integrating it into higher-level task planning for embodied robots (e.g., "Go to the kitchen and find the brown ottoman").

Summary updated

Abstract

3D visual grounding (3D-VG) is a task that bridges the natural language and 3D space, serving as a foundational capability for embodied intelligence and human-computer interaction. Although recent 3D-VG agents overcome the reliance on pre-processed 3D point clouds, they essentially employ a unidirectional, pipeline-style tool-calling architecture. This architecture separates semantic understanding from geometric reconstruction, leading to error accumulation and an inability to autonomously recover from failures. To address this flaw, this paper proposes the Dual-Anchor Collaborative Agent (DACA), which reinterprets 3D-VG as a semantic-geometric dual-anchor collaborative reasoning paradigm. DACA comprises two complementary specialized tools: the semantic anchor tool holds the visual semantic features of the target and achieves efficient target tracking via optical flow and VLM local verification; the geometric anchor tool holds the 3D spatial geometric features of the target and performs global spatial reasoning via voxelization and multi-keypoint projection. The two components exchange information bidirectionally via a standardized tool call protocol, autonomously deciding on the next perception-action step to form a closed-loop optimization. Experiments on the ScanRefer and Nr3D benchmarks demonstrate that zero-shot DACA achieves an overall [email protected] of 73.6\% on ScanRefer and 70.4\% overall accuracy on Nr3D, far surpassing the latest zero-shot 3D-VG methods. The code and models are available at https://github.com/liujia99/DACA
BibTeX
@misc{liu2026daca,
  author = {Jiaming Liu and Linghe Kong and Guixu Zhang and Huaxi Huang},
  title = {{DACA: Dual-Anchor Collaborative Agent for Zero-Shot 3D Visual Grounding}},
  year = {2026},
  eprint = {2611.25722},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25722},
}
Figure