Click3R: Interactive Stereo 3D Reconstruction with Sparse Correspondence Clicks

Hao Gu, Jin Yao, Zezhou ChengarXiv 2026 · arXiv

TL;DR

CLICK3R introduces an interactive framework for feed-forward 3D reconstruction that allows users to correct model errors using sparse point correspondences ("clicks"). By injecting these geometric constraints through a lightweight adapter, the system can resolve ambiguities in challenging scenes—like those with repetitive textures or extreme viewpoint changes—where fully automatic models typically fail.

What Problem Does This Paper Address?

Recent "feed-forward" 3D reconstruction models, such as DUSt3R, have revolutionized the field by replacing slow, iterative optimization with direct neural regression. While these models are incredibly fast, they suffer from a major drawback: they produce a single, irrevocable prediction. If the model fails—which it often does in scenes with low overlap, symmetric structures, or textureless regions—there is no way for a user to intervene and fix the result.

Traditional photogrammetry relied on manual tie-points to anchor geometry, but modern deep learning models have largely abandoned this interactive loop. CLICK3R aims to bring the "human-in-the-loop" back to 3D vision, providing a practical mechanism to guide feed-forward models without sacrificing their inference speed.

Methods

The authors build upon the DUSt3R architecture, which uses a Vision Transformer (ViT) encoder-decoder to predict 3D pointmaps. CLICK3R adds two key components:

  1. Correspondence Tokenizer: This module converts a pair of user-clicked coordinates (one in each image) into a set of tokens. It uses random Fourier features to encode sub-pixel positions and learnable embeddings to identify the specific view and link the matched points together.
  2. Point Correspondence Adapter: This is a lightweight, two-way attention transformer (only 2 layers deep) inserted between the encoder and decoder. It performs bidirectional exchange: prompt tokens attend to image features to absorb local context, while image tokens attend to prompt tokens to receive geometric constraints.

Crucially, the model is trained end-to-end with a "curriculum" of simulated clicks. This ensures that at test time, the model can handle anything from zero clicks (behaving like the base model) to a dozen clicks, progressively improving its accuracy as more information is provided.

Figure 2: CLICK3R architecture. A shared ViT encoder processes the two input images. Corre- spondence prompts are tokenized using our correspondence tokenizer (Section 3.2). A symmetric point correspondence adapter, implemented as a lightweight transformer with self-attention and cross-attention (Section 3.3), updates the decoder-input features for each view. The asymmetric decoder then interprets the updated embeddings and passes them to the original DUSt3R [44] heads to predict 3D pointmaps and confidence.

Datasets

The model was trained on a massive mixture of nine datasets, including BlendedMVS, MegaDepth, CO3D, and ScanNet++.

To test the limits of interactive reconstruction, the authors introduced Hard3DBench. This new benchmark is specifically curated to highlight the failure modes of current SOTA models. It categorizes "hard" cases into five types:

  • Large Viewpoint (> 60°)
  • Moderate Viewpoint (30°–60°)
  • Low Overlap (< 20%)
  • Large Baseline
  • Extreme Depth Range
Hard3DBench categorizes common failure modes for feed-forward 3D reconstruction models.

Key Insights and Ideas

  • The Power of One Click: One of the most surprising findings is that even a single well-placed correspondence click can dramatically reduce rotation error. This suggests that the model isn't just "averaging" the click data but is using it as a strong geometric anchor to re-align its internal representation.
  • Location Matters: Through a "Grid Oracle" experiment, the authors found that clicking in the "right" region is more effective than just adding more clicks. Strategic interaction is highly efficient.
  • Minimal Overhead: The correspondence adapter adds only about 34M parameters (a ~6% increase) and roughly 1ms of GPU compute time. This makes interactivity essentially "free" in terms of latency.
  • Robustness to Noise: The model is surprisingly robust to slightly "off" clicks (human error), though performance degrades if the clicks are significantly incorrect.

Key Results

CLICK3R shows substantial gains over fully automatic baselines, especially on the challenging Hard3DBench:

  • Quantitative: On the Hard3DBench 5-view in-distribution set, CLICK3R (with 8 prompts) improved the mean Average Accuracy (mAA@30°) from 39.4% (for a retrained DUSt3R) to 49.3%.
  • Qualitative: In scenes where standard models produce "hallucinated" or warped geometry due to low overlap, CLICK3R successfully recovers the correct structure with just two clicks.
  • Human Evaluation: When tested with real human users providing clicks via a custom labeling tool, the model recovered 96% of the improvement gap seen with perfect ground-truth clicks, proving its practical utility.
Figure 1: Overview. Given (a) an image pair and sparse correspondences (colored dot clicks), CLICK3R conditions DUSt3R [44] to produce accurate 3D reconstruction. (b) DUSt3R fails on a scene with limited overlaps. (c) With as few as two clicks, CLICK3R recovers correct geometry.

Limitations and Future Work

Despite its success, CLICK3R has clear boundaries:

  • Zero Overlap: If two images have absolutely no visual overlap, clicks cannot "bridge" them; the model still has to hallucinate the relative pose.
  • Extreme Rotations: Viewpoint changes greater than 90° remain an unsolved challenge, as the visual appearance of objects changes too drastically for the current ViT backbone to reconcile.
  • Large-Scale Scenes: In outdoor environments spanning tens of meters, a few clicks can fix local geometry, but distant structures may still exhibit "bending" or distortion.
  • Resolution: Due to compute constraints, the model is currently trained at 224px resolution. Scaling to higher resolutions is a natural next step.

The authors suggest that future work could involve deeper injection of geometric reasoning modules or extending the framework to handle "trajectory prompts" across long video sequences.

Summary updated

Abstract

Despite impressive progress, existing feed-forward large 3D reconstruction models produce a single, irrevocable prediction. When these models fail, there is no mechanism to correct their mistakes. To address this limitation, we introduce Click3R, a framework for interactive geometric correction of feed-forward models using sparse human correspondence clicks. Given an image pair and a small set of cross-view point correspondences provided by a user, Click3R injects geometric constraints into the 3D reconstruction networks through a lightweight point correspondence adapter, enabling the model to resolve ambiguities in challenging scenarios such as repetitive textures, symmetric structures, large viewpoint changes, or textureless regions. To evaluate interactive reconstruction, we curate a challenging benchmark specifically designed to expose failure modes of existing feed-forward 3D reconstruction methods. Experiments show that Click3R reduces reconstruction error with as few as a single correspondence click, improving performance on both standard and our curated benchmarks. Our results highlight that sparse human interaction provides an effective and practical mechanism for correcting geometric reconstruction errors.
BibTeX
@misc{gu2026click,
  author = {Hao Gu and Jin Yao and Zezhou Cheng},
  title = {{Click3R: Interactive Stereo 3D Reconstruction with Sparse Correspondence Clicks}},
  year = {2026},
  eprint = {2611.12875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12875},
}
Figure