TL;DR
CLICK3R introduces an interactive framework for feed-forward 3D reconstruction that allows users to correct model errors using sparse point correspondences ("clicks"). By injecting these geometric constraints through a lightweight adapter, the system can resolve ambiguities in challenging scenes—like those with repetitive textures or extreme viewpoint changes—where fully automatic models typically fail.
What Problem Does This Paper Address?
Recent "feed-forward" 3D reconstruction models, such as DUSt3R, have revolutionized the field by replacing slow, iterative optimization with direct neural regression. While these models are incredibly fast, they suffer from a major drawback: they produce a single, irrevocable prediction. If the model fails—which it often does in scenes with low overlap, symmetric structures, or textureless regions—there is no way for a user to intervene and fix the result.
Traditional photogrammetry relied on manual tie-points to anchor geometry, but modern deep learning models have largely abandoned this interactive loop. CLICK3R aims to bring the "human-in-the-loop" back to 3D vision, providing a practical mechanism to guide feed-forward models without sacrificing their inference speed.
Methods
The authors build upon the DUSt3R architecture, which uses a Vision Transformer (ViT) encoder-decoder to predict 3D pointmaps. CLICK3R adds two key components:
- Correspondence Tokenizer: This module converts a pair of user-clicked coordinates (one in each image) into a set of tokens. It uses random Fourier features to encode sub-pixel positions and learnable embeddings to identify the specific view and link the matched points together.
- Point Correspondence Adapter: This is a lightweight, two-way attention transformer (only 2 layers deep) inserted between the encoder and decoder. It performs bidirectional exchange: prompt tokens attend to image features to absorb local context, while image tokens attend to prompt tokens to receive geometric constraints.
Crucially, the model is trained end-to-end with a "curriculum" of simulated clicks. This ensures that at test time, the model can handle anything from zero clicks (behaving like the base model) to a dozen clicks, progressively improving its accuracy as more information is provided.
Datasets
The model was trained on a massive mixture of nine datasets, including BlendedMVS, MegaDepth, CO3D, and ScanNet++.
To test the limits of interactive reconstruction, the authors introduced Hard3DBench. This new benchmark is specifically curated to highlight the failure modes of current SOTA models. It categorizes "hard" cases into five types:
- Large Viewpoint (> 60°)
- Moderate Viewpoint (30°–60°)
- Low Overlap (< 20%)
- Large Baseline
- Extreme Depth Range
Key Insights and Ideas
- The Power of One Click: One of the most surprising findings is that even a single well-placed correspondence click can dramatically reduce rotation error. This suggests that the model isn't just "averaging" the click data but is using it as a strong geometric anchor to re-align its internal representation.
- Location Matters: Through a "Grid Oracle" experiment, the authors found that clicking in the "right" region is more effective than just adding more clicks. Strategic interaction is highly efficient.
- Minimal Overhead: The correspondence adapter adds only about 34M parameters (a ~6% increase) and roughly 1ms of GPU compute time. This makes interactivity essentially "free" in terms of latency.
- Robustness to Noise: The model is surprisingly robust to slightly "off" clicks (human error), though performance degrades if the clicks are significantly incorrect.
Key Results
CLICK3R shows substantial gains over fully automatic baselines, especially on the challenging Hard3DBench:
- Quantitative: On the Hard3DBench 5-view in-distribution set, CLICK3R (with 8 prompts) improved the mean Average Accuracy (mAA@30°) from 39.4% (for a retrained DUSt3R) to 49.3%.
- Qualitative: In scenes where standard models produce "hallucinated" or warped geometry due to low overlap, CLICK3R successfully recovers the correct structure with just two clicks.
- Human Evaluation: When tested with real human users providing clicks via a custom labeling tool, the model recovered 96% of the improvement gap seen with perfect ground-truth clicks, proving its practical utility.
Limitations and Future Work
Despite its success, CLICK3R has clear boundaries:
- Zero Overlap: If two images have absolutely no visual overlap, clicks cannot "bridge" them; the model still has to hallucinate the relative pose.
- Extreme Rotations: Viewpoint changes greater than 90° remain an unsolved challenge, as the visual appearance of objects changes too drastically for the current ViT backbone to reconcile.
- Large-Scale Scenes: In outdoor environments spanning tens of meters, a few clicks can fix local geometry, but distant structures may still exhibit "bending" or distortion.
- Resolution: Due to compute constraints, the model is currently trained at 224px resolution. Scaling to higher resolutions is a natural next step.
The authors suggest that future work could involve deeper injection of geometric reasoning modules or extending the framework to handle "trajectory prompts" across long video sequences.
Summary updated