TL;DR
CrossWeave is a novel framework that achieves state-of-the-art results in retrieving 3D scenes and objects from 2D image queries using a single model. By shifting supervision from global scene-level labels to sparse, local 2D-3D patch-point correspondences, the authors demonstrate that fine-grained local alignment is a more effective and flexible foundation for cross-modal spatial reasoning.
What Problem Does This Paper Address?
In robotics and AR/VR, a common challenge is "cross-modal retrieval": finding the right 3D map or specific object in a database based on a few 2D photos. This is split into two tasks:
- Cross-Modal Scene Retrieval (CMSR): Identifying which room or environment the photos were taken in.
- Cross-Modal Instance Retrieval (CMIR): Identifying a specific 3D object (like a particular chair) from a 2D query.
Existing methods usually treat these as separate problems, requiring different models and global supervision (e.g., "this whole image matches this whole 3D scan"). The authors argue that this global approach is a bottleneck: it ignores the fine-grained local structures that distinguish similar-looking rooms and requires expensive, task-specific labels.
Methods
CrossWeave introduces a unified architecture that learns from sparse local correspondences rather than global labels. The pipeline consists of three main stages:
- Sparse Sampling: Instead of processing every pixel and point, the system projects 3D points onto the 2D image plane, filters for visibility, and uses Feature-space Farthest Point Sampling (FPS). This ensures the model trains on a semantically diverse subset of local patches and points rather than being overwhelmed by redundant regions like flat walls or floors.
- Feature Extraction & Alignment: The model uses frozen, high-performance encoders—DINOv2 for images and Utonia (based on Point Transformer v3) for 3D point clouds. These features are projected into a shared embedding space.
- Optimal Transport Aggregation: To turn local features into a global descriptor, CrossWeave uses SALAD, an aggregator based on Sinkhorn distances. This clusters structurally redundant regions together, freeing up the descriptor's capacity to represent unique, semantically distinct local features.
The model is trained with a dual-contrastive loss: one at the local patch-point level ($L_{p2p}$) to preserve fine details, and one at the aggregated image-point cloud level ($L_{i2pc}$) to organize those details into retrieval-ready descriptors.
Datasets
The authors evaluated CrossWeave on two major indoor 3D datasets:
- ScanNet: A large-scale dataset of 1,513 indoor scans with semantic and instance annotations.
- 3RScan: A challenging dataset featuring 1,482 scans of 478 environments, including "re-scans" where objects have been moved, testing the model's robustness to temporal changes.
Key Insights and Ideas
- Local is Sufficient: The most striking finding is that you don't need scene-level labels to perform scene retrieval. Training purely on local patch-to-point correspondences allows the model to generalize better to both scenes and instances.
- Feature-Space Diversity: Using FPS in the feature space rather than Euclidean space during training prevents the model from being dominated by geometrically large but semantically boring areas (like large floors), forcing it to focus on discriminative objects.
- Aggregation vs. Averaging: Traditional methods often average features across frames. CrossWeave’s use of Sinkhorn-based clustering (SALAD) preserves the distributional structure of the scene, which is critical for distinguishing between rooms with similar layouts but different contents.
Key Results
CrossWeave sets a new bar for performance, significantly outperforming the previous state-of-the-art, CrossOver.
- Scene Retrieval: On ScanNet, CrossWeave achieved a Top-1 recall of 61.54%, a massive 21.16-point improvement over the updated CrossOver baseline. On 3RScan, it improved Top-1 recall by nearly 23 points.
- Instance Retrieval: The model improved Top-1 recall on ScanNet by 7.83 points (reaching 78.77%).
- Efficiency: Unlike previous SOTA methods that require separate models for scene and instance tasks, CrossWeave handles both with a single set of weights.
Qualitatively, Figure 2 demonstrates that even when query images only show partial or localized views, CrossWeave produces descriptors discriminative enough to rank the correct 3D scene at the very top, whereas baselines often fail to include the correct scene in their Top-5.
Limitations and Future Work
Despite its success, the authors note a few areas for improvement:
- Single Session Assumption: The current inference model assumes query images come from a single viewing session; it doesn't explicitly handle queries spanning multiple sessions.
- Upstream Dependencies: Performance is tied to the quality of camera calibration, pose estimation, and the initial 3D reconstruction. Errors in these areas lead to noisy training correspondences.
- Segmentation Reliance: For instance retrieval, the model still requires pre-segmented instance masks for both the 2D query and the 3D database.
- Foundation Model Bias: As the system relies on pre-trained models like DINOv2, it may inherit latent biases regarding architectural styles or indoor environments present in those models' training data.
Summary updated