Abstract
Semantic correspondence—the task of matching related parts across different instances of an object category—is often hampered by extreme viewpoint changes and visual ambiguities. SemAlign3D addresses these challenges by lifting 2D semantic features into a canonical 3D category representation. By optimizing for geometric consistency and semantic alignment during inference, the method effectively resolves common failure modes such as symmetry confusion. Evaluated on the SPair-71k benchmark, SemAlign3D achieves a new state-of-the-art PCK@0.1 of 88.9%, demonstrating significant improvements in rigid object categories.
What Problem Does This Paper Address?
Semantic correspondence involves identifying matching points between two different images of the same object category, such as the "nose" on two different airplanes. While Large Vision Models (LVMs) like DinoV2 excel at identifying local textures, they often lack an understanding of global geometric structure. This deficiency leads to two primary failure modes:
- Symmetry Confusion: 2D features for symmetric parts (e.g., left vs. right headlights) often appear identical, leading to flipped matches.
- Extreme Viewpoints: Significant rotations or occlusions make it difficult for 2D-only methods to maintain a coherent spatial map of the object.
Methods
The authors propose SemAlign3D, a two-stage approach that leverages 3D priors to guide 2D matching.
1. Building 3D Object-Class Representations
Rather than relying on generic CAD models, SemAlign3D learns a "canonical" 3D representation for specific categories (e.g., "chair") using sparsely annotated images. The process involves:
- Monocular Depth Estimation: Utilizing DepthAnythingV2 to estimate depth from single 2D images.
- Geometric Optimization: Optimizing camera focal lengths to minimize geometric variance across the training set.
- Semantic Lifting: Associating 2D semantic features from LVMs with 3D points to create a dense model that captures both spatial position and visual appearance.
2. Alignment via Gradient Descent
At inference time, the system fits the learned 3D model to a new 2D image by minimizing an alignment loss. This loss incorporates reconstruction accuracy, geometric consistency (ensuring the deformed model remains plausible), and background masking via SegmentAnything to prevent erroneous matches with the environment.
Key Insights and Ideas
- 3D as a Spatial Prior: 3D relationships are invariant to camera movement. By routing correspondence through a 3D model, the system inherently understands that a "left wing" must remain opposite the "right wing," regardless of the viewing angle.
- Data Efficiency: SemAlign3D builds robust 3D priors from a limited number of 2D images and monocular depth estimates, avoiding the need for massive 3D datasets.
- Robustness to Ambiguity: The fixed 3D structure acts as a tie-breaker when visual features are ambiguous, effectively eliminating symmetry-related errors.
Key Results
SemAlign3D was evaluated on benchmarks including SPair-71k, AP-10K, and PF-PASCAL. On the challenging SPair-71k dataset, the method established a new state-of-the-art, increasing the PCK@0.1 score from 85.6% to 88.9%. The most significant gains were observed in rigid object categories, with improvements exceeding 10 points for "Bottle," "Chair," and "TV." Qualitatively, the method successfully handles objects rotated nearly 180 degrees, where previous methods like GeoAware often failed.
Limitations and Future Work
Despite its performance, SemAlign3D faces challenges regarding runtime speed, taking 10–30 seconds per image on an NVIDIA 3090 due to the optimization loop. It also struggles with non-rigid objects like animals, suggesting a need for joint-angle aware representations. Future research aims to make the pipeline fully differentiable for end-to-end training.