Abstract
Traditional camera pose estimation and SLAM systems often struggle in scenarios with low visual overlap, where direct feature matching between frames is impossible. ObjectMatch introduces a framework that leverages object-centric reasoning to establish "indirect correspondences." By predicting dense canonical 3D coordinates for recognized objects, the system creates geometric anchors that allow for robust RGB-D registration even when frames share no common pixels.
The Challenge of Wide-Baseline Registration
Most modern SLAM and registration pipelines rely on direct correspondences. To determine the relative motion between two camera frames, algorithms typically search for identical physical points visible in both images. While highly effective for high-overlap sequences, this approach fails in "wide-baseline" scenarios. For instance, if a camera captures a chair from the front in one frame and from the side in the next, there may be zero shared pixels to match.
Humans, however, can easily infer the camera's relative position because we understand the underlying geometry of the object. ObjectMatch bridges this gap by imbuing computer vision systems with similar object-centric reasoning, using objects as landmarks to align disparate views.
Methodology
The ObjectMatch pipeline operates in three distinct stages:
- Object Recognition & NOCs Prediction: The system employs a multi-modal neural network (ResNet-FPN) that processes color, depth, and surface normals. It detects specific object instances and predicts Normalized Object Coordinates (NOCs) for every pixel. These NOCs map pixels to a 3D coordinate system local to the object, providing a canonical reference frame.
- Object Identification: To link frames effectively, the system must distinguish between different instances of the same object class (e.g., two identical chairs). A metric learning approach using triplet loss generates unique embeddings for detected objects, enabling precise instance matching across different viewpoints.
- Joint Optimization: The core of the framework is a joint energy function that combines traditional and object-centric constraints:
- Feature-matching energy ($E_c$): Utilizes standard keypoint matches from state-of-the-art tools like SuperGlue.
- Object-centric energy ($E_o$): Derived from predicted NOCs and identified object instances.
This non-linear least squares problem is solved via a Gauss-Newton optimizer, effectively using objects as a "geometric bridge" to register frames that lack direct visual features.
Evaluation and Datasets
The framework was rigorously tested across several key datasets:
- ScanNet: Used for primary training and evaluation of indoor RGB-D sequences.
- Scan2CAD: Provided ground truth object alignments and 9-DoF poses for training the NOC and scale prediction heads.
- TUM-RGBD: Evaluated performance on trajectory error in challenging, low-frame-rate SLAM scenarios.
Key Insights
- Objects as Geometric Anchors: Objects provide global context. Knowing the center and orientation of a table in two different frames allows for accurate transformation calculation, even if the frames view entirely different parts of the table.
- The Power of Multi-modal Fusion: Incorporating depth and 3D surface normals into the FPN backbone significantly improves the accuracy of object detection and coordinate prediction compared to using color alone.
- Hybrid Robustness: ObjectMatch is designed to complement, not replace, high-performance feature matchers. When overlap is high, traditional features provide precision; when overlap drops, object-based constraints prevent catastrophic failure.
Results
ObjectMatch demonstrates significant improvements in challenging conditions:
- Low-Overlap Performance: In pairs with 10% or less inter-frame overlap, registration recall improved from 24% to 45% over feature-matching baselines.
- SLAM Accuracy: In low-frame-rate settings (simulating fast motion), the method achieved a 35% reduction in trajectory error.
- Loop Closure: The system is notably more resilient when a camera returns to a previously seen area from a drastically different angle.
Limitations and Future Work
While robust, the method depends on the presence of recognizable objects. In sparse or highly cluttered environments with unknown object types, its advantages are reduced. Furthermore, while object correspondences provide excellent initial alignment, they are often coarser than pixel-perfect feature matches, currently requiring a post-processing ICP (Iterative Closest Point) step for refinement. Future research will explore part-level correspondences and more unified, end-to-end architectures.
Summary updated