ObjectMatch: Robust Registration using Canonical Object Correspondences

Can Gümeli, Angela Dai, Matthias Nießner

Abstract

Traditional camera pose estimation and SLAM systems often struggle in scenarios with low visual overlap, where direct feature matching between frames is impossible. ObjectMatch introduces a framework that leverages object-centric reasoning to establish "indirect correspondences." By predicting dense canonical 3D coordinates for recognized objects, the system creates geometric anchors that allow for robust RGB-D registration even when frames share no common pixels.

The Challenge of Wide-Baseline Registration

Most modern SLAM and registration pipelines rely on direct correspondences. To determine the relative motion between two camera frames, algorithms typically search for identical physical points visible in both images. While highly effective for high-overlap sequences, this approach fails in "wide-baseline" scenarios. For instance, if a camera captures a chair from the front in one frame and from the side in the next, there may be zero shared pixels to match.

Humans, however, can easily infer the camera's relative position because we understand the underlying geometry of the object. ObjectMatch bridges this gap by imbuing computer vision systems with similar object-centric reasoning, using objects as landmarks to align disparate views.

Overview of the ObjectMatch pipeline: from low-overlap RGB-D frames to canonical coordinate prediction and final 3D registration.

Methodology

The ObjectMatch pipeline operates in three distinct stages:

  1. Object Recognition & NOCs Prediction: The system employs a multi-modal neural network (ResNet-FPN) that processes color, depth, and surface normals. It detects specific object instances and predicts Normalized Object Coordinates (NOCs) for every pixel. These NOCs map pixels to a 3D coordinate system local to the object, providing a canonical reference frame.
  2. Object Identification: To link frames effectively, the system must distinguish between different instances of the same object class (e.g., two identical chairs). A metric learning approach using triplet loss generates unique embeddings for detected objects, enabling precise instance matching across different viewpoints.
  3. Joint Optimization: The core of the framework is a joint energy function that combines traditional and object-centric constraints:
  4. Feature-matching energy ($E_c$): Utilizes standard keypoint matches from state-of-the-art tools like SuperGlue.
  5. Object-centric energy ($E_o$): Derived from predicted NOCs and identified object instances.

This non-linear least squares problem is solved via a Gauss-Newton optimizer, effectively using objects as a "geometric bridge" to register frames that lack direct visual features.

Evaluation and Datasets

The framework was rigorously tested across several key datasets:

  • ScanNet: Used for primary training and evaluation of indoor RGB-D sequences.
  • Scan2CAD: Provided ground truth object alignments and 9-DoF poses for training the NOC and scale prediction heads.
  • TUM-RGBD: Evaluated performance on trajectory error in challenging, low-frame-rate SLAM scenarios.

Key Insights

  • Objects as Geometric Anchors: Objects provide global context. Knowing the center and orientation of a table in two different frames allows for accurate transformation calculation, even if the frames view entirely different parts of the table.
  • The Power of Multi-modal Fusion: Incorporating depth and 3D surface normals into the FPN backbone significantly improves the accuracy of object detection and coordinate prediction compared to using color alone.
  • Hybrid Robustness: ObjectMatch is designed to complement, not replace, high-performance feature matchers. When overlap is high, traditional features provide precision; when overlap drops, object-based constraints prevent catastrophic failure.

Results

ObjectMatch demonstrates significant improvements in challenging conditions:

  • Low-Overlap Performance: In pairs with 10% or less inter-frame overlap, registration recall improved from 24% to 45% over feature-matching baselines.
  • SLAM Accuracy: In low-frame-rate settings (simulating fast motion), the method achieved a 35% reduction in trajectory error.
  • Loop Closure: The system is notably more resilient when a camera returns to a previously seen area from a drastically different angle.

Limitations and Future Work

While robust, the method depends on the presence of recognizable objects. In sparse or highly cluttered environments with unknown object types, its advantages are reduced. Furthermore, while object correspondences provide excellent initial alignment, they are often coarser than pixel-perfect feature matches, currently requiring a post-processing ICP (Iterative Closest Point) step for refinement. Future research will explore part-level correspondences and more unified, end-to-end architectures.

Summary updated

Abstract

We present ObjectMatch, a semantic and object-centric camera pose estimator for RGB-D SLAM pipelines. Modern camera pose estimators rely on direct correspondences of overlapping regions between frames; however, they cannot align camera frames with little or no overlap. In this work, we propose to leverage indirect correspondences obtained via semantic object identification. For instance, when an object is seen from the front in one frame and from the back in another frame, we can provide additional pose constraints through canonical object correspondences. We first propose a neural network to predict such correspondences on a per-pixel level, which we then combine in our energy formulation with state-of-the-art keypoint matching solved with a joint Gauss-Newton optimization. In a pairwise setting, our method improves registration recall of state-of-the-art feature matching, including from 24% to 45% in pairs with 10% or less inter-frame overlap. In registering RGB-D sequences, our method outperforms cutting-edge SLAM baselines in challenging, low-frame-rate scenarios, achieving more than 35% reduction in trajectory error in multiple scenes.
BibTeX
@inproceedings{gumeli2022objectmatch,
  author = {Can Gümeli and Angela Dai and Matthias Nießner},
  title = {{ObjectMatch: Robust Registration using Canonical Object Correspondences}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure