Abstract
Image space feature detection is the act of selecting points or parts of an image that are easy to distinguish from the surrounding image region. By combining a repeatable point detection with a descriptor, parts of an image can be matched with one another, which is useful in applications like estimating pose from camera input or rectifying images. Recently, precise indoor tracking has started to become important for Augmented and Virtual reality as it is necessary to allow positioning of a headset in 3D space without the need for external tracking devices. Several modern feature detectors use homographies to simulate different viewpoints, not only to train feature detection and description, but test them as well. The problem is that, often, views of indoor spaces contain high depth disparity. This makes the approximation that a homography applied to an image represents a viewpoint change inaccurate. We claim that in order to train detectors to work well in indoor environments, they must be robust to this type of geometry, and repeatable under true viewpoint change instead of homographies. Here we focus on the problem of detecting repeatable feature locations under true viewpoint change. To this end, we generate labeled 2D images from a photo-realistic 3D dataset. These images are used for training a neural network based feature detector. We further present an algorithm for automatically generating labels of repeatable 2D features, and present a fast, easy to use test algorithm for evaluating a detector in an 3D environment.
Introduction
In computer vision, "features" are landmarks such as corners or edges that allow for matching different images of the same scene. This capability is fundamental to Augmented Reality (AR), autonomous navigation, and 3D reconstruction. Historically, feature detectors have been trained and evaluated using homographies—mathematical transformations that simulate camera movement by warping flat 2D images. While effective for planar surfaces, this approach fails in complex 3D environments where camera movement induces depth disparity and occlusions.
Current deep-learning models, such as SuperPoint, are often trained on these "flat" assumptions. Consequently, their performance degrades in cluttered indoor spaces where geometric complexity is high. This research addresses the "homography gap" by training detectors to be repeatable under true 3D viewpoint changes.
Methodology
The proposed pipeline transitions from simulated 2D movement to authentic 3D viewpoint changes using a multi-step process:
- 3D Rendering and Data Acquisition: The Gibson simulator was used to render thousands of photo-realistic 2D images from the Matterport3D dataset (150 building captures). This provided over 400,000 images with precise ground-truth camera positions and pixel-level depth data.
- The "Confidence Painter": To generate ground-truth labels, detections from multiple existing algorithms (e.g., SIFT, MagicPoint) were projected into a 3D Octomap (a voxel-based grid).
- Automated Labeling: Features were identified as repeatable if they were consistently detected from multiple viewpoints in the 3D map. These stable 3D points were then projected back into 2D images to create a high-quality training set.
- Model Training: A neural network based on the SuperPoint architecture was trained on this 3D-derived dataset, resulting in the Superpoint-Gibson model.
Results and Evaluation
The Superpoint-Gibson model was evaluated against traditional methods (SIFT, SURF, FAST) and deep-learning models (MagicPoint, SuperPoint-COCO, LF-Net) using the ScanNet dataset, which provides real-world RGB-D video of indoor environments.
Quantitative analysis demonstrates that Superpoint-Gibson achieves higher repeatability. At a 3-pixel threshold, it detected an average of 10.75 features per image, representing a 13% improvement over MagicPoint. While models trained on 2D homographies (like SuperPoint-COCO) perform well on flat surfaces, they suffer significant performance penalties in real 3D scenes compared to the Gibson-trained model.
Qualitatively, the proposed method successfully identifies stable corners on complex objects, such as office equipment, which are often missed by state-of-the-art methods trained on simpler geometric assumptions.
Conclusion
This work highlights that training on real 3D geometry is fundamentally different from training on 2D warps. By leveraging consensus-based labeling and photo-realistic 3D datasets, we can produce feature detectors significantly more robust to real-world geometric complexity. Future work will focus on mitigating artifacts introduced by neural rendering in simulators and expanding the dataset to include more diverse lighting and outdoor environments.
Summary updated