Distinguishing Refracted Features Using Light Field Cameras With Application to Structure From Motion

Dorian Tsai, Donald G. Dansereau, Thierry Peynot, Peter Corke

TL;DR

This paper introduces a novel method for robots to distinguish between "normal" (Lambertian) and "tricky" (refracted) features in images captured by light field cameras. This allows for more robust robotic vision, particularly in structure-from-motion tasks, by filtering out problematic refracted features that can lead to errors.

What Problem Does This Paper Address?

Robots need to interact with their environment, and many everyday objects are refractive – think wine glasses, water bottles, or clear plastic packaging. These objects pose a significant challenge for standard computer vision algorithms, especially those used for tasks like Structure from Motion (SfM). SfM algorithms typically rely on identifying and tracking distinct visual features (like corners or edges) across multiple images to reconstruct a 3D scene and the robot's movement. However, refractive objects warp the appearance of the background behind them, making these features appear to move in ways that violate the assumptions of SfM, leading to inaccurate reconstructions or even complete failure. Existing methods for dealing with refractive objects often require large, specialized camera setups or struggle with smaller baselines, limiting their practical application for mobile robots.

Methods

The core of this paper's contribution lies in a new way to analyze features captured by light field (LF) cameras. LF cameras capture not just a 2D image but also information about the direction of light rays, effectively providing multiple views of the scene simultaneously.

The authors propose:

  • Textural Cross-Correlation for Feature Curves: Instead of relying solely on image intensity, they use textural cross-correlation to track how a feature's appearance changes across the different views within the LF. This creates "feature curves" that represent the apparent motion of a feature in the 4D light field.
  • 4D Planarity and Slope Consistency: For Lambertian features, these curves should follow a predictable, linear path in 4D space, which can be represented by a hyperplane. Refracted features, however, deviate from this linear behavior. The method fits a plane to these feature curves in 4D and measures the "planar consistency" (how well the feature fits a plane). Crucially, they also introduce "slope consistency," which checks if the rate of change of the feature's position across views is consistent in different directions. This is particularly effective for distinguishing features that are only refractive in one dimension (like a cylinder).
  • Refracted Feature Distinguisher: By combining planar consistency and slope consistency, they create a distinguisher that can reliably identify features that are likely refracted.

Datasets

The paper uses publicly available light field datasets, specifically from the Stanford New Light Field Database [16], which includes scenes with a crystal ball. They also use data captured with a Lytro Illum camera, a lenslet-based LF camera, to evaluate performance with smaller baselines and different refractive object types (a glass sphere and an acrylic cylinder).

Key Insights and Ideas

  • Refraction is a Feature Motion Problem: The key insight is that refraction fundamentally alters how features appear to move across different viewpoints captured by an LF camera. This motion is the signal they exploit.
  • Slope Consistency is Crucial for Small Baselines: While planar consistency is important, for LF cameras with small baselines (where refractive effects are subtle), slope consistency becomes a much more powerful indicator of refraction. This is a significant advantage for practical, compact robot vision systems.
  • Refracted Features are "Bad" for SfM: The paper clearly demonstrates that even a small percentage of refracted features can significantly degrade SfM performance, leading to higher reprojection errors and even complete failure to reconstruct the scene.
  • Filtering is More Than Just Outlier Rejection: Standard outlier rejection methods (like RANSAC) in SfM are not sufficient on their own to handle refractive features. Proactively filtering them out before SfM is essential for robust reconstruction.

Key Results

  • Improved Detection Rates: The proposed method achieves a 34.3% higher true positive rate (TPR) compared to a previous state-of-the-art method (Xu's method) for large baselines. For smaller baselines, their method shows a 38.9% TPR increase and is able to detect refracted features where Xu's method fails entirely.
  • Reduced Reprojection Error: Rejecting refracted features using their distinguisher in an SfM pipeline resulted in up to 42.4% lower reprojection error and a lower failure rate when approaching refractive objects.
  • Robustness with Lenslet Cameras: The method successfully distinguishes refracted features from refractive objects (sphere and cylinder) using a lenslet-based LF camera, which has much smaller baselines than previously considered.
  • Mitigation of "Phantom Points": By filtering out refracted features, the SfM reconstruction produced fewer "phantom points" (erroneous 3D points in empty space), which can lead to incorrect occupancy maps for robots.

Limitations and Future Work

  • Dependence on Background Texture: The method's effectiveness relies on the presence of sufficient background texture distorted by the refractive object. If the object is very clear or the background is uniform, detection might be challenging.
  • Threshold Tuning: The current method uses empirically set thresholds, which limits its flexibility. Future work aims to implement ridge-following methods for more robust feature curve extraction.
  • Handling Occlusions and Specularities: While the method can distinguish some non-Lambertian effects, it doesn't explicitly differentiate between refraction, occlusions, and specular reflections. Future work will address these.
  • Exploiting Refractive Content: Currently, the paper focuses on rejecting refractive features. Future work plans to exploit the refractive content for robot motion and refractive shape recovery.
  • Camera Calibration and Surface Properties: The authors note that poor camera calibration or specific surface properties (like specular reflections) can also cause features to appear refractive, which is an area for further investigation.

Summary updated

Abstract

To be effective, robots will need to reliably operate in scenes with refractive objects in a variety of applications; however, refractive objects can cause many robotic vision algorithms, such as structure from motion, to become unreliable or even fail. We propose a novel method to distinguish between refracted and Lambertian image features using a light field camera. While previous refracted feature detection methods are limited to light field cameras with large baselines relative to the refractive object, our method achieves comparable performance, and we extend these capabilities to light field cameras with much smaller baselines than previously considered, where we achieve up to 50% higher refracted feature detection rates. Specifically, we propose to use textural cross correlation to characterize apparent feature motion in a single light field, and compare this motion to its Lambertian equivalent based on 4-D light field geometry. For structure from motion, we demonstrate that rejecting refracted features using our distinguisher yields lower reprojection error, lower failure rates, and more accurate pose estimates when the robot is approaching refractive objects. Our method is a critical step toward allowing robots to operate in the presence of refractive objects.
BibTeX
@article{tsai2019distinguishing,
  author = {Dorian Tsai and Donald G. Dansereau and Thierry Peynot and Peter Corke},
  title = {{Distinguishing Refracted Features Using Light Field Cameras With Application to Structure From Motion}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2019},
  url = {https://openreview.net/forum?id=VGGp2PzsoGL},
}
Figure