Refractive Light-Field Features for Curved Transparent Objects in Structure from Motion

Dorian Tsai, Peter Corke, Thierry Peynot, Donald G. DansereauarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel "Refracted Light-Field Feature" (RLFF) designed to overcome the challenges posed by transparent, refractive objects in robotic vision. By leveraging light-field cameras, these new features enable significantly more robust and accurate structure-from-motion (SfM) performance in complex environments, leading to better 3D reconstructions and camera pose estimation.

What Problem Does This Paper Address?

Robots often struggle to perceive and navigate environments containing transparent objects like glass or acrylic. Traditional computer vision algorithms, which often assume objects have fixed appearances regardless of viewpoint (Lambertian surfaces), fail because the appearance of refractive objects changes dramatically with viewpoint due to light distortion. This leads to incorrect camera trajectory estimates and 3D shape reconstructions, and can even cause algorithms to fail entirely. Existing methods for handling refractive objects often rely on pre-existing 3D models, which are not always available or practical.

Methods

The core innovation is the Refracted Light-Field Feature (RLFF). Unlike traditional 2D features that rely on object texture or geometry, RLFFs are derived from the patterns of light refracted through curved transparent objects.

  1. Light-Field (LF) Camera Exploitation: The paper uses LF cameras, which capture multiple views of a scene in a single exposure. This allows them to capture view-dependent effects like refraction.
  2. Astigmatic Lens Analogy: The optical behavior of light passing through a curved refractive object is modeled as an astigmatic lens. This lens has two focal lines at different depths, and the interval connecting these lines is key.
  3. RLFF Definition: The RLFF is defined by two characteristic 3D points that represent the endpoints of this "interval of Sturm." These points are stable with respect to viewpoint changes.
  4. Feature Detection and Extraction: The method uses existing 2D feature detectors (like SIFT) on LF sub-images to find initial feature candidates. These candidates are then processed to estimate the RLFF parameters (the two 3D points) by fitting a generalized point-plane correspondence model to the light field data.
  5. Integration with SfM: To use RLFFs with existing SfM pipelines (like COLMAP), the 3D RLFF points are projected into 2D image planes. Two strategies are proposed: projecting into the central view (RLFF mono) or into a stereo pair of views (RLFF stereo) to preserve more depth information.

Datasets

The authors created a new dataset consisting of 218 light-field images from 20 challenging scenes. These scenes include various refractive objects (glasses, bottles, spheres) and textured backgrounds, captured with different camera motions. The dataset and associated code are made available.

Key Insights and Ideas

  • Refraction as a Feature Source: The paper demonstrates that the distortion caused by refraction itself can be a reliable source of visual features, rather than just a nuisance to be filtered out.
  • LF Cameras are Crucial: Light-field cameras are essential for capturing the view-dependent nature of refraction needed to define these new features.
  • Astigmatic Optics as a Model: The analogy to astigmatic lenses provides a strong theoretical foundation for understanding and modeling the optical behavior of refractive objects.
  • RLFFs Complement, Not Replace: The RLFF approach can coexist with traditional Lambertian features, allowing SfM systems to leverage both for more robust scene understanding. Lambertian points result in a zero-length interval of Sturm, effectively appearing as single points.

Key Results

  • Improved SfM Convergence: The proposed RLFF methods significantly increased SfM convergence rates, succeeding in 95% (mono) and 90% (stereo) of scenes, compared to 75% (mono) and 60% (stereo) for SIFT. This improvement is most pronounced in scenes with a high proportion of refractive content.
  • Enhanced 3D Reconstruction: RLFFs led to a higher number of 3D points in reconstructions and improved overall matching scores.
  • More Accurate Camera Trajectories: The RLFF methods demonstrated substantially better accuracy in estimating camera pose (both translational and rotational error), especially in challenging sequences.
  • Robustness to Refraction: The RLFF approach is particularly advantageous in scenes dominated by refractive objects, where traditional methods often fail.

Limitations and Future Work

  • Spherical Lenses: The current method struggles with perfectly spherical lenses, as they produce well-formed images that behave like Lambertian surfaces. These are still used in SfM but not identified as refracted.
  • Texture Requirement: Like most feature-based methods, RLFFs require visible texture behind the refractive object to be detected. Frosted or highly complex surfaces will not work.
  • Defocus Effects: Strong defocus effects through highly distorting objects are not considered.
  • Future Work: The authors plan to develop direct LF feature detectors for RLFFs (similar to LiFF for Lambertian features), explore applications in visual servoing and place recognition, and generalize the approach to reflective surfaces. They also suggest improving descriptor design to avoid spurious matches between front and back focal points.

Summary updated

Abstract

Curved refractive objects are common in the human environment, and have a complex visual appearance that can cause robotic vision algorithms to fail. Light-field cameras allow us to address this challenge by capturing the view-dependent appearance of such objects in a single exposure. We propose a novel image feature for light fields that detects and describes the patterns of light refracted through curved transparent objects. We derive characteristic points based on these features allowing them to be used in place of conventional 2D features. Using our features, we demonstrate improved structure-from-motion performance in challenging scenes containing refractive objects, including quantitative evaluations that show improved camera pose estimates and 3D reconstructions. Additionally, our methods converge 15-35% more frequently than the state-of-the-art. Our method is a critical step towards allowing robots to operate around refractive objects, with applications in manufacturing, quality assurance, pick-and-place, and domestic robots working with acrylic, glass and other transparent materials.
BibTeX
@misc{tsai2021refractive,
  author = {Dorian Tsai and Peter Corke and Thierry Peynot and Donald G. Dansereau},
  title = {{Refractive Light-Field Features for Curved Transparent Objects in Structure from Motion}},
  year = {2021},
  eprint = {2103.15349},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.15349},
}
Figure