Joint 3D Shape and Motion Estimation From Rolling Shutter Light-Field Images

Hermès McGriff, Renato Martins, Nicolas Andreff, Cédric DemonceauxWACV 2024

TL;DR

This paper introduces a novel method for simultaneously estimating the 3D shape of a scene and the camera's motion using rolling shutter light-field (RSLF) images. By modeling the unique distortions introduced by rolling shutter sensors in light-field cameras, the approach achieves robust 3D reconstruction even with significant camera movement, a feat challenging for existing methods.

What Problem Does This Paper Address?

Traditional 3D reconstruction methods often rely on global shutter (GS) cameras, where the entire image is captured simultaneously. However, many modern consumer and specialized cameras, particularly those used in robotics and micro-robotics, employ rolling shutter (RS) sensors. RS sensors capture images sequentially, leading to distortions (e.g., warping, skewing) when the camera or scene is in motion. Light-field (LF) cameras, which capture directional information, offer rich depth cues. Combining LF imaging with RS sensors presents a unique challenge: how to accurately reconstruct 3D scene geometry and camera motion when the sensor itself introduces motion-dependent distortions. Existing methods for RS or LF reconstruction often struggle with this combined scenario, particularly when significant camera motion is involved.

Methods

The core of this work lies in a novel projection model for RSLF cameras. This model unifies the geometric properties of light-field imaging with the temporal distortions of rolling shutters. The authors propose a two-stage approach:

  1. Linear Initialization: A coarse estimate of the 3D scene structure is obtained using a multi-view stereo strategy applied only along the horizontal dimension. This ensures that points are captured at the same instant, mitigating RS distortions for this initial step. This step is crucial for the convergence of the subsequent non-linear optimization.
  2. Non-linear Bundle Adjustment: A bundle adjustment framework is employed to jointly refine the 3D scene structure (point cloud) and estimate the camera's motion parameters (rotation, translation, angular velocity, linear velocity). This stage minimizes a re-projection error, taking into account the RSLF projection model. Regularization is also applied to improve the estimation of the rotation center.

Datasets

The authors introduce a new benchmark dataset called Rolling Shutter Light Fields (RSLF). This dataset comprises:

  • Eleven light-field sequences per scene.
  • Seven different synthetic scenes, including photo-realistic ones derived from the Habitat-Matterport benchmark.
  • Various motion scenarios: static (GS equivalent), slow motion, and fast motion (with velocities three times higher than slow).
  • Each light field has a dimension of 9x9 micro-images with 512x512 pixels.
  • Ground truth depth maps are provided for the scenes.

This dataset aims to address the lack of publicly available RSLF data and serve as a common benchmark for evaluating and advancing RS applications in scene analysis with light fields.

Key Insights and Ideas

  • Leveraging RS Distortions: Instead of treating rolling shutter distortions as a nuisance, the paper demonstrates how they can be actively leveraged, in conjunction with light-field data, to estimate camera motion.
  • Unified RSLF Model: The proposed projection model elegantly combines the capabilities of light-field cameras with the sequential acquisition of rolling shutter sensors, providing a generic framework for both GS and RS scenarios.
  • Importance of Initialization: The linear initialization step is shown to be critical for the successful convergence of the non-linear bundle adjustment, highlighting the challenges of directly optimizing in the presence of RS distortions.
  • Rigid Scene Assumption: The method assumes rigid scenes and uniform camera motion, which simplifies the problem but also points towards future extensions for dynamic objects or non-uniform motion.

Key Results

  • The proposed method achieves state-of-the-art performance across various metrics (abs rel, abs diff, RMS, δ < 1.25, δ < 1.25², δ < 1.25³) on their newly created RSLF dataset, significantly outperforming existing methods, especially in "fast" motion scenarios.
  • For GS scenarios, the method shows competitive results with specialized GS algorithms.
  • Qualitative results demonstrate the ability of the proposed method to accurately reconstruct scene geometry and fit object shapes even with significant rolling shutter distortions and camera motion, unlike competitors whose reconstructions are visibly warped.

Limitations and Future Work

  • Rigid Scene Assumption: The current model assumes rigid scenes and uniform camera motion (zero acceleration).
  • Future Work: The authors suggest extending the approach to handle dynamic objects independently and to recover from non-rigid scenes by decomposing the plenoptic image into horizontal bands. They also plan to investigate piecewise decomposition for accelerated motions and explore applications in robot vision, manipulation, and micro-robotics.

Summary updated

BibTeX
@inproceedings{mcgriff2024joint,
  author = {Hermès McGriff and Renato Martins and Nicolas Andreff and Cédric Demonceaux},
  title = {{Joint 3D Shape and Motion Estimation From Rolling Shutter Light-Field Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure