TL;DR
This paper introduces a novel two-stage method for dense 3D scene reconstruction from light-field (LF) images, specifically addressing the challenges posed by rolling shutter (RS) effects. By leveraging a 2D Gaussian Splatting representation and an optimization-based approach, the method effectively disentangles scene geometry and camera motion, producing high-quality depth maps and compensated appearances even with significant RS distortions.
What Problem Does This Paper Address?
Modern light-field cameras offer rich 4D information for 3D scene reconstruction, but most consumer cameras utilize rolling shutter (RS) sensors. RS sensors capture image data line by line, leading to distortions when there's camera or object motion during capture. This "RS effect" complicates traditional depth estimation by entangling motion and shape information. Existing methods often rely on global shutter assumptions, multiple captures, or require explicit 2D point correspondences, which can lead to sparse reconstructions. This paper tackles the problem of achieving dense, accurate 3D scene reconstruction from single light-field captures affected by rolling shutter, without requiring prior knowledge of scene shape or motion.
Methods
The proposed method employs a two-stage approach:
- Stage 1: 2D Gaussian Splatting Initialization: The core of this stage is adapting the 3D Gaussian Splatting technique for LF images. Instead of full 3D Gaussians, they use 2D Gaussians to represent scene elements. This component estimates a dense representation of the scene, including the 2D location of Gaussians in the central sub-aperture image, their disparity, size, and intensity. This initial representation is "RS agnostic" and captures the scene shape "up to a motion." The key adaptation is using the central row of sub-aperture images to ensure that a given 3D point is projected onto the same line across these images, thus ensuring they are captured at the same time instant. This yields a geometry affected by a "3D rolling shutter effect."
- Stage 2: Motion Compensation and Refinement: In this stage, the 2D Gaussian Splatting representation is used to estimate camera motion parameters (rotation and translation). This is achieved through a differentiable rendering and "render-and-compare" strategy. By re-projecting the estimated static shape (the scene geometry corrected for motion) into the RS-deformed views, an appearance intensity error is minimized. This process disentangles the RS deformation from the true scene geometry and motion, allowing for the estimation of motion parameters that correct for both the RS effect and camera movement. The output is an RS-compensated dense depth map and appearance.
Datasets
- 4D LF dataset [6]: Used for quantitative evaluation.
- RSLF dataset [19]: The first dataset specifically for evaluating rolling shutter effects in LF images.
- RSLF+ dataset (New): The authors introduce their own synthetic dataset, RSLF+, designed to address limitations of existing datasets. It features scenes with higher texture, varying geometric details (from flat checkerboards to vegetation), and a wide range of motion scenarios across six dimensions of velocity. It also provides ground truth depth maps, central views with double field of view, and visibility masks to account for occlusions in RS-deformed scenes.
Key Insights and Ideas
- Leveraging RS for Motion: Instead of treating RS as a purely detrimental effect, the paper explores how the distortions themselves can provide cues for motion estimation when combined with LF data.
- 2D Gaussian Splatting for LF: Adapting 3D Gaussian Splatting to a 2D representation tailored for LF images is crucial. This allows for dense scene representation without explicit point correspondences, a significant advantage over prior work.
- "Up to a Motion" Geometry: The initial stage produces a scene representation that is geometrically correct but deformed by the RS effect and camera motion. This "up to a motion" representation is key to the subsequent motion estimation.
- Differentiable Rendering for Motion Estimation: The use of differentiable rendering within a "render-and-compare" framework is central to estimating the motion parameters by minimizing appearance errors.
Key Results
- The proposed method outperforms existing baselines (Jeon-CVPR, OACC-Net, RSLF-d) on the RSLF dataset in terms of depth accuracy metrics (abs diff, rmse, δ < 1.25), particularly in static (GS) and slow motion scenarios.
- On the new RSLF+ dataset, the method also shows superior performance, demonstrating its ability to handle more textured scenes and complex motions.
- Qualitative results show that the method effectively compensates for RS distortions, producing more precise depth maps and compensated appearances compared to prior methods like RSLF-d.
- The computational time is approximately 10 minutes per scene for the entire pipeline on a 16GB GPU.
Limitations and Future Work
- Low-Textured Regions: The paper notes that performance can struggle in low-textured regions, a common challenge in many computer vision tasks.
- Static vs. Motion: While the method excels with RS effects, the paper mentions that in purely static (GS) scenarios, the performance is comparable to other methods, but the small reduction in quality compared to methods not designed for RS is acknowledged.
- Future Work: The authors suggest that the rich information extraction and modeling from a single capture has strong potential for various vision tasks. The release of their source code and dataset will facilitate further research in this area.
Summary updated