Mirrored Light Field Video Camera Adapter

Dorian Tsai, Donald G. Dansereau, Steve Martin, Peter I. CorkearXiv 2016 · OpenReview

TL;DR

This paper introduces the "MirrorCam," a low-cost, DIY light field camera adapter that uses an array of mirrors to capture 4D light field video at frame rates suitable for robotics. It addresses the lack of affordable and accessible light field cameras for real-time robotic applications by providing a detailed design, construction, and calibration methodology, with plans for an open-source release.

What Problem Does This Paper Address?

Robotics demands real-time, low-cost, and computationally inexpensive camera systems. While light field cameras offer significant advantages for computer vision and robotics—capturing spatial and angular information to enable depth extraction, handle occlusions, and process non-Lambertian surfaces—commercially available options are often expensive, bulky, and lack real-time video capabilities. Creating a full camera array to achieve similar results introduces synchronization, bulk, and bandwidth challenges. This paper tackles the need for an accessible, real-time light field camera solution specifically for the robotics community.

Methods

The core innovation is the "MirrorCam," a mirror-based adapter that splits a single camera's field of view into multiple sub-images using an array of precisely positioned planar mirrors. This creates an array of virtual cameras with overlapping fields of view, effectively capturing a light field.

Key methodological aspects include:

  • Mirror Array Design & Optimization: The design utilizes an optimization routine that balances the desire for perfectly overlapping fields of view (FOV) with the relative positioning of virtual camera poses. A single tunable parameter guides this optimization, allowing for a trade-off between depth of field and FOV. Notably, the design allows for non-rectangular sub-images to maximize FOV overlap.
  • Construction: The adapter mount is 3D printed, and laser-cut acrylic mirrors are used. The design is specifically tailored for the readily available Logitech C920 webcam.
  • Decoding & Calibration: The process involves two main steps:
  • Intrinsic Camera Calibration: Standard calibration of the base camera (Logitech C920) is performed.
  • Mirror Geometry Estimation: A Levenberg-Marquardt optimization estimates the mirror geometry by minimizing the error between expected and observed checkerboard corner locations. A key difference from prior work is the use of 3-DOF reflection matrices for each mirror, reducing the overall degrees of freedom and improving robustness compared to 6-DOF transformations.
  • Light Field Feature Extraction: The paper extends prior work by augmenting feature detection and representation. It enforces light field geometry constraints to improve selectivity and reduce spurious detections. Features are augmented with local light field slope to implicitly encode depth. Point-plane correspondence is used to reject features that break this relationship, ensuring consistency in 3D spatial locations.

Datasets

Not discussed in the paper. The paper focuses on the design, construction, and calibration of the MirrorCam, and uses checkerboard patterns for calibration.

Key Insights and Ideas

  • DIY Light Field for Robotics: The MirrorCam demonstrates that a functional, real-time light field camera can be built affordably using readily available components and 3D printing, making advanced imaging accessible to robotics researchers.
  • Mirror Geometry Optimization: The approach of optimizing mirror positions and normals with a single tunable parameter to balance FOV overlap and virtual camera pose is a novel way to manage the complexity of mirror-based light field capture.
  • 3-DOF Mirror Calibration: Using 3-DOF reflection matrices for mirror calibration significantly simplifies the process and improves robustness, making it more practical for real-world robotic applications.
  • Open-Source Potential: The authors' intention to release the design, methodology, and code as open-source is a significant contribution, fostering wider adoption and further development within the research community.
  • Non-Rectangular Sub-images: Allowing for non-rectangular sub-images in the optimization is a clever way to maximize the usable FOV overlap between virtual cameras.

Key Results

  • The MirrorCam successfully reproduces a light field, demonstrating its viability as a low-cost, accessible alternative to commercial light field cameras.
  • The calibration routine achieved a 3D spatial reprojection root mean square (RMS) error of 1.80 mm, indicating that the camera design, manufacturing, and calibration resulted in observations close to an ideal light field.
  • The current unoptimized MATLAB implementation takes 5 seconds per frame for operation, with real-time performance expected after optimization.
  • The paper compares the MirrorCam favorably to other light field camera systems in terms of accessibility, customizability, and open-source availability.

Limitations and Future Work

  • Resolution and FOV: The paper acknowledges that the MirrorCam has a lower resolution and a lower FOV compared to some other approaches (e.g., a 3x3 array has 1/3 the FOV of the base camera).
  • Decoding Complexity: The decoding process is described as more complex than some alternatives.
  • Manufacturing Imperfections: The thickness of acrylic mirrors, epoxy, and slight bending/rippling of the mirrors caused image distortion that was not fully accounted for in the current calibration.
  • Calibration Limitations: The current calibration does not account for manufacturing aspects like mirror thickness or epoxy. It also only uses images without mirrors for initializing camera intrinsics, suggesting a future improvement would be to jointly consider all images.
  • Future Work: The authors plan to validate the MirrorCam in terms of image refocusing, depth estimation, and perspective shift against commercial cameras. They also aim to explore non-quadrilateral sub-images and optimize the system for real-time performance.

Summary updated

Abstract

This paper proposes the design of a custom mirror-based light field camera adapter that is cheap, simple in construction, and accessible. Mirrors of different shape and orientation reflect the scene into an upwards-facing camera to create an array of virtual cameras with overlapping field of view at specified depths, and deliver video frame rate light fields. We describe the design, construction, decoding and calibration processes of our mirror-based light field camera adapter in preparation for an open-source release to benefit the robotic vision community.
BibTeX
@misc{tsai2016mirrored,
  author = {Dorian Tsai and Donald G. Dansereau and Steve Martin and Peter I. Corke},
  title = {{Mirrored Light Field Video Camera Adapter}},
  year = {2016},
  eprint = {1612.05335},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1612.05335},
}
Figure