Out-of-focus: Learning Depth from Image Bokeh for Robotic Perception

Eric Cristofalo, Zijian Wang

Abstract

In this project, we propose a novel approach for estimating depth from RGB images. Traditionally, most work uses a single RGB image to estimate depth, which is inherently difficult and generally results in poor performance, even with thousands of data examples. In this work, we alternatively use multiple RGB images that were captured while changing the focus of the camera's lens. This method leverages the natural depth information correlated to the different patterns of clarity/blur in the sequence of focal images, which helps distinguish objects at different depths. Since no such data set exists for learning this mapping, we collect our own data set using customized hardware. We then use a convolutional neural network for learning the depth from the stacked focal images. Comparative studies were conducted on both a standard RGBD data set and our own data set (learning from both single and multiple images), and results verified that stacked focal images yield better depth estimation than using just single RGB image.

Introduction

Estimating depth from a single 2D RGB image is an "ill-posed" problem; a single image lacks the geometric information necessary to distinguish between a small object close by and a large object far away without heavy reliance on semantic priors. While humans use peripheral blur and "bokeh" to perceive depth, robotic vision systems typically struggle with this. Traditional "depth-from-focus" algorithms rely on hand-coded mathematical measures of sharpness (like Laplacian or Wavelet-based methods), which often fail in low-texture areas or varying light. This project explores whether deep learning can bypass hand-coded features and learn the direct mapping from a stack of blurred images to a dense depth map.

Methodology

The researchers propose a supervised learning approach using a Convolutional Neural Network (CNN) to interpret focal stacks.

The Focal Stack

Instead of one image, the input is a "stack" of $N=52$ images. Each image is captured with a slightly different focal length, ranging from the camera's minimum to maximum focus settings. The network doesn't just look at what is in focus; it learns from the pattern of blur. The way an object transitions from sharp to blurry across the stack provides a physical signature of its distance from the lens.

CNN Architecture

The team designed a CNN inspired by the VGG network but shallower to prevent overfitting on their specific dataset. It consists of six convolutional layers (3x3 kernels) with ReLU activation, four max-pooling layers to reduce spatial dimensions, and two fully connected layers that synthesize the extracted features into a final depth map.

CNN architecture for depth estimation from stacked focal images.

Data Collection and Processing

Because no standard dataset for "focal stack depth learning" existed, the authors built their own using a custom hardware rig.

Hardware Setup

The rig features a Microsoft Kinect V2 (for ground-truth depth) and a Logitech C920 webcam (to capture the focal stacks).

Custom hardware setup featuring a Microsoft Kinect V2 and a Logitech C920 webcam.

Dataset and Registration

The dataset contains 344 unique office scenes, augmented to 3,674 examples using random 3D perspective transformations. Because the webcam and the depth sensor were physically offset, the team used a perspective transformation to align the RGB focal stack with the ground-truth depth labels.

Results and Discussion

The researchers compared three experiments: (1) Single image from the NYU Depth V2 dataset, (2) Single image from their own dataset, and (3) The full focal stack.

  • Quantitative Improvement: The focal stack achieved the lowest error rates across the board. The Test Mean Squared Error (MSE) was 335.88 for the focal stack versus 355.27 for a single image. The Test Mean Absolute Error (MAE) was 12.11 for the focal stack versus 12.53 for a single image.
  • Qualitative Sharpness: Qualitatively, the depth maps generated from focal stacks were much better at capturing structural outlines, such as the rectangular shape of a doorway or the edges of a desk, which were often "smeared" or lost in single-image predictions.
  • End-to-End Learning: Unlike traditional methods that require high-contrast textures to calculate "sharpness," the CNN can learn to interpret subtle gradients and color patterns, making it more robust in varied environments.

Conclusion and Future Work

This work demonstrates that stacked focal images provide critical geometric cues that significantly outperform traditional single-image depth estimation. The core insight is that even a small amount of focal variation provides a massive boost in accuracy compared to a single high-resolution image, as it resolves the inherent ambiguity of monocular vision.

Future work involves using hardware with wider apertures (like DSLR cameras) to create more pronounced bokeh, which is expected to lead to even higher accuracy. Additionally, the authors aim to optimize the pipeline for real-time processing to allow autonomous mobile robots to navigate and avoid obstacles using only focal information.

Summary updated

Abstract

In this project, we propose a novel approach for estimating depth from RGB images. Traditionally, most work uses a single RGB image to estimate depth, which is inherently difficult and generally results in poor performance, even with thousands of data examples. In this work, we alternatively use multiple RGB images that were captured while changing the focus of the camera's lens. This method leverages the natural depth information correlated to the different patterns of clarity/blur in the sequence of focal images, which helps distinguish objects at different depths. Since no such data set exists for learning this mapping, we collect our own data set using customized hardware. We then use a convolutional neural network for learning the depth from the stacked focal images. Comparative studies were conducted on both a standard RGBD data set and our own data set (learning from both single and multiple images), and results verified that stacked focal images yield better depth estimation than using just single RGB image.
BibTeX
@misc{cristofalo2017out,
  author = {Eric Cristofalo and Zijian Wang},
  title = {{Out-of-focus: Learning Depth from Image Bokeh for Robotic Perception}},
  year = {2017},
  eprint = {1705.01152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.01152},
}
Figure