Efficient Depth Enhancement Using a Combination of Color and Depth Information

Kyungjae Lee, Yuseok Ban, Sangyoun Lee2017 · DOI

TL;DR

This paper introduces an efficient algorithm to improve the quality of depth images by combining color and depth information. It addresses common issues like missing depth values and noisy boundaries, leading to more accurate 3D scene reconstruction and applications.

What Problem Does This Paper Address?

Depth sensors, like the Kinect, are invaluable for capturing 3D information, enabling applications in areas like human-computer interaction, 3D modeling, and augmented reality. However, these sensors often produce depth images with inherent problems: missing depth values (holes) due to sensor limitations (e.g., infrared reflection issues) and noisy boundaries around objects. These imperfections significantly degrade the performance of downstream applications that rely on accurate depth data. Existing methods for depth enhancement often struggle with computational complexity, introduce blurriness, or fail to accurately recover object shapes, especially in areas with large holes.

Methods

The proposed method combines color and depth information in a three-phase approach:

  1. Image Acquisition and Preprocessing: Color and depth sensors are calibrated for alignment. The raw depth image undergoes preprocessing steps including morphological operations (erosion, dilation) and median filtering to reduce noise and stabilize the depth data.
  2. Image Segmentation and Weighted Linear Combination of Spatial Filtering:
    • Coordinate Transformation: Depth image pixels are transformed into an x-D coordinate system, which helps in separating objects from the background due to similar depth values within objects.
  3. Image Segmentation: A connected component labeling algorithm is applied in the x-D domain to identify object regions. Object detection from the color image is used to refine these regions.
  4. Inner Hole Filling: A novel weighted linear combination of spatial filtering algorithms is used to fill "inner holes" (gaps and interference within object regions). This method considers both the global properties of segmented depth regions and local depth information around the holes, specifically avoiding front-object depth values for rear-object hole filling.
  5. Depth Recovery by Asynchronous Cellular Automata (ACA): This phase addresses distorted object shapes and remaining depth holes. It utilizes an ACA-based depth recovery algorithm inspired by "GrowCut." Cells (pixels) have states representing depth, color features, and strength. The algorithm iteratively fills depth holes by simulating an "attack" from neighboring cells with valid depth information, propagating valid depth values into the holes. Asynchronous updates in ACA significantly speed up the process compared to synchronous methods. The use of a Lab color space for feature vectors is also employed to better represent human color perception and improve distance calculations.

Datasets

The paper evaluates its method on two main datasets:

  • Fernandez's Kinect dataset: Used for quantitative evaluation of object segmentation.
  • Tsukuba Stereo Dataset: A benchmark dataset for depth image enhancement, used for quantitative and qualitative evaluations of the proposed depth recovery method.

The authors also mention using their own captured real-world Kinect data and a public Kinect dataset [43,55] for qualitative evaluation of the overall enhancement method.

Key Insights and Ideas

  • Leveraging Color for Depth: The core idea is that color information, being richer in detail and less prone to certain types of noise, can guide and improve depth estimation, especially in challenging regions.
  • Distinguishing Hole Types: The method explicitly addresses different types of depth holes: "inner holes" within objects (due to occlusion or reflection) and holes related to object boundaries or background.
  • Asynchronous Cellular Automata for Speed: Using ACA instead of synchronous cellular automata drastically reduces computation time by allowing immediate state updates, making the method more efficient.
  • Lab Color Space for Better Feature Representation: Shifting from RGB to the Lab color space for feature vectors improves the algorithm's performance by better aligning with human color perception and facilitating more accurate distance metrics.
  • "GrowFill" Metaphor: The depth recovery process is conceptually explained using a "bacterium" (attacker) and "defender" metaphor, which intuitively describes how valid depth information spreads into invalid regions.

Key Results

  • Object Segmentation: The proposed method achieved competitive F₁ scores on the Fernandez's Kinect dataset, performing comparably to or better than existing methods like MOG4D, CB1D, CB4D, and DECB, with particular strengths in extracting object-oriented regions with low noise.
  • Inner Hole Filling: Qualitative results show the proposed method effectively fills inner holes, producing less blurred results and more accurate depth values compared to flood-fill, Navier-Stokes, fast marching, and joint bilateral filtering methods.
  • Depth Recovery (Tsukuba Stereo Dataset):
    • Quantitative: The method significantly improved PSNR (by an average of 14.485 dB with Moore's neighborhood) and SSIM (by 0.116) compared to the original depth image. It outperformed existing methods like Telea [34] and Gong [33] in PSNR and SSIM.
  • Speed: The ACA-based approach, especially when combined with NDM, achieved substantial speedups. The proposed method (ACA + NDM) was about 6.5-8 times faster than pure SCA-based methods and about 1.3-1.8 times faster than pure ACA-based methods, with minimal quality degradation.
  • Qualitative: Enhanced depth images showed clearer object boundaries and more complete depth information. Synthesized stereoscopic images demonstrated a noticeable improvement in visual quality.

Limitations and Future Work

  • Computational Complexity: While significantly improved by ACA, the method still involves iterative processes, and further optimization for real-time applications might be explored.
  • Parameter Tuning: Some parameters (e.g., weights in the weighted linear combination, neighborhood sizes) are empirically determined and might require fine-tuning for different scenarios.
  • Future Work: The authors suggest that the object segmentation algorithm could be easily utilized for activities such as view synthesis and virtual conference systems, implying potential for further integration and application development. The paper also implicitly suggests that continued exploration of efficient ACA variants and neighborhood systems could yield further performance gains.

Summary updated

BibTeX
@misc{lee2017efficient,
  author = {Kyungjae Lee and Yuseok Ban and Sangyoun Lee},
  title = {{Efficient Depth Enhancement Using a Combination of Color and Depth Information}},
  year = {2017},
  doi = {10.3390/s17071544},
}
Figure