Scene-adaptive Coded Apertures Imaging

Xuehui Wang, Jinli Suo, Jingyi Yu, Yongdong Zhang, Qionghai Dai

TL;DR

This paper introduces a novel "scene-adaptive" coded aperture imaging technique that dynamically adjusts its aperture pattern based on the dominant edge orientations within a scene. By combining depth-discriminative codes along dominant edges and broadband codes along orthogonal directions, it achieves superior depth estimation and all-focus image recovery compared to existing methods, especially for dynamic scenes.

What Problem Does This Paper Address?

Traditional coded aperture imaging systems aim to recover scene depth and achieve all-focus images from a single, defocused snapshot. However, designing a single coded aperture pattern that excels at both robust deconvolution (preserving high-frequency details) and accurate depth estimation (requiring distinct frequency responses for different depths) presents a fundamental dilemma. Existing methods often compromise on one aspect or require computationally expensive algorithms. Furthermore, these systems typically use a fixed aperture pattern, which is suboptimal for scenes with varying structures and dominant edge orientations. This paper addresses the need for a more flexible and effective coded aperture system that can adapt to scene content, particularly for dynamic scenes where temporal information can be leveraged.

Methods

The core innovation lies in a "scene-adaptive" approach. Instead of a static aperture, the system dynamically generates aperture patterns based on the scene's dominant edge directions.

  • Scene Analysis: The process begins with a sharp, low-light pinhole image to analyze the scene's principal edge orientations. This is crucial for identifying dominant directions.
  • Spatial Partitioning & Temporal Propagation: For each identified dominant direction, the aperture pattern is designed with two components:
  • Depth-Discriminative Code: Applied along the dominant edge direction. This code has multiple zero-crossings in its frequency domain, enabling high depth discrepancy.
  • Broadband Code: Applied along the orthogonal direction. This code has few zero-crossings, ensuring robust deconvolution and preserving image details.
  • Aperture Design: The 2D aperture pattern is constructed as an outer product of two 1D codes, one depth-discriminative and one broadband, tailored to the scene's dominant and orthogonal directions, respectively.
  • Implementation: A Liquid Crystal on Silicon (LCoS) microdisplay is used to rapidly switch between these scene-adaptive aperture patterns across multiple frames.
  • Reconstruction: Algorithms are developed for robust depth map estimation and all-focus image recovery. Temporal redundancy between successive frames is exploited to propagate high-quality results across the entire image, compensating for regions where a specific aperture might be less effective.

Datasets

The paper uses a collection of "exemplary nature scenes" and "Internet images" for demonstrating the cumulative distribution of edge orientations. For experimental evaluation, they use both synthetic toy data (including a scene with four principal directions and a stair-like depth map) and real-world dynamic scenes captured by their prototype system. No new benchmark datasets are introduced.

Key Insights and Ideas

  • Scene Structure is Key: Natural scenes often exhibit a limited number of dominant edge orientations. Exploiting this inherent structure is more effective than using a generic, one-size-fits-all aperture.
  • Conflicting Aperture Goals: The paper explicitly highlights the trade-off between depth discriminative power (requiring zero-crossings) and deconvolution robustness (requiring broadband frequency response). Their solution directly addresses this by spatially partitioning the aperture design.
  • Orthogonal Specialization: Applying depth-discriminative codes along dominant edges (where gradients are weaker) and broadband codes along orthogonal directions (where gradients are stronger) minimizes artifacts and maximizes performance.
  • Temporal Fusion for Completeness: While individual frames might have regions of high quality, temporal propagation is essential to achieve a uniformly high-performance depth map and all-focus image across the entire scene, especially in dynamic scenarios.

Key Results

  • Improved Depth Accuracy & Image Quality: Experiments on both synthetic and real scenes demonstrate that their scene-adaptive coded aperture combines the advantages of state-of-the-art patterns, yielding better quality depth maps and all-focus images.
  • Comparable to State-of-the-Art: The depth accuracy is comparable to high-discrepancy codes [9], and image quality is comparable to broadband codes [17].
  • Effective for Dynamic Scenes: The multi-shot strategy, combined with temporal propagation, allows for effective depth estimation and all-focus recovery even in dynamic scenes.
  • Qualitative Improvements: Figures show significant reduction in ringing artifacts and more accurate depth recovery in challenging regions compared to single-shot, fixed aperture methods.

Limitations and Future Work

  • Scene Structure Dependency: The authors acknowledge that if a scene exhibits no dominant spatial structures (e.g., uniform texture, grass), their current code scheme may not perform well.
  • Future Directions:
    • Exploring scene-adaptive coded patterns in the frequency domain for scenes lacking obvious spatial structures.
  • Investigating color coded apertures that encode different channels based on direction.
  • Developing combined motion- and scene-adaptive aperture designs.
  • Improving the portability of their prototype system.

Summary updated

Abstract

Coded aperture imaging systems have recently shown great success in recovering scene depth and extending the depth-of-field. The ideal pattern, however, would have to serve two conflicting purposes: 1) be broadband to ensure robust deconvolution and 2) has sufficient zero-crossings for a high depth discrepancy. This paper presents a simple but effective scene-adaptive coded aperture solution to bridge this gap. We observe that the geometric structures in a natural scene often exhibit only a few edge directions, and the successive frames are closely correlated. Therefore we adopt a spatial partitioning and temporal propagation scheme. In each frame, we address one principal direction by applying depth-discriminative codes along it and broadband codes along its orthogonal direction. Since within a frame only the regions with edge direction corresponding to its aperture code behaves well, we utilize the close among-frame correlation to propagate the high quality single frame results temporally to obtain high performance over the whole image lattice. To physically implement this scheme, we use a Liquid Crystal on Silicon (LCoS) microdisplay that permits fast changing pattern codes. Firstly, we capture the scene with a pinhole and analyze the scene content to determine primary edge orientations. Secondly, we sequentially apply the proposed coding scheme with these orientations in the following frames. Experiments on both synthetic and real scenes show that our technique is able to combine advantages of the state-of-the-art patterns for recovering better quality depth map and all-focus images.
BibTeX
@misc{wang2015scene,
  author = {Xuehui Wang and Jinli Suo and Jingyi Yu and Yongdong Zhang and Qionghai Dai},
  title = {{Scene-adaptive Coded Apertures Imaging}},
  year = {2015},
  eprint = {1506.05942},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1506.05942},
}

Loading related content...

Figure