Multiple-Layer Visibility Propagation-Based Synthetic Aperture Imaging through Occlusion

Tao Yang, Jing Li, Jingyi Yu, Yanning Zhang, Wenguang Ma, Xiaomin Tong, Rui Yu, Lingyan Ran

TL;DR

This paper introduces a novel "Multiple-Layer Visibility Propagation" approach for Synthetic Aperture Imaging (SAI) that can generate all-in-focus images even through significant occlusions. By treating the scene as a series of "visibility layers" and propagating visibility information between them, the method overcomes the limitations of traditional SAI that only focuses on a single depth plane.

What Problem Does This Paper Address?

Computer vision applications often struggle with occlusions – situations where objects are hidden behind others. Traditional cameras, capturing a 2D projection from a single viewpoint, are inherently limited in seeing through these occlusions. While Synthetic Aperture Imaging (SAI) using camera arrays offers a way to "see through" by simulating a large virtual lens and focusing on specific depth planes, existing methods have two major drawbacks: they can only produce images focused on a single depth, leaving other objects blurred, and they struggle to accurately reconstruct occluded regions, sometimes resorting to inpainting which can produce visually plausible but incorrect results. This paper aims to solve the problem of generating a single, all-in-focus image that reveals objects hidden behind occluders, ensuring correctness and clarity across all depths.

Methods

The core innovation of this paper lies in its "Multiple-Layer Visibility Propagation" framework. Instead of treating the scene as a collection of depth layers, it segments the scene into multiple visibility layers. Each visibility layer represents a set of rays that are not occluded by objects in previous layers.

The process involves:

  1. Visibility Layer Segmentation: The scene is divided into these visibility layers. For each layer, the algorithm identifies pixels that are visible across all active cameras (i.e., not occluded by objects in preceding layers).
  2. Visibility Labeling: For each layer, a binary energy minimization problem is formulated to determine which pixels are fully visible. This is solved using graph cuts, incorporating a data term based on Maximal Color Difference (MCD) and a smoothness term to encourage neighboring pixels to have the same visibility label.
  3. Depth Estimation: Once the fully visible pixels for a layer are identified, a multiple-label energy minimization problem is solved (again, via graph cuts) to estimate the optimal focus depth for these pixels. This also uses MCD as a key metric, along with a depth smoothness constraint.
  4. Visibility Propagation: Crucially, the visibility information from one layer is used to inform the next. This allows the algorithm to progressively "remove" occluders and reveal what's behind them.
  5. All-in-Focus Image Synthesis: Finally, the information from all the processed visibility layers is combined to construct a single, all-in-focus image.

The use of Maximal Color Difference (MCD) as a focus metric is highlighted as a key advantage, as it is shown to be more robust to occlusion and textureless regions compared to other metrics like variance.

Datasets

The paper evaluates its method on several datasets:

  • Stanford Light Field Dataset: A publicly available dataset used for demonstrating the method's ability to handle occlusions.
  • UCSD/MERL Light Field Dataset: Another publicly available dataset used for evaluating performance on a crowd surveillance scene with occlusions.
  • Three datasets captured by the authors: These were used to demonstrate the method's effectiveness in various scenarios, including complex outdoor scenes and scenes with multiple occluders.

Key Insights and Ideas

  • Visibility Layers vs. Depth Layers: The shift from thinking about depth layers to visibility layers is a fundamental conceptual advancement. This allows the algorithm to directly address occlusion by considering what is visible given the occluders in front, rather than trying to reconstruct depth directly.
  • Propagation is Key: The iterative propagation of visibility information from layer to layer is what enables the "seeing through" capability. Each layer builds upon the knowledge of the previous one, effectively peeling back occlusions.
  • MCD as a Robust Focus Metric: The paper makes a strong case for using Maximal Color Difference (MCD) as a focus metric, particularly in the presence of occlusions and textureless regions. Its stability and ability to find global minima make it superior to variance and other traditional metrics in these challenging scenarios.
  • Graph Cuts for Multi-Layer Optimization: The formulation of both visibility labeling and depth estimation as energy minimization problems solvable by graph cuts is a powerful technique, allowing for efficient and globally optimal solutions within each layer.

Key Results

The paper demonstrates strong performance across several experimental setups:

  • CD Case behind Plants: The proposed method achieved a PSNR of 31.1088, significantly outperforming Vaish et al. (18.1225) and Pei et al. (20.8774). Qualitatively, the results show a much clearer and more complete all-in-focus image compared to state-of-the-art methods that only focus on a single plane.
  • Crowd Surveillance Scene: The method successfully generated clear all-in-focus images, seeing through occlusions between people, whereas Vaish et al.'s method produced blurred results.
  • Complex Outdoor Scene: The approach handled severe occlusions from flowers, producing a depth-free, all-in-focus image of the scene behind, even with complex depth changes.
  • Multiple Occluded Objects: The method effectively removed multiple layers of occluders (playing card, yellow box) to reveal objects behind them, producing clear all-in-focus images of both the first and second occluded objects.

Limitations and Future Work

  • Visibility Requirement: A key limitation noted is that a scene point needs to be visible in at least two camera views for the method to reconstruct it. Otherwise, "black holes" can appear in the all-in-focus image.
  • Future Work: The authors suggest several avenues for future research:
  • Designing more robust cost functions for focus depth estimation.
  • Extending the work to unstructured light field imaging.
  • Adapting the approach for use with handheld mobile phones.

Summary updated

Abstract

Heavy occlusions in cluttered scenes impose significant challenges to many computer vision applications. Recent light field imaging systems provide new see-through capabilities through synthetic aperture imaging (SAI) to overcome the occlusion problem. Existing synthetic aperture imaging methods, however, emulate focusing at a specific depth layer, but are incapable of producing an all-in-focus see-through image. Alternative in-painting algorithms can generate visually-plausible results, but cannot guarantee the correctness of the results. In this paper, we present a novel depth-free all-in-focus SAI technique based on light field visibility analysis. Specifically, we partition the scene into multiple visibility layers to directly deal with layer-wise occlusion and apply an optimization framework to propagate the visibility information between multiple layers. On each layer, visibility and optimal focus depth estimation is formulated as a multiple-label energy minimization problem. The layer-wise energy integrates all of the visibility masks from its previous layers, multi-view intensity consistency and depth smoothness constraint together. We compare our method with state-of-the-art solutions, and extensive experimental results demonstrate the effectiveness and superiority of our approach.
BibTeX
@inproceedings{yang2015multiple,
  author = {Tao Yang and Jing Li and Jingyi Yu and Yanning Zhang and Wenguang Ma and Xiaomin Tong and Rui Yu and Lingyan Ran},
  title = {{Multiple-Layer Visibility Propagation-Based Synthetic Aperture Imaging through Occlusion}},
  booktitle = {Sensors},
  year = {2015},
  url = {https://openreview.net/forum?id=W6iqTu9AbJ},
}
Figure