Estimating Depth-Salient Edges and Its Application to Stereoscopic Image Quality Assessment

Sameeulla Khan Md, Sumohana S. ChannappayyaIEEE TIP 2018 · OpenReview

TL;DR

This paper introduces a novel method for identifying "depth-salient edges" in stereoscopic 3D (S3D) images, which are crucial for human depth perception. They leverage these edges to develop a more accurate full-reference S3D image quality assessment (FRSIQA) metric that considers both luminance and depth quality, achieving state-of-the-art results on several benchmarks.

What Problem Does This Paper Address?

Assessing the quality of stereoscopic 3D (S3D) images is challenging. While 2D image quality assessment (IQA) is well-studied, S3D introduces the added dimension of depth. Human visual perception of S3D images is influenced not only by the visual content of each individual image (luminance) but also by the perceived depth. Existing S3D IQA methods often struggle to effectively incorporate this depth information, leading to suboptimal quality predictions. This paper aims to bridge this gap by proposing a method to specifically identify the image features that contribute most to depth perception and using this information to enhance S3D quality assessment.

Methods

The core innovation of this paper is the Salient Edges with respect to Depth perception (SED) algorithm. The authors hypothesize that only a subset of image edges are critical for depth perception. SED aims to localize these specific edges.

Here's a breakdown of their approach:

  • Depth Saliency Estimation (SED):
    • They start by analyzing the disparity map of the S3D image.
  • They quantify occlusions, which occur at depth discontinuities and are known to influence depth perception. This is done by comparing image patches across stereo pairs.
  • They also analyze the relative positional difference of edge locations between the left and right images, as parallax creates these differences and aids depth perception.
  • By combining information from occlusions and positional differences at edge locations, they derive the SED map, which highlights edges that are likely to contribute to depth perception.
  • A Log-enhanced SED (LeSED) map is also computed for better visualization and perceptual comparison.
  • Stereoscopic Image Quality Assessment (FRSIQA):
    • The proposed FRSIQA metric combines two main components: luminance quality and depth quality.
  • Luminance quality is estimated by comparing saliency maps and gradient magnitude (GM) maps of the reference and test stereo pairs.
  • Depth quality is specifically assessed by evaluating the luminance quality maps at the edge locations identified by the SED maps. This focuses the quality assessment on the perceptually relevant depth features.
  • Finally, the luminance and depth quality scores are combined using a weighted geometric approach to produce the overall S3D image quality score.

Datasets

The authors evaluated their proposed FRSIQA algorithm on seven publicly available S3D IQA databases:

  • IRCCYN
  • MICT
  • LIVE Phase-I
  • LIVE Phase-II
  • MCL
  • WIVC-I
  • WIVC-II

They also used the Middlebury stereo database for demonstrating the SED and LeSED maps.

Key Insights and Ideas

  • Depth is not uniform across all edges: The paper's central hypothesis is that only a specific subset of image edges significantly contributes to depth perception in S3D images. This is a crucial insight that moves beyond simply considering all edges or global disparity.
  • Occlusions and parallax are key depth cues: The authors identify occlusions (areas hidden in one view but visible in another) and parallax-induced positional differences at edges as primary drivers of depth perception.
  • SED as a targeted feature for depth quality: By focusing quality assessment on these depth-salient edges, the SED maps allow for a more perceptually relevant evaluation of depth quality, rather than a general assessment of the entire image.
  • Combining luminance and depth quality is essential: The paper demonstrates that a comprehensive S3D IQA metric must integrate both luminance and depth quality assessments for optimal performance.

Key Results

  • The proposed SED method effectively localizes depth-salient edges, showing better correlation with ground truth disparity edges than estimated disparity maps.
  • The FRSIQA metric incorporating SED achieved competitive performance across all seven tested S3D IQA databases.
  • The proposed metric achieved state-of-the-art performance on three of the seven databases, including full HD resolution databases, highlighting its effectiveness.
  • The paper provides extensive quantitative results (PLCC and SROCC scores) in several tables, demonstrating the superiority of their combined approach over methods that only consider luminance or average left/right scores.

Limitations and Future Work

  • Computational Cost: While the SED maps are computed once per stereopair, the initial computation of disparity maps and SED maps can be computationally intensive. However, the paper notes that once these are available, the FRSIQA computation is very fast (1 second per test stereopair).
  • Future Work: The authors suggest that the SED maps have potential applications beyond S3D IQA, including image compression and full-reference stereoscopic video quality assessment (FRSVQA). They plan to explore FRSVQA as a future research direction.

Summary updated

Abstract

The human visual system pays attention to salient regions while perceiving an image. When viewing a stereoscopic 3-D (S3D) image, we hypothesize that while most of the contribution to saliency is provided by the 2-D image, a small but significant contribution is provided by the depth component. Further, we claim that only a subset of image edges contribute to depth perception while viewing an S3D image. In this paper, we propose a systematic approach for depth saliency estimation, called salient edges with respect to depth perception (SED) which localizes the depth-salient edges in an S3D image. We demonstrate the utility of SED in full reference stereoscopic image quality assessment. We consider gradient magnitude and inter-gradient maps for predicting structural similarity. A coarse quality map is estimated first by comparing the 2-D saliency and gradient maps of reference and test stereo pairs. We average this quality map to estimate luminance quality and refine this quality map using SED maps for evaluating depth quality. Finally, we combine this luminance and depth quality to obtain an overall stereo image quality. We perform a comprehensive evaluation of our metric on seven publicly available S3D IQA databases. The proposed metric shows competitive performance on all seven databases with state-of-the-art performance on three of them.
BibTeX
@article{md2018estimating,
  author = {Sameeulla Khan Md and Sumohana S. Channappayya},
  title = {{Estimating Depth-Salient Edges and Its Application to Stereoscopic Image Quality Assessment}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=60fRzxF-M-},
}
Figure