TL;DR
This paper introduces HEDIP, a novel deep learning model for predicting the quality of synthesized views before the computationally expensive rendering process. By fusing texture and depth images and employing a two-channel network to capture both spatial and gradient information, HEDIP effectively predicts visual quality, outperforming existing methods and enabling more flexible view synthesis systems.
What Problem Does This Paper Address?
Virtual view synthesis, a technology powering multi-view video and free-view television, relies on algorithms like Depth-Image-Based Rendering (DIBR) to generate new viewpoints from existing texture and depth data. However, the DIBR process itself is computationally intensive. Furthermore, distortions can occur during the acquisition, compression, or transmission of the initial texture and depth images, leading to artifacts in the synthesized views. Existing methods for evaluating the quality of these synthesized views often require the DIBR process to be completed first, or they rely on hand-crafted features that struggle to capture the complex, heterogeneous distortions (both traditional and geometric) introduced by DIBR. This paper addresses the need for a "blind" quality prediction method that can assess the quality of synthesized views without performing the DIBR process and without requiring a reference image, using only the original texture and depth inputs.
Methods
The proposed HEDIP model tackles this problem through several key innovations:
- Pre-DIBR Quality Prediction: Instead of evaluating the synthesized image, HEDIP predicts its quality using only the input texture and depth images.
- Image Fusion via DCT: To simulate the distortions present in DIBR-synthesized images, the texture and depth images are fused using Discrete Cosine Transform (DCT). This fusion preserves low-frequency information (brightness) from the texture image and averages high-frequency information from both, effectively mimicking some DIBR-induced distortions.
- Two-Channel Convolutional Neural Network (TCCNN): A TCCNN is employed to extract features from two domains:
- Spatial Domain Channel: Captures standard image features.
- Gradient Domain Channel: Uses the Sobel operator to extract edge information, which is crucial for detecting geometric distortions common in synthesized views.
- Contextual Multi-Level Feature Fusion (CMLFF): This module fuses shallow detail features with deep semantic features extracted by the TCCNN to better represent geometric distortions.
- Heterogeneous Distortion Perception (HDP) Module: A critical component designed to address the challenge of local distortions within image patches. Standard training labels derived from the entire source image are insufficient. The HDP module uses the BRISQUE metric to evaluate image patches and the source image, generating effective training labels that account for local quality variations. This allows the model to learn from patches with varying degrees of distortion.
Datasets
The paper evaluates HEDIP on two publicly available datasets:
- MCL-3D [36]: Contains 684 synthesized image pairs with Mean Opinion Scores (MOS). These pairs are generated using the View Synthesis Reference Software (VSRS) and include various distortions (Gaussian blur, JPEG compression, etc.) applied to texture, depth, or both.
- IST [37]: Contains 180 synthesized image pairs with MOS values. These are synthesized using both the VSIM [38] and VSRS algorithms, with texture and depth images often suffering from compression artifacts.
Key Insights and Ideas
- Predicting Quality Before Rendering is Feasible and Valuable: The core idea that quality can be predicted from pre-synthesis inputs is a significant shift, offering computational savings and system flexibility.
- Heterogeneous Distortions Require Specialized Handling: The paper highlights that synthesized views suffer from a mix of traditional and geometric distortions, necessitating a model that can capture both. The HDP module is a clever solution to this by creating patch-level labels that reflect local quality.
- Gradient Information is Key for Geometric Distortions: The TCCNN's gradient channel proves vital for detecting the geometric artifacts introduced by DIBR, which are often missed by standard image quality metrics.
- Deep Learning Outperforms Traditional Methods: The consistent superior performance of HEDIP (a deep learning model) over traditional methods, even those specifically designed for view synthesis, underscores the power of learned features.
Key Results
- State-of-the-Art Performance: HEDIP achieves state-of-the-art results on both the MCL-3D and IST datasets, outperforming existing "Post-DIBR" (evaluating rendered images) and "Pre-DIBR" (predicting from inputs) methods in terms of accuracy (PLCC) and monotonicity (SRCC).
- MCL-3D: HEDIP achieved the best PLCC (0.934) and SRCC (0.927).
- VSIM-based IST: HEDIP achieved the best PLCC (0.866) and a strong SRCC (0.787).
- VSRS-based IST: HEDIP achieved the best SRCC (0.767) and the second-best PLCC (0.750).
- Robustness to Distortions: Radar plots demonstrate that HEDIP maintains high accuracy and monotonicity across various distortion types (e.g., additive white noise, Gaussian blur, JPEG compression).
- Ablation Study: The ablation study confirms the importance of both the CMLFF and HDP modules, with the full HEDIP model (TCCNN + CMLFF + HDP) significantly outperforming the base TCCNN and TCCNN + CMLFF models.
Limitations and Future Work
- Backbone Network Improvement: The authors suggest that further improvements to the TCCNN backbone could potentially enhance overall performance.
- Extension to Video: The current work focuses on still images. Extending the model to evaluate the quality of synthesized videos is identified as a promising but challenging future direction.
- 3D Quality Evaluation: Moving from 2D quality evaluation to 3D quality evaluation is also proposed as a future research avenue.
Summary updated