TL;DR
This paper introduces a novel, lightweight approach for estimating depth from dual-pixel (DP) images. By explicitly modeling the physics and inherent errors of DP sensors, their method achieves state-of-the-art performance with significantly fewer parameters than existing deep learning models, making it ideal for resource-constrained applications like mobile cameras.
What Problem Does This Paper Address?
Estimating depth from images is crucial for many computer vision tasks, from 3D reconstruction to autonomous navigation. Dual-pixel (DP) sensors, found in many mobile cameras, offer a unique way to capture depth information by splitting each pixel into two photodiodes. This allows for depth estimation without active illumination, leveraging the defocus blur inherent in the two slightly different views.
However, existing deep learning methods for DP depth estimation, while powerful, often suffer from several drawbacks:
- High Parameter Count: They tend to be "heavy," requiring many parameters, which is problematic for mobile devices with limited computational resources.
- Underutilization of Constraints: They don't fully exploit the physical properties and inherent disparity constraints of DP sensors, leading to suboptimal performance.
- Black Box Nature: The internal workings of deep networks can be opaque, making it hard to diagnose and fix errors.
- Data Scarcity: Acquiring large, labeled DP datasets with ground truth depth is labor-intensive.
This paper aims to overcome these limitations by developing a lightweight, physics-informed method that explicitly models DP sensor characteristics and errors, leading to more accurate and efficient depth estimation.
Methods
The authors propose a two-pronged approach:
- Physics-Informed Disparity Completion:
- Modeling DP Errors: They first model the specific errors that arise during template matching (TM) on DP images. This involves simulating DP image formation and using symbolic regression to derive parametric functions that capture the relationship between depth, focus distance, and disparity error. This allows them to generate "pseudo-ground truth" disparity that reflects real-world DP sensor behavior.
- Disparity Sampling for Training: During training, they use this modeled error to sample disparities. This "physics-informed" sampling helps the network learn to be robust to these specific DP errors, improving its ability to complete sparse disparity maps.
- Learning from RGBD Data: To address the lack of large DP datasets, they convert standard monocular RGBD datasets (like NYUDv2) into pseudo-ground truth DP disparity. This enables large-scale training in a domain that closely mimics DP data.
- Non-Learning-Based Refinement Framework:
- Addressing Disparity Expansion: A key issue in DP depth estimation is the inherent expansion of disparity regions, especially near edges, due to defocus blur. This can lead to inaccurate depth estimates.
- Fast Global Smoother (FGS): Instead of relying on complex, parameter-heavy neural networks for refinement, they employ a non-learning-based approach using the Fast Global Smoother (FGS). FGS is an efficient algorithm that performs global optimization.
- Confidence Map Integration: They integrate the confidence map produced by their completion network into the FGS. This allows the refinement process to focus on areas where the network is less certain, effectively correcting disparity expansion errors without adding significant computational overhead. This refinement framework is designed to be generally applicable to other disparity estimation models.
Datasets
- Training: The authors leverage widely available monocular RGBD datasets, specifically mentioning NYUDv2, RGB-D-D, and their own CG data. These datasets are converted to "pseudo-ground truth" DP disparity for training.
- Evaluation: The primary evaluation dataset used is the 100-pair DP dataset from Punnappurath et al. [40].
Key Insights and Ideas
- Physics-Informed Learning is Key: Explicitly modeling the physical properties and error characteristics of DP sensors, rather than treating them as a black box, leads to significant performance gains and a more lightweight model.
- Simulating Errors for Robustness: Generating training data that incorporates realistic DP sensor errors (like TM errors) is crucial for building robust disparity completion networks.
- Non-Learning Refinement for Efficiency: Complex refinement tasks can be handled effectively by efficient, non-learning-based methods like FGS, especially when guided by a network's confidence map, avoiding the parameter bloat of deep refinement networks.
- Bridging the Domain Gap: Converting existing RGBD datasets to pseudo-DP disparity is an effective strategy to overcome the scarcity of real DP datasets for training.
Key Results
- State-of-the-Art Performance: The proposed method achieves state-of-the-art results on the evaluation dataset, outperforming existing DP estimation methods across key metrics like AI(1), AI(2), and 1-|ps|.
- Lightweight Model: The system has significantly fewer parameters (1.9M) compared to conventional methods (e.g., Kim et al. [24] with 10.6M), achieving this with less than 1/5 the parameters of some competitors.
- Improved Accuracy and Sharpness: Qualitative results show that their method avoids artifacts like false disparities in uniform regions (e.g., walls) and produces sharper boundaries compared to other methods.
- Effective Refinement: The non-learning refinement framework successfully corrects disparity expansion errors, leading to more accurate results, especially at edges.
Limitations and Future Work
- Dataset Specificity: The evaluation was performed on a specific dataset. A more exhaustive evaluation across different settings (lens aperture, focal length) is needed.
- Assumed Optical Parameters: The method assumes and assigns various optical parameters. While qualitatively adequate on their own dataset, a more thorough validation of these assumptions is suggested.
- Future Applications: The authors envision their method being valuable for future applications with an increasing number of cameras equipped with DP sensors. They also suggest further research into more exhaustive evaluations.
Summary updated