Abstract
This paper presents an end-to-end deep learning framework designed to colorize high-quality monochrome images using lower-quality color references from dual-lens systems. By transitioning from traditional "best-pixel matching" to a weighted average approach and incorporating a "color residue" module for occlusion handling, the proposed method achieves state-of-the-art performance, significantly outperforming hand-crafted techniques.
Introduction
Modern smartphone photography often leverages dual-lens systems, typically pairing a monochrome sensor with a color sensor. While monochrome sensors lack color filters—allowing them to capture more light and produce images with superior signal-to-noise ratios (SNR) and finer detail—they lack chromatic information. Conversely, color sensors provide hue but often at lower quality due to the Bayer filter. The goal of this research is to effectively "transfer" color from a low-quality reference to a high-quality monochrome image.
Challenges in Dual-Lens Colorization
Transferring color between disparate sensors is non-trivial due to three primary factors:
- Parallax and Disparity: The physical separation of the lenses means images do not perfectly overlap.
- Occlusions: Certain regions visible to the monochrome lens may be obscured from the perspective of the color lens.
- Matching Ambiguity: Textureless or repetitive regions make it difficult to find exact pixel correspondences.
Proposed Methodology
The authors propose a two-stage deep convolutional network architecture:
1. Weight Volume Generation
Unlike previous methods that search for a single "best match," this framework calculates a weight volume. A ResNet-based feature extractor pulls deep features from both inputs. An attention mechanism then builds a 4-D feature volume, focusing the network on the most relevant candidate pixels. Finally, 3-D convolutions regulate these weights to ensure spatial consistency. The resulting "rough" colorization is a weighted average of candidates along the disparity line, providing robustness against noise and matching errors.
2. Color Residue Joint Learning
To address errors in occlusion zones, the network employs a color residue module. This component takes the rough colorization and the original monochrome image to predict a "color residue"—the delta between the initial guess and the ground truth. This acts as a deep joint filter, utilizing the sharp edges of the monochrome image to propagate color accurately, even in occluded regions.
Experimental Evaluation
The model was trained on the Scene Flow dataset (~35,000 images) and validated across Cityscapes, Middlebury, and Sintel. To simulate real-world conditions, signal-dependent Gaussian noise was added to mimic the light-efficiency gap between sensors.
Key Results
The proposed method demonstrated significant quantitative and qualitative improvements over existing techniques:
- Quantitative Performance: On the Cityscapes dataset, the model achieved a PSNR of 44.26 dB, vastly outperforming the next best method (39.33 dB). SSIM scores consistently exceeded 0.98.
- Qualitative Fidelity: Visual analysis confirms the model's ability to recover color in complex, small-scale regions where traditional methods suffer from "color bleeding."
Conclusion
By integrating matching and color propagation into a single differentiable pipeline, this research offers a robust solution for dual-lens colorization. While the computational cost of 3-D convolutions remains a consideration for mobile deployment, the shift toward weighted averaging and residue learning sets a new benchmark for the field. Future work may focus on optimizing these architectures for real-time performance and adapting to non-parallel lens geometries.
Summary updated