Learning a Deep Convolutional Network for Colorization in Monochrome-Color Dual-Lens System

Xuan Dong, Weixin Li, Xiaojie Wang, Yunhong Wang

Abstract

This paper presents an end-to-end deep learning framework designed to colorize high-quality monochrome images using lower-quality color references from dual-lens systems. By transitioning from traditional "best-pixel matching" to a weighted average approach and incorporating a "color residue" module for occlusion handling, the proposed method achieves state-of-the-art performance, significantly outperforming hand-crafted techniques.

Introduction

Modern smartphone photography often leverages dual-lens systems, typically pairing a monochrome sensor with a color sensor. While monochrome sensors lack color filters—allowing them to capture more light and produce images with superior signal-to-noise ratios (SNR) and finer detail—they lack chromatic information. Conversely, color sensors provide hue but often at lower quality due to the Bayer filter. The goal of this research is to effectively "transfer" color from a low-quality reference to a high-quality monochrome image.

Challenges in Dual-Lens Colorization

Transferring color between disparate sensors is non-trivial due to three primary factors:

  1. Parallax and Disparity: The physical separation of the lenses means images do not perfectly overlap.
  2. Occlusions: Certain regions visible to the monochrome lens may be obscured from the perspective of the color lens.
  3. Matching Ambiguity: Textureless or repetitive regions make it difficult to find exact pixel correspondences.
Figure 4: A high-quality monochrome input image highlighting a specific region for colorization analysis.

Proposed Methodology

The authors propose a two-stage deep convolutional network architecture:

1. Weight Volume Generation

Unlike previous methods that search for a single "best match," this framework calculates a weight volume. A ResNet-based feature extractor pulls deep features from both inputs. An attention mechanism then builds a 4-D feature volume, focusing the network on the most relevant candidate pixels. Finally, 3-D convolutions regulate these weights to ensure spatial consistency. The resulting "rough" colorization is a weighted average of candidates along the disparity line, providing robustness against noise and matching errors.

2. Color Residue Joint Learning

To address errors in occlusion zones, the network employs a color residue module. This component takes the rough colorization and the original monochrome image to predict a "color residue"—the delta between the initial guess and the ground truth. This acts as a deep joint filter, utilizing the sharp edges of the monochrome image to propagate color accurately, even in occluded regions.

Experimental Evaluation

The model was trained on the Scene Flow dataset (~35,000 images) and validated across Cityscapes, Middlebury, and Sintel. To simulate real-world conditions, signal-dependent Gaussian noise was added to mimic the light-efficiency gap between sensors.

Key Results

The proposed method demonstrated significant quantitative and qualitative improvements over existing techniques:

  • Quantitative Performance: On the Cityscapes dataset, the model achieved a PSNR of 44.26 dB, vastly outperforming the next best method (39.33 dB). SSIM scores consistently exceeded 0.98.
  • Qualitative Fidelity: Visual analysis confirms the model's ability to recover color in complex, small-scale regions where traditional methods suffer from "color bleeding."
Figure 3: Qualitative comparison showing the improvement in color fidelity and detail recovery (right) compared to baseline methods (left).

Conclusion

By integrating matching and color propagation into a single differentiable pipeline, this research offers a robust solution for dual-lens colorization. While the computational cost of 3-D convolutions remains a consideration for mobile deployment, the shift toward weighted averaging and residue learning sets a new benchmark for the field. Future work may focus on optimizing these architectures for real-time performance and adapting to non-parallel lens geometries.

Summary updated

Abstract

In the monochrome-color dual-lens system, the gray image captured by the monochrome camera has better quality than the color image from the color camera, but does not have color information. To get high-quality color images, it is desired to colorize the gray image with the color image as reference. Related works usually use hand-crafted methods to search for the best-matching pixel in the reference image for each pixel in the input gray image, and copy the color of the best-matching pixel as the result. We propose a novel deep convolution network to solve the colorization problem in an end-to-end way. Based on our observation that, for each pixel in the input image, there usually exist multiple pixels in the reference image that have the correct colors, our method performs weighted average of colors of the candidate pixels in the reference image to utilize more candidate pixels with correct colors. The weight values between pixels in the input image and the reference image are obtained by learning a weight volume using deep feature representations, where an attention operation is proposed to focus on more useful candidate pixels and a 3-D regulation is performed to learn with context information. In addition, to correct wrongly colorized pixels in occlusion regions, we propose a color residue joint learning module to correct the colorization result with the input gray image as guidance. We evaluate our method on the Scene Flow, Cityscapes, Middlebury, and Sintel datasets. Experimental results show that our method largely outperforms the state-of-the-art methods.
BibTeX
@inproceedings{dong2019learning,
  author = {Xuan Dong and Weixin Li and Xiaojie Wang and Yunhong Wang},
  title = {{Learning a Deep Convolutional Network for Colorization in Monochrome-Color Dual-Lens System}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33018255},
}
Figure