Abstract
Compared to color images captured by conventional RGB cameras, monochrome images usually have better signal-to-noise ratio (SNR) and richer textures due to their higher quantum efficiency. It is thus natural to apply a mono-color dual-camera system to restore color images with higher visual quality. In this paper, we propose a mono-color image enhancement algorithm that colorizes the monochrome image with the color one. Based on the assumption that adjacent structures with similar luminance values are likely to have similar colors, we first perform dense scribbling to assign colors to the monochrome pixels through block matching. Two types of outliers, including occlusion and color ambiguity, are detected and removed from the initial scribbles. We also introduce a sampling strategy to accelerate the scribbling process. Then, the dense scribbles are propagated to the entire image. To alleviate incorrect color propagation in regions that have no color hints at all, we generate extra color seeds based on the existing scribbles to guide the propagation process. Experimental results show that our algorithm can efficiently restore color images with higher SNR and richer details from mono-color image pairs, and achieves good performance in solving the color bleeding problem.
Introduction
In modern smartphone photography, the dual-camera setup—combining a monochrome sensor with an RGB sensor—is a standard approach for capturing high-detail, low-noise images. However, merging these two images is technically challenging due to the physical separation of the sensors. This separation creates "disparity," or different viewing angles, which makes perfect alignment difficult. Traditional registration-based methods often result in "color bleeding," where color from one object smears onto another due to tiny alignment errors.
This paper presents a novel alignment-independent algorithm that treats the color image as a source of "automatic scribbles" to colorize the sharp monochrome frame. By shifting the focus from strict pixel-to-pixel alignment to a robust colorization framework, the authors overcome the limitations of traditional stereo matching.
Methodology
The proposed pipeline moves away from strict registration and instead adopts a Dense Scribbling approach. The process consists of four main stages:
- Dense Scribbling via Block Matching: For every patch in the monochrome image, the algorithm searches for the most similar luminance patch in the color image within a search window, "borrowing" the color from that match.
- Probabilistic Patch Sampling: To optimize performance, the authors introduced a sampling strategy. They found that sampling only 1/64 of the patches maintains a 96% confidence level in finding a correct color estimate, drastically reducing computation time.
- Outlier Removal: The system identifies and removes "bad" matches caused by occlusions (areas visible to only one lens) and color ambiguity (where identical luminance values represent different colors).
- Seed Generation & Propagation: In unreliable regions, the algorithm generates "extra seeds" based on neighboring reliable colors. These are then propagated across the image using an optimization-based method that respects the sharp edges of the monochrome source.
Key Insights
- Robustness over Alignment: The paper demonstrates that for high-quality colorization, a perfect disparity map is unnecessary. Treating color as a set of "hints" or "scribbles" is far more robust to the noise and inconsistencies found in real-world dual-camera systems.
- Efficiency via Sampling: The probabilistic patch sampling model reduced the scribbling time from 184 seconds to just 5.9 seconds per image with negligible loss in quality, making the approach practical for consumer devices.
- Handling Ambiguity: By explicitly tackling "color ambiguity," the method detects when a pixel has multiple conflicting color candidates and filters them out before they can cause visual artifacts.
Experimental Results
The authors evaluated their method on standard benchmarks like Middlebury and SceneFlow, as well as real-world captures.
- Quantitative Performance: On the Middlebury dataset, the proposed method achieved a PSNR of 41.50 dB, significantly outperforming other state-of-the-art methods which typically reach around 39.00 dB.
- Visual Quality: Qualitatively, the results show much sharper edges and almost zero color bleeding compared to registration-based methods, which often struggle with "halos" around objects.
- Computational Speed: The total processing time is approximately 10.1 seconds per image in MATLAB, making it significantly faster than dense matching alternatives.
Conclusion
While the algorithm is highly robust, it still relies on the assumption that similar luminance values likely share similar colors. In cases of massive occlusions, the algorithm must rely on neighboring seeds, which can occasionally lead to inaccuracies. Future work involves implementing the system in C++ with parallel computing to achieve real-time processing on mobile devices and potentially integrating deep learning to further improve seed generation in complex occluded regions.
Summary updated