Abstract
Reconstruction of high-resolution extreme dynamic range images from a small number of low dynamic range (LDR) images is crucial for many computer vision applications. Current high dynamic range (HDR) cameras based on CMOS image sensor technology rely on multi-exposure bracketing which suffers from motion artifacts and signal-to-noise (SNR) dip artifacts in extreme dynamic range scenes. Recently, single-photon cameras (SPCs) have been shown to achieve orders of magnitude higher dynamic range for passive imaging than conventional CMOS sensors. SPCs are becoming increasingly available commercially, even in some consumer devices. Unfortunately, current SPCs suffer from low spatial resolution. To overcome the limitations of CMOS and SPC sensors, we propose a learning-based CMOS-SPC fusion method to recover high-resolution extreme dynamic range images. We compare the performance of our method against various traditional and state-of-the-art baselines using both synthetic and experimental data. Our method outperforms these baselines, both in terms of visual quality and quantitative metrics.
Introduction
Capturing scenes with extreme dynamic range (DR)—where brightness levels vary by over six orders of magnitude—remains a significant challenge for conventional imaging systems. While standard CMOS sensors offer high spatial resolution, they struggle with saturation in bright areas and noise in dark regions. Traditional high dynamic range (HDR) techniques like exposure bracketing introduce motion artifacts and signal-to-noise ratio (SNR) dips. This paper introduces a novel fusion approach that combines high-resolution CMOS images with data from Single-Photon Cameras (SPCs), which offer nearly infinite dynamic range but limited spatial resolution.
The Challenge of Extreme Dynamic Range
Conventional CMOS sensors are limited by their pixel dynamic range. When faced with high-contrast scenes, such as a bright sun against a landscape, these sensors often saturate, losing all detail in the highlights. Conversely, Single-Photon Cameras (SPCs) utilize Single-Photon Avalanche Diodes (SPADs) to count individual photons, providing a linear response over a massive range of intensities. However, current SPC technology is constrained by low spatial resolution.
As shown in the figure above, the CMOS sensor captures high spatial detail but fails to resolve the bright sun due to saturation, whereas the SPC captures the intensity of the sun accurately but at a very low resolution. The histogram illustrates that the SPC's dynamic range extends several orders of magnitude beyond that of a typical CMOS sensor, providing a critical guide for reconstructing extreme DR scenes.
Proposed CMOS-SPC Fusion Method
The researchers developed a deep-learning framework based on a dual-encoder U-Net architecture to fuse these complementary data sources.
- Pre-processing: Data is linearized to account for the SPC's "dead-time" (the recovery period after a photon detection), allowing for accurate photon flux estimation.
- Dual-Encoder Architecture: One encoder extracts high-resolution spatial features from the CMOS image, while a second encoder processes the low-resolution, high-DR SPC data.
- Feature Fusion: A decoder merges these features. Long-skip connections from the CMOS encoder preserve sharp edges, while SPC features guide the reconstruction of saturated regions, preventing the "hallucination" of details common in single-image HDR methods.
- Loss Function: The model is trained using a combination of pixel-wise $\ell_1$ loss and VGG-based perceptual loss in a tone-mapped domain to ensure balanced detail recovery across all brightness levels.
Results and Key Insights
The method was evaluated using a sophisticated simulation pipeline and real-world captures from a prototype hardware setup.
- Superior Reconstruction: The fusion method significantly outperformed baselines like single-image HDR and super-resolution in both quantitative metrics (HDR-VDP3, MSE) and qualitative visual quality.
- Uniform SNR: Unlike exposure bracketing, which suffers from SNR "dips" at transition points between exposures, the CMOS-SPC fusion maintains a smooth, consistent noise profile across the entire dynamic range.
- Guided Detail: By using the SPC as a low-resolution ground truth, the network recovers actual scene details in blown-out areas rather than guessing textures.
Conclusion and Future Work
This research demonstrates that fusing CMOS and SPC data can effectively reconstruct high-resolution, extreme dynamic range images without the artifacts associated with traditional multi-exposure techniques. Future work will focus on developing co-axial hardware to eliminate alignment issues, incorporating color filters for the SPC, and extending the framework to high-DR video applications.
Summary updated