Abstract
Demosaicing is a fundamental process in digital photography, reconstructing full-color images from incomplete sensor data. Traditional deep learning approaches often fail due to spatial inconsistency in convolutional filters and a reliance on synthetic training data. This paper presents the Spatial Adaptive Network (SANet), which utilizes a weight-generation function to dynamically adjust convolution kernels based on the local Color Filter Array (CFA) pattern. Coupled with a new high-quality dataset of paired real-world RAW and RGB images captured via pixel-shift technology, SANet significantly outperforms existing methods in clarity and noise reduction.
Introduction
Most digital cameras employ a single sensor overlaid with a Color Filter Array (CFA), typically the Bayer pattern, where each pixel records only one of the three primary colors (Red, Green, or Blue). Demosaicing is the computational task of interpolating the missing color values. Current deep learning models for demosaicing face two primary challenges:
- Spatial Inconsistency: Standard CNNs apply uniform filters across the image, ignoring the fact that interpolation logic varies depending on whether the target pixel is at a Red, Green, or Blue location in the Bayer grid.
- The Reality Gap: Models trained on synthetic noise often perform poorly on real sensor data, which exhibits complex, signal-dependent noise characteristics.
Methodology: Spatial Adaptive Convolution
The core innovation of SANet is the Spatial Adaptive Convolution layer. Unlike static CNNs, SANet employs a weight-generation function, $\phi$, that adapts to the local mosaic pattern.
Pattern-Aware Weights
By incorporating the Bayer grid position into the network's logic, SANet generates specific convolution kernels for different spatial locations. This allows the model to learn specialized interpolation functions for the distinct R, G, and B sub-grids, effectively acting as a collection of experts tailored to specific CFA phases.
Kernel Decomposition and Efficiency
To maintain computational efficiency, the authors decompose the large adaptive kernels into two components: one focusing on spatial correlation ($G$) and another on inter-channel relationships ($F$). This decomposition reduces memory requirements significantly, allowing SANet to achieve state-of-the-art results with only 10.78M parameters and 19.18 GFLOPs—making it faster and lighter than many competing architectures.
The Paired Real Image Demosaicing Dataset
To bridge the gap between synthetic and real-world performance, the authors introduced a new dataset. Using a Sony A7R4 with pixel-shift technology, they captured "true" ground truth images by physically moving the sensor to record full color at every pixel. The dataset includes 100 indoor and outdoor scenes, featuring both long-exposure (clean) and short-exposure (noisy) pairs to rigorously test robustness against real sensor noise.
Experimental Results
SANet was evaluated against traditional algorithms and modern deep learning models.
- Noiseless Scenarios: SANet achieved a Peak Signal-to-Noise Ratio (PSNR) of 52.01 dB, surpassing the previous best model (SGNet) by 0.24 dB.
- Noisy Scenarios: In challenging noisy conditions, SANet maintained its lead with a PSNR of 42.58 dB.
- Qualitative Analysis: Visually, SANet effectively eliminates common artifacts such as "zippering" (jagged edges), checkerboard patterns, and moiré fringes, particularly in high-frequency image regions.
Conclusion
SANet represents a significant step forward in practical demosaicing by acknowledging the spatially variant nature of the problem. By combining pattern-aware dynamic convolutions with high-quality real-world training data, the model provides a robust solution for modern camera pipelines. Future work will explore adapting these weights to image content and extending the architecture to non-standard CFA patterns.
Summary updated