TL;DR
This paper introduces a novel RAW-to-RAW image restoration pipeline for under-display cameras (UDCs). By decoupling the UDC image restoration from the complex, proprietary Image Signal Processing (ISP) pipelines found in smartphones, their method can be seamlessly integrated with existing hardware, producing high-quality images that match regular cameras.
What Problem Does This Paper Address?
The quest for larger, more immersive smartphone displays has led to the development of under-display cameras (UDCs). However, placing a camera behind a display introduces significant image quality issues: reduced contrast, blur, color shifts, scattering artifacts, and increased noise. These distortions are challenging to correct, and existing methods often struggle to integrate with the highly customized Image Signal Processing (ISP) pipelines that manufacturers invest heavily in. The authors aim to solve this by creating an "ISP-agnostic" restoration method that can work with any existing ISP, preserving the desired brand-specific look of the final image.
Methods
The core of the proposed solution is a RAW-to-RAW image restoration pipeline. This means the neural network takes the raw, distorted image from the UDC sensor and outputs a restored raw image. This raw output can then be fed into any standard smartphone ISP for further processing (like demosaicing, white balancing, etc.) to produce the final image. This approach avoids the complexity of trying to learn or emulate the proprietary ISP itself.
Key methodological contributions include:
- ISP-Agnostic RAW-to-RAW Pipeline: This is the central innovation, allowing for broad compatibility with existing smartphone hardware.
- Deep Learning Architecture: A Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP) is employed for its ability to preserve texture detail and handle image-to-image translation tasks effectively.
- HDR Monitor-Based Data Capture: A novel setup using an HDR monitor is developed to generate large quantities of diverse training data with high contrast and scene variety, which is crucial for capturing UDC artifacts accurately.
- Data Augmentation: Techniques like resizing and brightness augmentation are used to improve robustness.
- Combined Loss Function: A combination of SSIM and perceptual loss is used to balance detail preservation and perceptual quality.
Datasets
The authors created their own datasets for training and evaluation:
- Monitor Dataset: Captured using an Eizo CG3145 HDR monitor. This dataset provides a large diversity of scenes and high contrast, crucial for training the UDC restoration model.
- HDR Portrait Dataset: Synthetically generated by compositing SDR portraits with HDR backgrounds, aiming to mimic common selfie scenarios.
- Real-World Dataset: Captured using actual UDC and reference cameras. This dataset, though smaller and with less scene diversity, is used for fine-tuning the model to achieve full camera resolution and capture real-world nuances.
Key Insights and Ideas
- Decoupling is Key: The most significant insight is that trying to learn the ISP and the UDC restoration simultaneously is overly complex and data-intensive. By focusing solely on RAW-to-RAW restoration, the problem becomes more tractable and the solution more generalizable.
- HDR is Crucial for Artifacts: UDC artifacts like diffraction are most pronounced in high-contrast scenes. Using an HDR monitor is essential for generating training data that accurately reflects these challenging conditions.
- RAW-to-RAW for ISP Compatibility: The RAW-to-RAW approach is a clever way to ensure that the restored image can be processed by any existing ISP, preserving the manufacturer's desired aesthetic without needing to retrain or redesign the ISP itself.
- Monitor Data for Diversity, Real Data for Detail: The strategy of using monitor data for broad scene diversity and real-world data for fine-tuning resolution and specific details is a practical approach to dataset creation.
Key Results
- Quantitative Improvements: The proposed method achieves significant improvements in PSNR and SSIM metrics compared to baseline methods, particularly on real-world data after fine-tuning. For example, on the monitor dataset, the restored image achieves a higher PSNR than the raw restored image after ISP processing.
- Qualitative Improvements: The restored images exhibit significantly reduced blur, improved contrast, and corrected color shifts, closely matching the ground truth images. The method effectively removes diffraction artifacts, such as the "stripe-like" patterns seen in the degraded images.
- ISP Compatibility: The restored images, when processed by a standard smartphone ISP, produce natural-looking results that are visually indistinguishable from images captured by a regular camera.
Limitations and Future Work
- Resolution Limitation of Monitor Data: The monitor dataset's resolution is limited by the display itself, necessitating the use of real-world data for fine-tuning to achieve full camera resolution.
- Alignment Challenges in Real-World Data: Automatic pixel-to-pixel alignment in the real-world dataset can be challenging in difficult lighting or high-depth complexity scenes, potentially affecting evaluation metrics.
- Future Work: The authors suggest that their RAW-to-RAW paradigm could be beneficial for other image restoration tasks on consumer devices. They also mention that their method could be combined with display design optimizations for UDCs.
Summary updated