Abstract
Lensless imaging is a new, emerging modality where image sensors utilize optical elements in front of the sensor to perform multiplexed imaging. There have been several recent papers to reconstruct images from lensless imagers, including methods that utilize deep learning for state-of-the-art performance. However, many of these methods require explicit knowledge of the optical element, such as the point spread function, or learn the reconstruction mapping for a single fixed PSF. In this paper, we explore a neural network architecture that performs joint image reconstruction and PSF estimation to robustly recover images captured with multiple PSFs from different cameras. Using adversarial learning, this approach achieves improved reconstruction results that do not require explicit knowledge of the PSF at test-time and shows an added improvement in the reconstruction model’s ability to generalize to variations in the camera’s PSF. This allows lensless cameras to be utilized in a wider range of applications that require multiple cameras without the need to explicitly train a separate model for each new camera.
Introduction
Lensless cameras represent a significant shift in computational imaging, offering ultra-thin form factors, reduced costs, and enhanced privacy by capturing multiplexed light rather than traditional images. However, recovering the original scene from this "optical blur" typically requires precise knowledge of the system's Point Spread Function (PSF). This PSF acts as an optical fingerprint, unique to the specific mask or diffuser used.
The primary challenge in current lensless imaging is the sensitivity of reconstruction algorithms to PSF variations. Traditional methods require tedious per-camera calibration. If the optical element shifts or a different one is used, reconstruction quality degrades significantly. This work addresses the "calibration-free" challenge by proposing a framework that reconstructs images without prior knowledge of the PSF at test time.
Methodology
The proposed architecture performs joint image reconstruction and PSF estimation using a dual-generator framework.
Dual-Generator Architecture
The system employs two U-Net-style generators that process the raw lensless capture simultaneously:
- Image Reconstruction Generator: Produces an estimate of the original scene ($\hat{I}$).
- PSF Generator: Estimates the PSF ($\hat{PSF}$) responsible for the capture.
Loss Functions and Constraints
To ensure high-quality and physically consistent results, the model incorporates several key innovations:
- Forward Model Consistency Loss: The estimated image and PSF are convolved ($PSF * \hat{I}$) and compared against the original raw input. This self-supervised constraint enforces physical plausibility.
- Adversarial Learning: Utilizing a Wasserstein GAN (WGAN) framework, discriminators for both the image and the PSF push the generators toward sharper, more realistic outputs.
- Perceptual Loss: A VGG-16 based perceptual loss is used to prioritize structural features and human-perceived image quality over simple pixel-wise accuracy.
Experimental Setup
The model was evaluated using three distinct datasets:
- DiffuserCam Dataset: A benchmark of 25,000 images from a real diffuser-based camera.
- Simulated Multi-PSF Dataset: 20 unique PSFs were generated to test the model's ability to generalize across different optical conditions.
- Real Hardware Prototype: A custom lensless camera was built using a Panasonic Lumix G85 and a PET plastic diffuser, capturing over 8,000 training frames.
Results and Discussion
The proposed method demonstrates significant improvements in both robustness and efficiency compared to traditional optimization and fixed-PSF deep learning models.
Performance and Speed
On the DiffuserCam benchmark, the model achieved a PSNR of 20.56 dB, matching or exceeding state-of-the-art methods like Le-ADMM-U, even without test-time PSF knowledge. Crucially, inference takes only 0.32 seconds per frame—a 5x speedup over unrolled networks and orders of magnitude faster than traditional ADMM optimization.
Qualitative Comparison
Visual results indicate that the joint estimation approach successfully recovers scene structure and color across various subjects, from floral scenes to complex patterns.
Robustness to Noise and PSF Variation
A key finding was the importance of noise augmentation during training. Lensless reconstruction is highly sensitive to sensor noise; incorporating Gaussian noise into the training pipeline was essential for real-world hardware performance. Furthermore, the model successfully generalized to three different real-world PSFs using a single trained network, eliminating the need for individual camera calibrations.
Conclusion
This work demonstrates that blind deconvolution in lensless imaging is achievable through deep adversarial learning. By jointly estimating the PSF and the scene, the proposed framework offers a robust, high-speed solution that generalizes across different hardware configurations. Future work will focus on improving high-frequency detail recovery and exploring unsupervised training methods to further reduce the reliance on paired ground-truth data.
Summary updated