TL;DR
This paper tackles the challenge of low-light image restoration (LLIR) by proposing a novel semi-supervised learning (SSL) approach. They introduce a method to generate high-quality pseudo-labels for unlabeled low-light images using an ensemble of restoration models and a self-supervised quality assessment metric, significantly improving LLIR performance with limited labeled data.
What Problem Does This Paper Address?
Restoring images captured in low-light conditions is crucial for many applications, but traditional deep learning methods for this task, known as Low-Light Image Restoration (LLIR), typically require a large number of paired low-light and well-exposed images for training. Acquiring such paired data is often expensive and time-consuming. While semi-supervised learning (SSL) offers a way to leverage abundant unlabeled data alongside a small labeled set, existing SSL methods for image restoration are often task-specific (e.g., dehazing, deraining) and not directly applicable to LLIR. This paper addresses the gap of effectively utilizing unlabeled low-light images to train LLIR models when only a limited number of labeled pairs are available.
Methods
The core of the proposed method lies in generating high-quality pseudo-labels for unlabeled low-light images. This is achieved in two main steps:
- Ensemble of Restoration Models: The authors first create an ensemble of LLIR networks. Each network in the ensemble is trained to specialize in restoring specific types of distortions found in the labeled low-light images. This is done by estimating distortion models from labeled pairs and then generating synthetic distorted low-light images from well-exposed ones to train individual models. When applied to an unlabeled low-light image, this ensemble produces multiple restored versions, each potentially good at correcting different distortions.
- Quality-Assisted Pseudo-Label Selection: To select the best restored version from the ensemble to serve as a pseudo-label, a novel unsupervised quality assessment (QA) model is designed. This model, termed Quality Feature CNN (QFCNN), is trained using a contrastive self-supervised learning approach. It learns to extract quality-relevant features from restored low-light subbands. The quality score is then computed by measuring the similarity between these learned features and the average features of well-exposed subbands. Additionally, a Structural Similarity (SSIM) index is used to capture local artifacts, combining with the QFCNN score for a more robust quality metric. The restored image with the highest combined quality score is chosen as the pseudo-label.
The trained LLIR network (specifically, the lowpass subband CNN of their SMSNet architecture) is then trained using both the original labeled pairs and the generated pseudo-labeled unlabeled data.
Datasets
The paper uses the following datasets for training and evaluation:
- See-In-the-Dark (SID) [1]: A dataset for low-light image restoration.
- LOw-Light (LOL) [36]: Another popular dataset for low-light image enhancement.
The authors randomly select 5% and 10% of the ground truth training images and their corresponding low-light versions as labeled data, with the remaining data used as unlabeled.
Key Insights and Ideas
- Ensemble Specialization for LLIR: The idea that an ensemble of models, each tuned to specific distortions, can collectively produce a diverse set of high-quality restorations for unlabeled images is a key insight. This allows for a better chance of finding a good pseudo-label.
- Self-Supervised QA for LLIR: Developing a quality assessment metric that doesn't rely on ground truth and can effectively capture LLIR-specific distortions is crucial. The contrastive learning approach for QFCNN is a novel way to achieve this without human annotations.
- Pseudo-labeling as a Bridge: The paper demonstrates that pseudo-labeling, when guided by a robust quality assessment, can effectively bridge the gap between labeled and unlabeled data in the context of image restoration.
- SMSNet's Suitability for SSL: The analysis of their Simple Multi-Scale restoration Network (SMSNet) reveals that its bandpass subband CNNs generalize well even with very limited data, making it a suitable base architecture for SSL focused on the lowpass subband.
Key Results
- The proposed semi-supervised method achieves superior restoration performance compared to several baselines, including a baseline trained only on labeled data, augmented data, Mean-Teacher, and EnlightenGAN.
- On the SONY dataset with 5% labeled data, the proposed method achieves an SSIM of 0.63, outperforming the baseline (0.58) and Mean-Teacher (0.57). The 100% data trained model achieves 0.74.
- Qualitative results (Figure 4) show that the images enhanced by the proposed method exhibit better perceptual quality than those from other methods.
- Ablation studies (Table 4) indicate that both the QFCNN-based quality index and the SSIM-0 term are important for selecting effective pseudo-labels.
Limitations and Future Work
- Dependence on Labeled Data Diversity: The paper acknowledges that the effectiveness of the ensemble approach depends on the diversity of distortions present in the labeled dataset. If no labeled data point has distortions similar to an unlabeled subband, the method might not produce a good pseudo-label.
- Future Work: The authors suggest exploring more sophisticated ensemble strategies and quality assessment metrics. They also mention the potential for further investigation into the limitations related to distortion diversity in the labeled set.
Summary updated