TL;DR
This paper tackles the challenge of poor saliency detection performance on low-light images, a common issue in real-world applications. They propose a novel, computationally efficient method to translate well-lit images into realistic-looking low-light proxies using Fourier domain filtering. Training existing saliency detection models on these synthetic low-light images significantly boosts their performance on actual low-light scenes, outperforming other image translation techniques.
What Problem Does This Paper Address?
Saliency detection, the task of identifying prominent objects in an image, is crucial for applications like robot navigation and satellite imagery analysis. However, most existing saliency detection models are trained on well-lit images and perform poorly under low-light conditions. Creating large datasets of annotated low-light images is prohibitively tedious and time-consuming. While image translation methods exist to generate synthetic low-light images, they often require extensive training, are prone to artifacts, or don't yield sufficient improvements. This paper aims to bridge this gap by providing a fast, effective, and generalizable method for creating realistic low-light training data from readily available well-lit images.
Methods
The core of the proposed method lies in a spectrum-inspired image translation technique operating in the Fourier domain. Given a well-lit image ($I_{well}$) and a real low-light image ($I_{low}$), the authors decompose both into their amplitude and phase spectra using the 2D Fourier Transform. Crucially, they retain the phase spectrum of the well-lit image, as phase information is known to preserve structural content. The amplitude spectra are then fused using a weighted average, where more weight is given to the low-light image's amplitude at low frequencies and the well-lit image's amplitude at high frequencies. This fusion is performed within a specific frequency band (band-pass filtering) and incorporates a windowing function (inspired by Blackman windowing) to ensure a smooth transition between frequencies, thereby suppressing ringing artifacts common in Fourier-based transformations. The resulting fused amplitude spectrum, combined with the original well-lit image's phase spectrum, is then transformed back to the image domain using the inverse Fourier Transform to produce the proxy low-light image ($I_{prx}$). This method is computationally efficient, requiring only a few hyper-parameters and a small number of real low-light images for the translation process.
Datasets
- NLPR dataset: Used for its well-lit images with ground truth saliency annotations. A subset of this dataset was used to generate proxy low-light images.
- LIME dataset: A small dataset of real low-light images (5 images) used to guide the translation process.
- SICE dataset: Contains well-lit images with corresponding real low-light images. Used for testing saliency detection models. For ground truth in low-light scenarios, they used saliency predictions from models trained on well-lit SICE images.
- SID dataset: Used for depth estimation experiments, containing well-lit and real low-light images.
- NYU dataset: Used for depth estimation experiments, providing well-lit images.
Key Insights and Ideas
- Phase is key for structure, amplitude for style: The paper leverages the understanding that the phase spectrum of an image primarily encodes its structural information, while the amplitude spectrum influences its appearance and style. By preserving the well-lit image's phase and fusing amplitude spectra, they can transfer the "low-light style" without losing the original image's semantic content.
- Fourier domain for efficient translation: Operating in the Fourier domain allows for a computationally efficient translation process, avoiding the need for complex GAN architectures and their associated training challenges.
- Band-pass filtering and windowing for artifact suppression: The careful application of band-pass filtering and windowing in the frequency domain is critical for mitigating ringing artifacts and ensuring smooth transitions, leading to more visually plausible proxy low-light images.
- Generalizability beyond saliency: The proposed spectrum-inspired translation method is shown to be effective not only for saliency detection but also for other computer vision tasks like depth estimation, highlighting its broad applicability.
Key Results
- Saliency Detection: Networks trained on the proposed proxy low-light images significantly outperform those trained on real low-light images or other translated low-light images (from methods like CUT, HiDT, CoMoGAN, FDA) on the SICE dataset. For example, using BASNet, their method achieved an F-measure of 0.921, compared to 0.874 for FDA and 0.731 for CoMoGAN. The Mean Absolute Error (MAE) was also significantly reduced to 0.092, compared to 0.168 for FDA.
- Depth Estimation: Similarly, for depth estimation on the SID dataset, their method achieved an RMSE of 0.569, outperforming FDA (0.644) and a baseline trained on well-lit images (0.725).
- Efficiency: The proposed method is significantly faster than GAN-based translation methods, requiring minimal training time and only a few hyper-parameters, unlike methods with millions of parameters.
Limitations and Future Work
- Limited real low-light images: The method relies on a small number of real low-light images to guide the translation. While this is a strength in terms of data requirements, the quality of the proxy images is still influenced by the characteristics of these few real low-light examples.
- Fine-tuning benefits: While training on proxy images provides a substantial boost, fine-tuning on a limited set of real low-light images further improves performance, suggesting that a hybrid approach might be optimal.
- Future Work: The authors suggest that their method can be easily generalized to other computer vision tasks in low-light conditions, which they demonstrate with depth estimation. Further exploration of its applicability to other domains is a natural next step.
Summary updated