HSVFormer: Robust and Unsupervised HSV-based Transformer Framework for Low-Light Image Enhancement

Xiaogang Du, Meng Yang, Tao Lei, Xuejun Zhang, Yingbo Wang, Asoke K. Nandi2024 · DOI

This paper, recently accepted for ICME 2024, introduces HSVFormer, a novel framework designed to tackle the persistent challenges of low-light image enhancement (LLIE) without the need for paired training data.

TL;DR

HSVFormer is an unsupervised low-light enhancement framework that combines HSV color theory with Transformer architectures to prevent color distortion and suppress noise. By introducing a "value-guided" attention mechanism and simulating real-world brightness corruptions, it achieves state-of-the-art results across multiple benchmarks without ever seeing a "perfect" ground-truth image during training.

What Problem Does This Paper Address?

Enhancing low-light images isn't just about "turning up the brightness." Existing methods often struggle with three major hurdles:

  1. Corruptions and Noise: Boosting the signal in dark areas inevitably boosts the noise, leading to grainy artifacts.
  2. Color Distortion: Most models operate in RGB space. When you adjust the R, G, and B channels independently or even together, the resulting colors often look "off" or unnatural.
  3. Data Scarcity: Supervised learning requires pairs of low-light and "normal" light images. Collecting these in the real world is incredibly difficult, and synthetic data often fails to generalize to real-world camera sensors.

While Transformers have shown promise in capturing long-range dependencies (helping with global consistency), they can be computationally expensive and prone to "checkerboard" artifacts if not properly constrained.

Methods

The authors propose a two-pronged strategy: decoupling color from brightness and using a guided Transformer architecture.

1. The HSV Decoupling

Instead of working in RGB, the model converts images to the HSV (Hue, Saturation, Value) color space. By keeping the Hue and Saturation channels mostly intact and focusing enhancement efforts on the Value (V) channel, the model inherently avoids the color-shifting issues that plague RGB-based methods.

2. UVENet (Unsupervised Value Enhancement Network)

The core of the system is the UVENet, which consists of two sub-networks:

  • Value Estimation Network (VEN): This simulates real-world corruptions. It introduces "brightness disturbance" to the V-channel, making the model more robust to the messy, non-linear noise found in real low-light scenes.
  • Value-Guided Transformer (VGT): This is a three-scale encoder-decoder. It takes the corrupted V-channel and restores it.

3. VG-MCSA (Value-Guided Multi-head Channel Self-Attention)

This is the paper's primary architectural innovation. Standard self-attention treats all parts of an image equally. VG-MCSA uses the brightness information from the V-channel to "guide" the attention mechanism. This allows the model to focus specifically on dark regions that require more aggressive denoising and detail restoration while maintaining the integrity of better-lit areas.

4. Unsupervised Training

Since there are no "perfect" labels, the model is trained using a suite of non-reference loss functions, including illumination smoothness, reflectance consistency, and exposure control.

Datasets

The authors evaluated HSVFormer on several rigorous benchmarks:

  • SICE Dataset: Used for primary training and full-reference evaluation (4,413 images).
  • Generalization Testing: To prove the model works in the "wild," they tested it on five popular no-reference datasets: DarkFace, DICM, LIME, MEF, and VV. These datasets are challenging because they contain diverse real-world lighting conditions without any ground-truth comparisons.

Key Insights and Ideas

  • Simulating the "Mess": By intentionally disturbing the brightness channel during training (the VEN module), the authors essentially "pre-train" the model to handle the unpredictable noise that appears when you brighten a real-world dark photo.
  • Efficiency through Guidance: By using the V-channel to guide attention, HSVFormer uses fewer attention heads (4 vs. 8 in similar models like Retinexformer) and a smaller input dimension, making it more computationally efficient while achieving better results.
  • The Power of Priors: The paper proves that classical color theory (HSV) still has a vital place in the era of deep learning. Using the right color space acts as a powerful "physical prior" that prevents the neural network from making nonsensical color choices.

Key Results

HSVFormer outperformed several state-of-the-art (SOTA) methods, including LIME, RUAS, and LLFormer.

  • Quantitative: On the SICE dataset, it achieved a PSNR of 17.245 and an SSIM of 0.535, taking the top spot in both metrics.
  • No-Reference Performance: On the five generalization datasets, it achieved the best average scores in BRISQUE and PIQE (metrics that measure perceived image quality without a reference image).
  • Qualitative: Visual results show significantly less noise in deep shadows and more natural skin tones and sky colors compared to CNN-based unsupervised methods like Zero-DCE.

Limitations and Future Work

While the model is more efficient than some of its Transformer-based predecessors, the authors note that reducing computational complexity further is a goal for the future. Additionally, while the HSV decoupling solves many color issues, extremely low-light conditions where the sensor records almost zero color information still pose a challenge for any restoration-based approach.

The code is available for the community at: https://github.com/m0fig/HSVFormer

Summary updated

Abstract

The following three factors restrict the application of existing low-light image enhancement methods: corruptions induced by the light-up process, color distortion, and a restricted generalization capacity due to limited paired training data. To address these limitations, we first combine HSV theory and Transformer, proposing a robust unsupervised low-light image enhancement framework, named HSVFormer. Secondly, we introduce brightness disturbance and design an unsupervised value enhancement network, which estimates brightness information and restores degraded brightness information to obtain enhanced reflectance. Finally, we utilize the V-subspace and devise a value-guided multi-head channel self-attention to capture brightness representations of regions with different brightness conditions and guide the modeling of non-local interactions. Experiment results on publicly available datasets demonstrate that HSVFormer can achieve superior performance compared with state-of-the-art approaches. The code is available at https://github.com/m0fig/HSVFormer.
BibTeX
@misc{du2024hsvformer,
  author = {Xiaogang Du and Meng Yang and Tao Lei and Xuejun Zhang and Yingbo Wang and Asoke K. Nandi},
  title = {{HSVFormer: Robust and Unsupervised HSV-based Transformer Framework for Low-Light Image Enhancement}},
  year = {2024},
  doi = {10.1109/icme57554.2024.10688351},
}
Figure