RAVE: Residual Vector Embedding for CLIP-Guided Backlit Image Enhancement

Tatiana Gaintseva, Martin Benning, Gregory Slabaugh

TL;DR

This paper introduces RAVE, a novel method for enhancing backlit images using CLIP guidance. RAVE significantly speeds up training and improves image quality by directly learning a "residual vector" in CLIP's embedding space, bypassing the need for iterative prompt tuning.

What Problem Does This Paper Address?

Backlit images, common in photography, suffer from dark areas with lost detail due to strong light sources behind the subject. While manual correction is tedious, automated solutions struggle with global adjustments that overexpose already well-lit regions. A key challenge is the lack of paired data (backlit image with its corresponding well-lit version), making unsupervised learning crucial. Existing methods like CLIP-LIT use CLIP's vision-language understanding to guide enhancement but require complex, iterative prompt tuning, which is slow and can lead to artifacts.

Methods

The authors build upon the CLIP-LIT framework, which uses CLIP's ability to align images and text in a shared embedding space to guide image enhancement. They propose two main improvements:

  • CLIP-LIT-Latent: Instead of learning text prompts, this method directly learns "vectors" in CLIP's latent space that represent well-lit and backlit images. This simplifies the process and potentially allows for using other vision models besides CLIP.
  • RAVE (Residual Vector Embedding): This is the core contribution. RAVE computes a single "residual vector" by taking the difference between the average CLIP embeddings of well-lit and backlit training images. This vector acts as a direct guide for the enhancement network, pushing backlit images towards the "well-lit" region of the CLIP embedding space. This approach eliminates the need for iterative prompt refinement, drastically reducing training time and stabilizing the process.

Datasets

  • BAID: Used for both training and testing, containing paired backlit and well-lit images.
  • DIV2K: Used for training in the unpaired setting.
  • LOL-v1: Used for evaluating performance on low-light image enhancement.

Key Insights and Ideas

  • Direct Latent Space Manipulation: Tuning embeddings directly in CLIP's latent space (CLIP-LIT-Latent and RAVE) is more efficient and effective than manipulating text prompts.
  • The Power of the Residual Vector: A single, pre-computed residual vector in CLIP's embedding space can effectively guide image enhancement, simplifying the training pipeline and improving results.
  • Interpretability of Residual Vectors: The residual vector learned by RAVE can be interpreted by examining its similarity to vocabulary tokens in CLIP. This revealed biases in unpaired training data (e.g., differences between city and nature imagery) and showed that the vector primarily captures the shift from "dark" to "normal" lighting.
  • Bias Correction: The interpretability of the residual vector opens avenues for correcting biases in training data.

Key Results

  • Speedup: RAVE converges up to 25 times faster than CLIP-LIT and CLIP-LIT-Latent.
  • Performance: RAVE achieves state-of-the-art results on the BAID test dataset in both paired and unpaired training settings, outperforming previous methods in PSNR, SSIM, and LPIPS metrics.
  • Quality: RAVE produces high-quality enhanced images with fewer artifacts, particularly in highly under-exposed regions, compared to CLIP-LIT and CLIP-LIT-Latent.
  • Efficiency: RAVE is more lightweight, requiring less compute and memory, making it suitable for higher resolution images.
  • Low-Light Performance: While not competitive with specialized low-light methods, RAVE shows decent visual quality on the LOL-v1 dataset.

Limitations and Future Work

  • Non-optimal Enhancement of Lost Information: The paper acknowledges that their models struggle with regions where information is completely lost, leading to ambiguous restoration. They cannot regenerate highly-accurate textures in such cases.
  • Sub-optimal Lightening: Models are not trained to perfectly match ground truth pixels, which can lead to non-ideal visual quality in some lightening scenarios.
  • Low-Light Performance: RAVE does not outperform specialized low-light enhancement methods.
  • Future Work: The authors suggest that the interpretability of residual vectors can be used for bias correction. They also hint at further modifications to the residual vector (RAVE shifted) to improve results, especially in unpaired settings.

Summary updated

Abstract

In this paper we propose a novel modification of Contrastive Language-Image Pre-Training (CLIP) guidance for the task of unsupervised backlit image enhancement. Our work builds on the state-of-the-art CLIP-LIT approach, which learns a prompt pair by constraining the text-image similarity between a prompt (negative/positive sample) and a corresponding image (backlit image/well-lit image) in the CLIP embedding space. Learned prompts then guide an image enhancement network. Based on the CLIP-LIT framework, we propose two novel methods for CLIP guidance. First, we show that instead of tuning prompts in the space of text embeddings, it is possible to directly tune their embeddings in the latent space without any loss in quality. This accelerates training and potentially enables the use of additional encoders that do not have a text encoder. Second, we propose a novel approach that does not require any prompt tuning. Instead, based on CLIP embeddings of backlit and well-lit images from training data, we compute the residual vector in the embedding space as a simple difference between the mean embeddings of the well-lit and backlit images. This vector then guides the enhancement network during training, pushing a backlit image towards the space of well-lit images. This approach further dramatically reduces training time, stabilizes training and produces high quality enhanced images without artifacts, both in supervised and unsupervised training regimes. Additionally, we show that residual vectors can be interpreted, revealing biases in training data, and thereby enabling potential bias correction.
BibTeX
@misc{gaintseva2024rave,
  author = {Tatiana Gaintseva and Martin Benning and Gregory Slabaugh},
  title = {{RAVE: Residual Vector Embedding for CLIP-Guided Backlit Image Enhancement}},
  year = {2024},
  eprint = {2404.01889},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.01889},
}
Figure