PRISM: A New Lens for Improved Color Understanding

Arjun R. Akula, Garima Pruthi, Inderjit S. Dhillon, Pradyumna Narayana, Sugato Basu, Varun JampaniEMNLP 2024 · DOI

TL;DR

This paper introduces PRISM, a novel method to significantly improve the ability of vision-language models (VLMs) like CLIP to understand and differentiate precise color shades. By introducing a new dataset and fine-tuning strategy, PRISM achieves state-of-the-art performance in color-specific retrieval tasks without compromising general VLM capabilities.

What Problem Does This Paper Address?

Vision-language models (VLMs), such as CLIP, have shown remarkable success in understanding the relationship between images and text. However, a critical limitation persists: their ability to grasp nuanced and precise color distinctions. While VLMs can identify general colors, they often struggle with subtle variations or specific brand colors that are crucial for applications like branding, advertising, and detailed image generation. This lack of precise color understanding can lead to misinterpretations and errors in downstream tasks, hindering the development of truly sophisticated AI systems. The paper aims to bridge this gap by enhancing the color discernment capabilities of VLMs.

Methods

The core of PRISM lies in a fine-tuning approach that extends CLIP's capabilities. Key techniques include:

  • Rare-token Identifiers: For colors not recognized as standard HTML colors, PRISM associates them with unique, rare tokens in CLIP's vocabulary. This allows the model to learn specific representations for these custom colors.
  • Disentangled Fine-tuning: The method aims to disentangle color-specific information from general semantic content. This is achieved by using hard negatives during training, which are images that are similar but differ subtly in color, forcing the model to focus on precise color distinctions.
  • Prior Preservation Loss: To prevent overfitting and language drift during fine-tuning on a smaller, specialized dataset, PRISM employs regularization losses. These losses ensure that the model's general image and text embeddings are preserved, maintaining its performance on broader tasks.
  • Repurposable Training Data: PRISM utilizes a curated dataset of 100 image-text pairs that can be easily adapted for fine-tuning with any desired RGB color.

Datasets

  • Training Dataset: A custom dataset of 100 diverse and high-quality image-text pairs was created. This dataset was generated using GPT-4 for prompt creation, Stable Diffusion XL for image synthesis, and SAM for object segmentation, allowing for precise control over object colors.
  • Evaluation Benchmark: ColorLens: A novel evaluation framework introduced by the authors. ColorLens includes both "seen" and "unseen" test sets, designed to assess a model's precision in understanding precise colors. It features text-to-image retrieval and image-to-text retrieval settings with various negative sampling strategies, including "hard negatives" that differ only in subtle color variations.
  • Existing Datasets: The LAION-400M dataset was used to source images for common colors, and a subset of it was used for prior preservation loss.

Key Insights and Ideas

  • The Importance of Nuance: The paper highlights that while VLMs are powerful, they often overlook the critical role of precise color in human perception and branding. PRISM demonstrates that even subtle color differences can be learned and leveraged.
  • Synthetic Data for Specific Tasks: The creation of a targeted synthetic dataset (100 image-text pairs) proved effective for fine-tuning VLMs for a specialized task like color understanding, without requiring massive amounts of real-world data for that specific niche.
  • Balancing Specialization and Generalization: The prior preservation loss is crucial for ensuring that fine-tuning for color understanding doesn't degrade the model's performance on general image-text matching tasks, a common pitfall in specialized fine-tuning.
  • Hard Negatives as a Key Tool: The strategic use of hard negatives is instrumental in forcing the model to learn fine-grained color distinctions, rather than relying on broader semantic cues.

Key Results

  • State-of-the-Art Performance: PRISM significantly outperforms baseline models, including CLIP and its fine-tuned variants, on the ColorLens benchmark for both text-to-image and image-to-text retrieval tasks.
  • Improved Precision and Rank: In color-based retrieval, PRISM shows substantial improvements in precision@1 and rank metrics, especially when dealing with subtle color variations. For instance, in some color-specific retrieval tasks, PRISM achieves precision scores of 30.0-50.0, while baselines struggle to reach double digits.
  • Robustness: PRISM maintains strong performance on standard image-text matching tasks, demonstrating that its color specialization does not come at the cost of general VLM capabilities.
  • Qualitative Improvements: Visualizations show PRISM accurately matching specific color shades (e.g., "slate blue ball," "papayawhip umbrella") where CLIP falters.

Limitations and Future Work

  • Dataset Size: While the 100-pair dataset is effective for fine-tuning, the authors acknowledge that larger, more diverse datasets could further enhance generalization.
  • Computational Cost: Fine-tuning large VLMs can be computationally intensive, though PRISM's approach aims for efficiency.
  • Future Work: The authors suggest extending PRISM to more complex color-related tasks, such as color-aware image editing and generation, and exploring its application in domains where precise color is paramount, like fashion and interior design. They also propose further research into disentangling other visual attributes beyond color.

Summary updated

BibTeX
@inproceedings{akula2024prism,
  author = {Arjun R. Akula and Garima Pruthi and Inderjit S. Dhillon and Pradyumna Narayana and Sugato Basu and Varun Jampani},
  title = {{PRISM: A New Lens for Improved Color Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.121},
}
Figure