EnColor: Improving Visual Accessibility with a Deep Encoder-Decoder Image Corrector for Color Vision Deficient Individuals

Satyam Goyal, Kavya Sasikumar, Rohan Sheth, Akash Seelam, Taeyeong Choi, Xin LiuAAAI 2024 · OpenReview

Introduction

Color Vision Deficiency (CVD), often colloquially called color blindness, affects millions of people worldwide. It creates significant daily challenges, from navigating traffic signals to identifying ripe produce in agricultural settings. For instance, a farmer with CVD might struggle to distinguish a ripe red tomato from an unripe green one against a backdrop of green leaves.

Physiologically, CVD is caused by deficiencies in the spectral sensitivity of the eye's cone cells (S, M, and L cones), which act as a biological filter on incoming light.

Spectral sensitivity of S, M, and L cones in the human eye, illustrating the physiological basis of color perception.

While previous technological solutions exist, they often rely on simplistic linear models that focus on contrast enhancement—making objects look different from one another—rather than color restoration—making objects look like their "true" colors. EnColor aims to bridge this gap by using deep learning to transform images so that a person with CVD perceives a result as close as possible to what a person with normal trichromatic vision sees.

Methods

The core of EnColor is a Deep Convolutional Encoder-Decoder architecture designed for non-linear color mapping.

  1. The Network: The encoder uses two convolutional layers to compress the input image into an abstract, low-dimensional representation. The decoder then uses two transposed convolutional layers to "up-sample" this representation back into a full-sized image with modified color values.
  2. The CVD Simulator: The most novel aspect of the training process is the inclusion of a "CVD Simulator" in the loss calculation. This simulator uses established physiological models (linear transformation matrices) to mimic how a person with a specific type of CVD (like deuteranopia) would see an image.
  3. The Training Loop: During training, an original image ($I$) is passed through EnColor to produce a corrected image ($I'$). This corrected image is then passed through the CVD Simulator ($S$) to produce $S(I')$. The loss function calculates the Mean Absolute Error (MAE) between the simulated corrected image $S(I')$ and the original image $I$.

Essentially, the model is trained to answer the question: "How should I change this image so that when a color-blind person looks at it, it looks 'normal' to them?"

Experimental Setup and Datasets

The researchers focused their evaluation on a practical, high-impact domain: agriculture. They utilized the Laboro Tomato Dataset, which contains over 600 high-quality images of ripe and unripe tomatoes in various lighting conditions and stages of growth. The authors used 600 images for training, 43 for validation, and 150 for testing, with all images resized to 648 × 484 to optimize computation.

Key Results

EnColor prioritizes "color retention," aiming for a naturalistic look that preserves the original intent of the visual information rather than just maximizing contrast.

  • Quantitative Performance: EnColor achieved a 16.8% improvement in color restoration (measured via Mean Squared Error) compared to "Fuzzy logic" baselines and a 29.1% improvement over standard linear models.
  • Structural Integrity: Using the Structural Similarity Index (SSIM), the authors demonstrated that EnColor maintains the texture and feel of the original image better than most heuristic-based methods.
  • Human Subjective Assessment: A survey of 43 participants ranked EnColor the highest for both "contrast" and "ease of identifying ripe tomatoes." On a scale of 1-5 (where 1 is best), EnColor scored 1.59 for contrast and 1.45 for ripeness identification.
Visual comparison: (a) Original image, (b) EnColor corrected image, (c) Simulation of original image for CVD, (d) Simulation of corrected image for CVD. Note how (d) more closely resembles (a) than (c) does.

Discussion and Future Work

EnColor demonstrates that end-to-end non-linear mapping can significantly improve visual accessibility. By using the CVD simulator as a "teacher" within the loss function, the model automatically adapts to different types of CVD without manual parameter tuning.

However, limitations remain. CVD is a spectrum, and future work is needed to develop methods that can automatically identify an individual's specific "intensity" of CVD to provide bespoke correction. Additionally, the authors suggest that incorporating Generative Adversarial Networks (GANs) or including SSIM directly in the loss function could further improve the visual quality and sharpness of the corrected images.

Summary updated

Abstract

Individuals with color vision deficiencies (CVDs) often face significant challenges in accessing vital information for decision-making. In response, we introduce EnColor—a deep Encoder-decoder Color corrector for images, enabling individuals with CVDs to perceive the contents in originally intended colorization. Our network architecture is designed to effectively capture essential visual features for reconstructing standard images into color-corrected versions. In particular, our training pipeline is integrated with a CVD simulator so as to ensure the fidelity of output throughout the lens of individuals with impaired color vision. For evaluation, we focus primarily on tomato images, considering the profound impact of color vision deficiencies on practical domains like agri-food systems. Our quantitative results demonstrate that the EnColor model achieves over 16.8% improvement over previously introduced algorithms in terms of color retention, supporting our design choices. Furthermore, a survey with 43 participants provides subjective assessments with the highest scores on our method. Additionally, specific visual examples are presented to highlight accurately restored colors. We also publicly share all our codes of EnColor as well as the baseline methods to ensure reproducibility and facilitate more studies in CVD correction.
BibTeX
@inproceedings{goyal2024encolor,
  author = {Satyam Goyal and Kavya Sasikumar and Rohan Sheth and Akash Seelam and Taeyeong Choi and Xin Liu},
  title = {{EnColor: Improving Visual Accessibility with a Deep Encoder-Decoder Image Corrector for Color Vision Deficient Individuals}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=MI8gdibqXX},
}
Figure