Using colorization as a tool for automatic makeup suggestion

Shreyank N GowdaarXiv 2019 · DOI · arXiv

Abstract

Colorization is the method of converting an image in grayscale to a fully color image. There are multiple methods to do the same. Old school methods used machine learning algorithms and optimization techniques to suggest possible colors to use. With advances in the field of deep learning, colorization results have improved consistently with improvements in deep learning architectures. The latest development in the field of deep learning is the emergence of generative adversarial networks (GANs) which is used to generate information and not just predict or classify. As part of this report, 2 architectures of recent papers are reproduced along with a novel architecture being suggested for general colorization. Following this, we propose the use of colorization by generating makeup suggestions automatically on a face. To do this, a dataset consisting of 1000 images has been created. When an image of a person without makeup is sent to the model, the model first converts the image to grayscale and then passes it through the suggested GAN model. The output is a generated makeup suggestion. To develop this model, we need to tweak the general colorization model to deal only with faces of people.

Introduction

Traditionally, image colorization was a manual, painstaking process involving "scribbling" colors onto regions or finding near-identical reference images. While deep learning has automated general colorization—turning old black-and-white photos into color—applying this technology to makeup suggestion presents a unique challenge. Existing makeup tools often require complex 3D facial modeling, expensive hardware, or manual user intervention. This research explores whether a fully automatic system can understand facial features well enough to suggest aesthetically pleasing makeup without human expertise in the loop.

Methods

The core innovation lies in treating makeup as a distribution of chrominance (color) over a luminance (brightness) map.

The Architecture

The proposed novel architecture improves upon previous models by replacing simple autoencoders with Generative Adversarial Networks (GANs). Specifically, DC-GANs are utilized for their ability to generate high-quality, realistic imagery.

Proposed architecture for automatic makeup suggestion using dual-channel DC-GANs and DenseNet-161.

Dual-Channel Prediction in LAB Space

Instead of predicting an RGB image directly, the model operates in the LAB color space. The 'L' channel (lightness) serves as the input. The system uses two separate DC-GANs: one to predict the 'A' channel (green-to-red) and one for the 'B' channel (blue-to-yellow). Using the LAB color space minimizes per-pixel correlation, making it easier for the neural network to learn colorization compared to the highly correlated RGB channels.

Feature Extraction

To provide contextual understanding, a DenseNet-161 (pretrained on ImageNet) is used as a backbone. For the specific makeup task, the model is refined to focus purely on facial features. The Generator creates the A and B channels, while a Discriminator attempts to distinguish the "fake" colorized face from a real photo with makeup. This adversarial training pushes the model toward more realistic results.

Datasets

The study utilizes two distinct datasets:

  • ImageNet: A massive set of 600,000 images used to train and evaluate the model for general colorization tasks (landscapes, animals, etc.).
  • Custom Makeup Dataset: The authors curated 1,000 high-resolution (256x256) images from Google and Flickr. These images were manually cropped to focus on key facial features (eyes, lips, skin) and selected to represent diverse ethnicities.

Results

The paper provides qualitative evidence of the model's success across general and specialized tasks.

General Colorization

The model successfully colorized complex scenes with high fidelity compared to ground truth images. Examples include natural landscapes and human activities, demonstrating the robustness of the underlying GAN architecture.

Qualitative results for general colorization showing grayscale input, model output, and ground truth.

Makeup Generation

In facial tests, the model automatically applied distinct lipstick shades, eyeshadow, and skin-tone enhancements. The results appear natural and aligned with facial landmarks without requiring manual "hints." The model learns that "makeup" is essentially the delta between a bare face and a styled one in the chrominance space.

Conclusion and Future Work

This research demonstrates that makeup suggestion can be effectively framed as a specialized colorization task. While results are promising, future work could involve:

  • Dataset Scaling: Expanding the 1,000-image dataset to improve GAN robustness and diversity.
  • User Control: Allowing users to provide "hints" or select specific styles (e.g., "evening look") to guide the GAN's output.
  • Quantitative Metrics: Implementing standardized metrics or user studies to measure the aesthetic quality of the suggested makeup.

Summary updated

Abstract

Colorization is the method of converting an image in grayscale to a fully color image. There are multiple methods to do the same. Old school methods used machine learning algorithms and optimization techniques to suggest possible colors to use. With advances in the field of deep learning, colorization results have improved consistently with improvements in deep learning architectures. The latest development in the field of deep learning is the emergence of generative adversarial networks (GANs) which is used to generate information and not just predict or classify. As part of this report, 2 architectures of recent papers are reproduced along with a novel architecture being suggested for general colorization. Following this, we propose the use of colorization by generating makeup suggestions automatically on a face. To do this, a dataset consisting of 1000 images has been created. When an image of a person without makeup is sent to the model, the model first converts the image to grayscale and then passes it through the suggested GAN model. The output is a generated makeup suggestion. To develop this model, we need to tweak the general colorization model to deal only with faces of people.
BibTeX
@misc{gowda2019using,
  author = {Shreyank N Gowda},
  title = {{Using colorization as a tool for automatic makeup suggestion}},
  year = {2019},
  eprint = {1906.07421},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.07421},
}
Figure