Fine-Tuning InstructPix2Pix for Advanced Image Colorization

Zifeng An, Zijing Xu, Eric Fan, Qi CaoarXiv 2023 · DOI · arXiv

Introduction

General-purpose image editing models, such as InstructPix2Pix, have demonstrated remarkable versatility in following natural language instructions to modify images. However, these models often encounter difficulties when applied to specialized, pixel-level tasks that require high contextual precision, such as image colorization. Colorizing human faces is particularly challenging; it requires a sophisticated understanding of skin textures, lighting, and anatomical context to produce results that appear natural rather than "muddy" or over-saturated. This research explores a specialized fine-tuning approach to adapt InstructPix2Pix specifically for high-fidelity facial colorization.

Methodology

The researchers utilized the InstructPix2Pix architecture, which integrates a Variational Auto-Encoder (VAE), a CLIP text encoder, and a conditional U-Net. To achieve specialization without losing the model's foundational capabilities, the team employed a selective freezing strategy. The VAE and CLIP text encoder remained frozen, while only the U-Net—the core component responsible for denoising and content generation—was fine-tuned.

Prompt Engineering and Robustness

To ensure the model could handle various user inputs, the researchers used GPT-4 to generate 30 synonymous instructions for the task (e.g., "Add color to this photo" or "Make this image colorful"). This diversity helps the model generalize across different phrasing styles, although observations indicated that the CLIP text encoder naturally projects these similar instructions into a relatively tight embedding space.

Dual Loss Evaluation

The training process monitored two distinct metrics:

  1. Training Loss: A standard noise prediction accuracy metric.
  2. Validation Loss: Calculated using Mean Squared Error (MSE) within the LAB color space.

The choice of the LAB color space is significant because it separates lightness (L) from color information (A and B), allowing for a more precise evaluation of color accuracy independent of the image's original brightness levels.

Datasets

The study leveraged a subset of 766 images from the IMDB-WIKI dataset, the largest public repository of facial images.

A grayscale input image used for training, derived from the IMDB-WIKI dataset.

During preprocessing, original color images were converted to grayscale to serve as inputs, while the original versions were retained as the "ground truth" for loss calculation and performance evaluation. The processed dataset has been released as the "colorized_people-dataset" on Hugging Face.

Key Insights

  • Learning Rate Sensitivity: Hyperparameter tuning proved vital. A learning rate of $1 \times 10^{-6}$ was too conservative, producing images nearly identical to the grayscale input. Conversely, $1 \times 10^{-5}$ led to "neon" faces with unrealistic contrast. The researchers identified $5 \times 10^{-6}$ as the "Goldilocks" rate for balanced results.
  • Loss Misalignment: The team noted a disconnect between noise-based training loss and pixel-based validation loss. Since multiple color palettes can be valid for a single grayscale image, a model might be improving its generative logic even if its specific pixel-wise MSE increases relative to a single ground truth.

Key Results

The fine-tuned model demonstrated superior performance over the base InstructPix2Pix model across all quantitative benchmarks:

  • SSIM (Structural Similarity Index): Increased from 0.43 to 0.53, marking a significant improvement in preserving the "feel" and structural integrity of the faces.
  • PSNR (Peak Signal-to-Noise Ratio): Improved from 19.78 to 19.90.
  • MAE (Mean Absolute Error): Reduced from 22.10 to 21.30.
A high-fidelity colorization result demonstrating the model's ability to generate natural, vibrant skin tones.

Qualitatively, the fine-tuned model avoids the muted, grayish tones common in general-purpose models, instead producing vibrant and harmonious skin tones that look significantly more realistic to human observers.

Limitations and Future Directions

Despite the improvements, the researchers identified several areas for future work. The current training objective (noise prediction) is not perfectly aligned with color accuracy (LAB MSE). Future iterations may explore joint optimization, where both loss functions are weighted and trained simultaneously. Additionally, the team suggests exploring "soft prompting"—using learnable prompt vectors—to further refine the model's interpretation of colorization instructions and mitigate the risks of overfitting on small datasets.

Summary updated

Abstract

This paper presents a novel approach to human image colorization by fine-tuning the InstructPix2Pix model, which integrates a language model (GPT-3) with a text-to-image model (Stable Diffusion). Despite the original InstructPix2Pix model's proficiency in editing images based on textual instructions, it exhibits limitations in the focused domain of colorization. To address this, we fine-tuned the model using the IMDB-WIKI dataset, pairing black-and-white images with a diverse set of colorization prompts generated by ChatGPT. This paper contributes by (1) applying fine-tuning techniques to stable diffusion models specifically for colorization tasks, and (2) employing generative models to create varied conditioning prompts. After finetuning, our model outperforms the original InstructPix2Pix model on multiple metrics quantitatively, and we produce more realistically colored images qualitatively. The code for this project is provided on the GitHub Repository https://github.com/AllenAnZifeng/DeepLearning282.
BibTeX
@misc{an2023fine,
  author = {Zifeng An and Zijing Xu and Eric Fan and Qi Cao},
  title = {{Fine-Tuning InstructPix2Pix for Advanced Image Colorization}},
  year = {2023},
  eprint = {2312.04780},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.04780},
}
Figure