Introduction
General-purpose image editing models, such as InstructPix2Pix, have demonstrated remarkable versatility in following natural language instructions to modify images. However, these models often encounter difficulties when applied to specialized, pixel-level tasks that require high contextual precision, such as image colorization. Colorizing human faces is particularly challenging; it requires a sophisticated understanding of skin textures, lighting, and anatomical context to produce results that appear natural rather than "muddy" or over-saturated. This research explores a specialized fine-tuning approach to adapt InstructPix2Pix specifically for high-fidelity facial colorization.
Methodology
The researchers utilized the InstructPix2Pix architecture, which integrates a Variational Auto-Encoder (VAE), a CLIP text encoder, and a conditional U-Net. To achieve specialization without losing the model's foundational capabilities, the team employed a selective freezing strategy. The VAE and CLIP text encoder remained frozen, while only the U-Net—the core component responsible for denoising and content generation—was fine-tuned.
Prompt Engineering and Robustness
To ensure the model could handle various user inputs, the researchers used GPT-4 to generate 30 synonymous instructions for the task (e.g., "Add color to this photo" or "Make this image colorful"). This diversity helps the model generalize across different phrasing styles, although observations indicated that the CLIP text encoder naturally projects these similar instructions into a relatively tight embedding space.
Dual Loss Evaluation
The training process monitored two distinct metrics:
- Training Loss: A standard noise prediction accuracy metric.
- Validation Loss: Calculated using Mean Squared Error (MSE) within the LAB color space.
The choice of the LAB color space is significant because it separates lightness (L) from color information (A and B), allowing for a more precise evaluation of color accuracy independent of the image's original brightness levels.
Datasets
The study leveraged a subset of 766 images from the IMDB-WIKI dataset, the largest public repository of facial images.
During preprocessing, original color images were converted to grayscale to serve as inputs, while the original versions were retained as the "ground truth" for loss calculation and performance evaluation. The processed dataset has been released as the "colorized_people-dataset" on Hugging Face.
Key Insights
- Learning Rate Sensitivity: Hyperparameter tuning proved vital. A learning rate of $1 \times 10^{-6}$ was too conservative, producing images nearly identical to the grayscale input. Conversely, $1 \times 10^{-5}$ led to "neon" faces with unrealistic contrast. The researchers identified $5 \times 10^{-6}$ as the "Goldilocks" rate for balanced results.
- Loss Misalignment: The team noted a disconnect between noise-based training loss and pixel-based validation loss. Since multiple color palettes can be valid for a single grayscale image, a model might be improving its generative logic even if its specific pixel-wise MSE increases relative to a single ground truth.
Key Results
The fine-tuned model demonstrated superior performance over the base InstructPix2Pix model across all quantitative benchmarks:
- SSIM (Structural Similarity Index): Increased from 0.43 to 0.53, marking a significant improvement in preserving the "feel" and structural integrity of the faces.
- PSNR (Peak Signal-to-Noise Ratio): Improved from 19.78 to 19.90.
- MAE (Mean Absolute Error): Reduced from 22.10 to 21.30.
Qualitatively, the fine-tuned model avoids the muted, grayish tones common in general-purpose models, instead producing vibrant and harmonious skin tones that look significantly more realistic to human observers.
Limitations and Future Directions
Despite the improvements, the researchers identified several areas for future work. The current training objective (noise prediction) is not perfectly aligned with color accuracy (LAB MSE). Future iterations may explore joint optimization, where both loss functions are weighted and trained simultaneously. Additionally, the team suggests exploring "soft prompting"—using learnable prompt vectors—to further refine the model's interpretation of colorization instructions and mitigate the risks of overfitting on small datasets.
Summary updated