Enhancing Video Colorization with Deep Learning: A Comprehensive Analysis of Training Loss Functions

Leandro Stival, Ricardo da Silva Torres, Helio PedriniLecture Notes in Networks and Systems 2024 · DOI · OpenAlex

Abstract

Video colorization remains a complex task in computer vision, primarily due to the difficulty of maintaining temporal consistency across frames. This paper presents a systematic investigation into how different combinations of training loss functions—ranging from basic pixel-wise errors to advanced perceptual metrics—influence the performance of deep learning-based colorization models. Utilizing a U-Net architecture integrated with a Vision Transformer (ViT), the study evaluates seven loss functions in various combinations. Our findings indicate that while single-loss models often produce visual artifacts, a multi-loss approach—specifically combining Mean Absolute Error (MAE), Structural Similarity (SSIM), and Perceptual metrics—yields superior results in both objective scores and subjective visual realism.

Introduction

The primary objective of video colorization is to transform grayscale sequences into vibrant, realistic color videos. A significant hurdle in this process is ensuring "temporal consistency," where colors remain stable as objects move or lighting changes across frames. Despite advancements in Deep Learning Video Colorization (DLVC), there is currently no industry-standard methodology for selecting loss functions during training. This research addresses this gap by analyzing seven distinct loss functions to determine the optimal "recipe" for high-quality, flicker-free video colorization.

Methodology

The researchers developed an example-based colorization pipeline centered on a U-Net architecture enhanced with a Vision Transformer (ViT) bottleneck. In this framework, the network processes a grayscale target frame alongside a fully colored reference frame. The ViT’s self-attention mechanism is leveraged to effectively map and propagate colors from the reference to the target.

The U-Net architecture with a Vision Transformer (ViT) bottleneck used to propagate color from a reference frame.

The study systematically evaluates seven loss functions:

  1. Basic Metrics: Mean Absolute Error (MAE) and Mean Squared Error (MSE).
  2. Structural/Perceptual Metrics: Structural Similarity Index (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), and Perceptual Loss (VGG-19 based).
  3. Style/Content Metrics: Style Loss and Content Loss.

These were tested in single, double, and triple combinations. A key innovation in the triple-loss experiments was the introduction of a temporal constraint: the third loss function compared the colorized output to the adjacent frame, enforcing smoothness over time.

Experimental Setup and Datasets

The model was trained and validated using the DAVIS (Densely Annotated VIdeo Segmentation) dataset, which contains 50 high-quality video sequences. To test generalization capabilities, the final evaluation was conducted on 30 unseen videos from the Videvo repository.

Key Insights and Results

The analysis revealed several critical findings regarding model performance:

  • Insufficiency of Simple Losses: Models trained solely on MSE or MAE performed similarly but poorly, often leaving "dead zones" or uncolored regions. This confirms that pixel-level accuracy alone does not translate to high-quality visual results.
  • The Impact of Perceptual Loss: Integrating Perceptual Loss—which evaluates high-level features rather than individual pixels—dramatically reduced visual artifacts and produced more natural color tones.
  • Temporal Anchoring: The most stable results were achieved through "temporal anchoring," where a third loss function compared the current frame's output with its neighbor. This prevented color "jumping" between frames.
  • Quantitative Performance: The highest objective scores were achieved by triple-loss combinations: MAE + SSIM + Perceptual and MSE + SSIM + Perceptual. These models reached a Peak Signal-to-Noise Ratio (PSNR) of 49.532 and an SSIM of 0.975, significantly outperforming single-loss baselines (PSNR ~44-45).

Conclusion and Future Work

This research demonstrates that a multi-loss training strategy is essential for high-fidelity video colorization. While triple-loss models excel in objective metrics, the study also highlights a persistent gap between mathematical metrics and human perception, as some double-loss models were perceived as more "natural."

Future work will focus on:

  1. Developing evaluation metrics that better align with human visual judgment.
  2. Integrating more complex temporal architectures, such as Recurrent Neural Networks (RNNs), with optimized loss functions.
  3. Enhancing color consistency in scenarios involving high-speed motion or rapid scene transitions.

Summary updated

BibTeX
@inproceedings{stival2024enhancing,
  author = {Leandro Stival and Ricardo da Silva Torres and Helio Pedrini},
  title = {{Enhancing Video Colorization with Deep Learning: A Comprehensive Analysis of Training Loss Functions}},
  booktitle = {Lecture Notes in Networks and Systems},
  year = {2024},
  doi = {10.1007/978-3-031-66329-1\_32},
}
Figure