Introduction
Automatic video colorization is a challenging task that involves transforming grayscale video sequences into realistic color versions without human intervention. While image colorization has seen significant progress, extending these techniques to video introduces complex issues regarding temporal consistency and the inherent ambiguity of color assignments. This paper presents a novel framework that leverages self-regularization and diversity loss to achieve vibrant and temporally stable results.
Challenges in Video Colorization
The authors identify two primary hurdles in fully automatic video colorization:
- Multi-modality: Grayscale pixels do not map to a single unique color. For instance, a grayscale car could realistically be red, blue, or silver. Standard models often "average" these possibilities, leading to desaturated or "brownish" outputs.
- Temporal Inconsistency: Independent colorization of frames often results in "flickering" artifacts, where the color of an object shifts erratically between consecutive frames due to slight variations in model predictions.
Proposed Framework
The proposed system utilizes a two-step pipeline comprising two deep neural networks designed to handle spatial colorization and temporal refinement respectively.
1. Colorization Network ($f$)
The colorization network takes a grayscale frame and produces multiple diverse colorization candidates. It utilizes "hypercolumn" features from a pre-trained VGG-19 model to integrate high-level semantic information (e.g., identifying a tree) with low-level textural details.
2. Refinement Network ($g$)
To ensure temporal consistency, the refinement network processes the output of the colorization network by considering neighboring frames. It employs optical flow to warp colors from one frame to the next. To handle errors such as occlusions, the network uses confidence maps to determine which warped pixels are reliable.
Key Innovations
Self-Regularization via Bilateral Loss
The model is trained using a Bilateral Loss function that enforces spatial consistency. By finding the $K$-nearest neighbors for pixels in a bilateral space (accounting for both position and color), the model learns to assign similar colors to pixels belonging to the same object, resulting in sharper boundaries and reduced color bleeding.
Diversity Loss
To address the multi-modality problem, a diversity loss encourages the model to explore different plausible color "modes." This allows the system to generate multiple output versions with distinct styles (e.g., warmer vs. cooler tones) rather than collapsing into a single, dull average.
Experimental Results
The framework was evaluated on the ImageNet, DAVIS, and Videvo datasets. Quantitative results show that the model achieves superior performance, with a PSNR of 30.35 on DAVIS and 30.50 on Videvo, outperforming existing baselines.
In perceptual user studies, the proposed method was preferred over state-of-the-art techniques in more than 80% of cases on the DAVIS dataset. Qualitatively, the results exhibit a significant reduction in flickering artifacts, producing videos that appear professionally colorized.
Limitations and Conclusion
While the framework represents a significant advancement, it still faces challenges. The temporal refinement is dependent on the accuracy of optical flow, which can struggle with extremely fast motion or complex occlusions. Additionally, selecting the "best" colorization from the diverse outputs currently relies on simple heuristics like saturation. Future work may involve interactive systems where users can select their preferred artistic style. Overall, this work demonstrates that self-regularization and diversity are key to achieving high-quality, automatic video colorization.
Summary updated