Abstract
Video colorization remains a challenging task due to the ill-posed nature of mapping grayscale values to color and the requirement for temporal consistency across frames. We present ST-HVC, a novel framework that leverages global color histograms and spatial-temporal connections to produce vibrant, flicker-free video colorization. By integrating temporal sharpness and optical flow into a U-shaped transformer-like architecture, ST-HVC effectively mitigates motion blur and inconsistent coloring. Experimental results on DAVIS and Videvo datasets demonstrate that ST-HVC outperforms state-of-the-art methods in both visual quality and temporal stability.
1. Introduction
The task of video colorization involves converting legacy grayscale footage or monochromatic sensor data into realistic color video. This process is fundamentally "ill-posed" because a single grayscale intensity can correspond to a wide variety of plausible colors. When processed frame-by-frame, this ambiguity often leads to "temporal flickering," where the color of an object shifts inconsistently between consecutive frames.
Existing methodologies typically fall into two categories: image-based methods and video-based methods. Image-based approaches often produce high-quality individual frames but lack the temporal coherence necessary for video. Conversely, existing video-based methods often prioritize consistency at the expense of vibrancy, leading to washed-out or blurry results. ST-HVC (Histogram-guided Video Colorization with Spatial-Temporal connection) is designed to bridge this gap, providing a system that is both temporally stable and visually vivid without requiring extensive manual intervention.
2. Methodology
The ST-HVC framework introduces several structural innovations to handle the complexities of video colorization.
2.1 Histogram Guidance
To maintain a consistent color palette throughout a video sequence, the model extracts a color histogram from the middle frame of the clip. This histogram serves as a global reference, guiding the network to maintain a stable color distribution across all frames. This approach simplifies the guidance process compared to methods that require user-provided exemplars or frame-by-frame manual hints.
2.2 Temporal Sharpness and Flow Integration
A significant challenge in video colorization is handling motion blur. ST-HVC explicitly incorporates "temporal sharpness" and optical flow as input priors. Optical flow is calculated using the RAFT algorithm, allowing the model to track pixel movement. By feeding sharpness data into the encoder, the network can distinguish between intentional motion blur and areas that require high-frequency detail preservation, thereby reducing artifacts and smudging.
2.3 Joint Flow and Histogram Module (JFHM)
The core of the ST-HVC architecture is the Joint Flow and Histogram Module (JFHM). This module utilizes a decoupled attention mechanism to process spatial and temporal information efficiently:
- Spatial Attention: Focuses on understanding textures and relationships within neighboring pixels of a single frame.
- Temporal Attention: Tracks object movement across the sequence to ensure that color remains "attached" to specific objects as they move through the scene.
2.4 U-Shape Network and DRBlocks
The architecture follows a U-Net style structure, utilizing Dynamic Region-aware Convolutions (DRBlocks) during the feature recombination phase. DRBlocks adaptively learn different kernels based on local illumination and structural characteristics, which allows the model to refine the final color output with high precision.
3. Experiments
The model was evaluated on two prominent benchmarks:
- DAVIS: A dataset featuring diverse moving objects and complex motion, originally designed for video segmentation.
- Videvo: A dataset containing longer video clips (approximately 300 frames each), which provides a rigorous test for long-term temporal consistency.
4. Results and Discussion
ST-HVC was compared against several state-of-the-art (SOTA) methods, including image-based models (e.g., CIC, IDC) and video-based models (e.g., FAVC, TCVC).
4.1 Quantitative Performance
On the Videvo dataset, ST-HVC achieved a PSNR of 26.95 and an SSIM of 0.9623. Notably, the model achieved a Warp Error of 0.02612, the lowest among all tested models. Warp Error is a critical metric for video colorization as it directly measures the degree of temporal flickering.
4.2 Qualitative Analysis
Visual evaluations indicate that ST-HVC excels at coloring small, high-contrast details—such as the red beak of a black swan—while maintaining deep saturation in moving regions. Other models often produce "undersaturated" patches or gray artifacts in areas of high motion.
4.3 Ablation Study
Ablation experiments highlighted the importance of the histogram guidance. Removing this component resulted in a PSNR drop of approximately 1.7 dB, confirming that the global color hint is essential for high-quality reconstruction and consistency.
5. Conclusion and Future Work
ST-HVC represents a significant advancement in automated video colorization by combining global histogram guidance with sophisticated spatial-temporal attention. While the model performs exceptionally on continuous clips, it faces challenges during rapid scene changes. If a video contains a sudden cut to a different environment, the middle-frame histogram may no longer be representative. Future research will explore "scene-aware" mechanisms to dynamically update color guidance based on detected transitions.
Summary updated