SVCNet: Scribble-Based Video Colorization Network With Temporal Aggregation

Yuzhi Zhao, Lai-Man Po, Kangcheng Liu, Xuehui Wang, Wing Yin Yu, Pengfei Xian, Yujia Zhang, Mengyang Liu

Abstract

Video colorization remains a challenging task in computer vision, primarily due to the difficulties in maintaining temporal consistency and preventing color bleeding across object boundaries. We present SVCNet, a novel two-stage framework designed to colorize monochrome videos using sparse user-provided scribbles on the initial frame. By integrating a multi-task colorization-segmentation network with a hybrid temporal aggregation module, SVCNet produces vivid, flicker-free results. Our approach effectively addresses short-range motion through optical flow and long-range dependencies via global correspondence, while an auxiliary segmentation task serves as a regularizer to minimize color leakage. Experimental results on the DAVIS and Videvo datasets demonstrate that SVCNet outperforms state-of-the-art methods in both quantitative metrics and human preference studies.

1. Introduction

Colorizing legacy black-and-white film is a labor-intensive process that traditionally requires significant manual intervention. Existing computational approaches generally fall into two categories: fully automatic methods and reference-based methods. Automatic models often suffer from "color uncertainty," leading to desaturated and dull outputs as the network predicts the most statistically likely (often gray) tones. Reference-based methods, while more vivid, require high-quality exemplar images that closely match the video content, which are not always available.

SVCNet bridges this gap by utilizing sparse user scribbles on the first frame to guide the colorization process. This approach allows for user intent and vividness while maintaining the efficiency of an automated pipeline. The framework specifically targets three technical hurdles: color vividness, temporal consistency (flicker reduction), and color bleeding (leakage across object boundaries).

2. Methodology

SVCNet employs a two-stage architecture designed to separate the initial color propagation from the temporal smoothing process.

2.1 Stage 1: Color Propagation Network (CPNet)

The first stage focuses on translating grayscale frames and user scribbles into a colorized output. CPNet utilizes a Pyramid Feature Encoder to fuse the grayscale luminance channel with the sparse user input. To combat the common issue of color bleeding, the network is trained using multi-task learning. While the primary branch predicts the chrominance channels, a secondary auxiliary branch predicts a segmentation map. This architectural choice forces the encoder to learn sharp object boundaries, effectively "locking" colors within their respective regions.

2.2 Stage 2: Spatiotemporal Smoothing Network (SSNet)

To ensure the video remains flicker-free, SSNet aggregates temporal information using a hybrid approach:

  • Short-range Aggregation: A Refinement Module uses optical flow (estimated via PWCNet) to align the current frame with its immediate neighbors (three frames preceding and following). This handles local motion and ensures smooth transitions between adjacent frames.
  • Long-range Aggregation: A Correspondence Module links the current frame back to the initial colorized frame. This prevents "color drift," where the hues gradually shift or fade over long sequences, by constantly referencing the original user-defined colors.

2.3 Computational Efficiency

To maintain a manageable memory footprint, SVCNet performs the majority of its operations at a fixed low resolution. A dedicated Super-resolution (SR) Module is employed at the end of the pipeline to restore the colorized video to its original high-definition dimensions, allowing the model to handle various input sizes without exponential increases in computational cost.

3. Experiments and Results

3.1 Datasets and Training

The model underwent large-scale pre-training on ImageNet to establish general visual priors. It was subsequently fine-tuned on the DAVIS and Videvo video benchmarks. Because these datasets lack manual segmentation labels for every frame, the authors utilized saliency detection to generate "pseudo-binary" segmentation maps, which provided the necessary supervision for the color-bleeding prevention branch.

3.2 Quantitative and Qualitative Performance

SVCNet was evaluated against several state-of-the-art (SOTA) methods, including both fully automatic and exemplar-based models.

  • Accuracy: SVCNet achieved a PSNR of 25.71 and an SSIM of 0.9565 on the DAVIS dataset, indicating superior pixel-level accuracy and structural preservation.
  • Boundary Integrity: In terms of mean Intersection over Union (mIoU), the segmentation-aware approach significantly reduced color leakage compared to baseline models.
  • User Preference: In a subjective user study, SVCNet was preferred 49% of the time, outperforming the next best method (RUIC + DEVC) by a margin of 8%. Participants specifically highlighted the vividness of the colors and the lack of temporal flickering.

4. Discussion and Conclusion

SVCNet introduces a robust mechanism for video colorization by combining segmentation-based regularization with hybrid temporal aggregation. The use of segmentation as a regularizer proves to be a powerful "aha" moment in the architecture, improving edge awareness even when the segmentation map itself is not the end goal.

However, limitations remain. The model is sensitive to scene switching; because it relies on the first frame's scribbles, a hard cut to a new scene renders the long-range connections irrelevant. Additionally, complex textures and very small objects can still lead to minor blurring. Future research will focus on integrating automatic keyframe detection to prompt for new scribbles during scene transitions and improving the propagation of high-frequency details.

Summary updated

BibTeX
@article{zhao2023svcnet,
  author = {Yuzhi Zhao and Lai-Man Po and Kangcheng Liu and Xuehui Wang and Wing Yin Yu and Pengfei Xian and Yujia Zhang and Mengyang Liu},
  title = {{SVCNet: Scribble-Based Video Colorization Network With Temporal Aggregation}},
  journal = {IEEE Trans. Image Process.},
  year = {2023},
  doi = {10.1109/tip.2023.3298537},
}
Figure