Abstract
Under the limited storage, computing and network bandwidth resources, the video compression coding technology plays an important role for visual communication. To efficiently compress raw video data, a colorization-based video compression coding method is proposed in this paper. In the proposed encoder, only the video luminance components are encoded and transmitted. To restore the video chrominance information, a generative adversarial network (GAN) model is adopted in the proposed decoder. In order to make the GAN work efficiently for video colorization, the generator of the proposed GAN model adopts an optimized MultiResUNet, an attention module, and a mixed loss function. Experimental results show that when compared with the H.265/HEVC video compression coding standard using all-intra coding structure, the proposed video compression coding method achieves an average of 72.05% BDBR reduction, and an average of 4.758 dB BDPSNR increase. Moreover, to our knowledge, this is the first work which compresses videos by using GAN-based colorization, and it provides a new way for addressing the video compression coding problems.
Introduction
As video resolutions and frame rates continue to rise, traditional compression standards like H.264 and H.265 face significant challenges under limited bandwidth and storage constraints. These conventional codecs typically compress both luminance (Y) and chrominance (Cb, Cr) components. However, the human visual system is biologically more sensitive to brightness variations than color changes.
This paper exploits this characteristic by proposing a "compression via colorization" framework. The core idea is to discard color data at the encoder entirely, transmitting only the luminance stream. The decoder then uses a Generative Adversarial Network (GAN) to reconstruct the missing chrominance information, effectively treating video decoding as a supervised image-to-image translation task.
The Proposed Framework
The proposed system architecture splits the workload between a simplified encoder and a sophisticated GAN-based decoder.
At the encoder side, the input color video is preprocessed to extract the luminance component, which is then encoded into a bitstream using standard methods (like HEVC). At the decoder side, the luminance video is reconstructed and passed through a Generator network to produce the final colorized video.
GAN Architecture and Training
The heavy lifting of chrominance restoration is performed by a specialized GAN architecture designed for spatial and perceptual consistency.
Generator: Optimized MultiResUNet
The generator utilizes a MultiResUNet backbone, which is superior to standard U-Nets at capturing multi-scale features. Key enhancements include:
- Global Long Residual Connection (GLRC): This improves feature transfer from encoder to decoder layers, boosting learning stability and objective quality (PSNR).
- Attention Module: A self-attention mechanism captures long-range spatial dependencies, ensuring color consistency across large objects.
Mixed Loss Function
To ensure the colorization is both realistic and accurate, the authors employ a multi-faceted loss function:
- GAN Loss: Drives overall subjective realism.
- MSE Loss: Ensures pixel-level smoothness.
- Content Loss: Uses a pre-trained VGG-19 network for perceptual consistency.
- Optimized Color Loss: Employs Gaussian blurring to compare color distributions, preventing the "dimming" effect often seen in standard MSE-based colorization.
Experimental Results
The model was evaluated using eight standard video conferencing sequences across high (1280x720) and low (352x288) resolutions. The proposed method was compared against the H.265/HEVC (HM 16.0) reference software.
Key Performance Metrics
- Bitrate Efficiency: The method achieved an average 72.05% BDBR reduction, meaning it can match HEVC quality at less than a third of the bitrate.
- Objective Quality: An average increase of 4.758 dB in BDPSNR was observed.
- Subjective Resilience: At high compression levels (e.g., QP37), where standard HEVC results become blocky and desaturated, the GAN-based approach maintains sharp details and vibrant colors by "imagining" the chrominance based on learned scene features.
Conclusion and Future Work
This work represents a paradigm shift in video coding by replacing explicit color transmission with generative reconstruction. While achieving massive bitrate savings, the approach currently focuses on "all-intra" coding. Future research aims to incorporate temporal redundancy to further optimize performance. Additionally, addressing the high computational cost of GAN inference will be crucial for deployment on resource-constrained mobile devices.
Summary updated