SR-CL-DMC: P-frame coding with Super-Resolution, Color Learning, and Deep Motion Compensation

Man M. Ho, Jinjia Zhou, Gang He, Muchen Li, Lei LiWorkshops @ CVPR 2020 · DOI

Abstract

SR-CL-DMC is a novel video compression framework designed to achieve ultra-high compression ratios for P-frames by combining traditional codecs with deep learning. By stripping color and down-sampling at the encoder, then employing "Super-Resolution with Color Learning" (SR-CL) and "Deep Motion Compensation" (DMC) at the decoder, the system restores high-quality, full-color video from a fraction of the original data. The framework achieves a 64:1 compression ratio while maintaining competitive visual fidelity.

Introduction

In video compression, P-frames (predictive frames) are essential for efficiency as they store only the changes relative to a previous reference frame. However, at extremely low bitrates, standard codecs like H.264 or the newer Versatile Video Coding (VVC) produce significant artifacts, loss of fine detail, and poor color accuracy. SR-CL-DMC addresses this by drastically reducing the data sent over the wire and tasking a "smart" decoder with reconstructing missing color and resolution by leveraging temporal information from the previous frame.

Methodology

The SR-CL-DMC framework splits the compression task between a standard codec and a custom deep neural network, as illustrated in the end-to-end pipeline below.

Figure 1: Overview of the SR-CL-DMC encoding and decoding pipeline.

The Encoder (Data Reduction)

To achieve a massive 64:1 compression ratio, the encoder performs aggressive data reduction:

  1. Grayscale Conversion: The input frame is converted to grayscale, removing two chrominance components.
  2. Down-sampling: The frame is down-sampled by 2x to a lower resolution.
  3. Residual Encoding: The system calculates the residual (difference) between this low-res grayscale frame and the reference frame. This minimal residual is then encoded using the VVC codec in a "Low Delay P" configuration.

The Decoder (Deep Reconstruction)

The decoder uses a deep neural network to reverse the encoder's reductions:

  • SR-CL (Super-Resolution with Color Learning): A two-stage network that first restores color and removes compression artifacts at low resolution, then performs super-resolution to return the frame to its original dimensions.
  • DMC (Deep Motion Compensation): To handle inter-frame movement, the authors utilize a recursive-feedback architecture for optical flow estimation. This uses Dilated Inception Blocks, allowing the network to capture multiple scales of motion simultaneously without a massive increase in parameters.
  • Refinement: A feedback loop iteratively sharpens the predicted frame to ensure temporal consistency.

Experimental Results

The model was trained on 80,000 high-quality frame pairs from the User Generated Content (UGC) dataset, filtered for meaningful motion and scene consistency. Evaluation was conducted on the CLIC 2020 P-frame track.

Key Performance Metrics

  • Compression Ratio: Successfully achieved a 64:1 ratio.
  • Objective Quality: Reached 41.81 dB PSNR and 0.9959 MS-SSIM on the validation set.
  • Model Efficiency: The decoder size is approximately 4.7 MB, which is 22.5x smaller than the largest competitors in the CLIC challenge.

Qualitative Analysis

The framework excels at synthesizing complex details and accurate colors even in challenging scenarios, such as occluded areas where new objects appear in the target frame.

Figure 6: Qualitative results comparing the reference and target frames with the SR-CL-DMC reconstruction.

Conclusion and Future Work

SR-CL-DMC demonstrates that learning-based decoders can significantly offset aggressive data reduction at the encoder. While the model is highly weight-efficient, the computational intensity of recursive feedback loops and optical flow estimation remains a challenge for real-time mobile playback. Future work will focus on improving transitions during abrupt scene changes and reducing the system's sensitivity to reference frame quality to prevent error propagation.

Summary updated

BibTeX
@inproceedings{ho2020sr,
  author = {Man M. Ho and Jinjia Zhou and Gang He and Muchen Li and Lei Li},
  title = {{SR-CL-DMC: P-frame coding with Super-Resolution, Color Learning, and Deep Motion Compensation}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00070},
}
Figure