Learning Binary Residual Representations for Domain-specific Video Streaming

Yi-Hsuan Tsai, Ming-Yu Liu, Deqing Sun, Ming-Hsuan Yang, Jan Kautz

Introduction

Modern video streaming relies heavily on universal compression standards like H.264 or HEVC. While these are excellent general-purpose tools, they often struggle at low bitrates, introducing noticeable blocking and blurring artifacts. These artifacts stem from the "residual"—the difference between the original and compressed video—which contains complex, non-linear patterns that traditional linear predictive coding cannot efficiently capture.

However, many streaming applications are strictly domain-specific. For instance, cloud gaming services like GeForce Now involve users playing specific titles for extended periods, and dashcam footage consists almost entirely of street scenes. This paper explores whether knowing the video domain in advance allows for the use of deep learning to compress this residual information more effectively, thereby delivering higher quality over the same bandwidth.

Proposed Framework

The authors propose a hybrid pipeline that integrates the temporal efficiency of H.264 with the non-linear modeling capabilities of deep neural networks.

The proposed hybrid video compression pipeline combining H.264 with a deep-learning residual module.

The workflow consists of several key stages:

  1. Standard Compression: The video is initially compressed using H.264 at a bitrate slightly lower than the total available bandwidth.
  2. Residual Calculation: The system computes the residual frame by subtracting the H.264-compressed video from the original.
  3. Binary Residual Autoencoder: A convolutional autoencoder processes this residual. A critical component is the binarizer layer, which converts continuous feature maps into discrete binary values $\{-1, 1\}$. This allows for highly efficient lossless compression via Huffman coding. To train this non-differentiable step, a straight-through estimator is employed during backpropagation.
  4. Transmission and Reconstruction: The binary map is sent alongside the H.264 stream (e.g., within metadata). On the client side, a lightweight neural decoder reconstructs the residual, which is then added back to the decoded H.264 frame to produce the final high-quality output.

Experimental Results

The pipeline was evaluated on datasets representing two distinct domains: street-view driving (KITTI) and popular video games (Assassin's Creed, Skyrim, and Borderlands).

PSNR performance comparison on the Skyrim dataset across various bitrates.

The proposed framework consistently outperformed both standard H.264 and state-of-the-art artifact-removal networks across all tested bitrates. Key findings include:

  • PSNR Gains: At a 5Mbps bandwidth, the method achieved an average PSNR of 33.26dB, representing a 1.71dB improvement over H.264 and a 0.84dB improvement over artifact-removal networks.
  • SSIM Improvements: At lower bitrates (2Mbps), the pipeline demonstrated a 5.3% improvement in SSIM compared to H.264.
  • Real-time Performance: The client-side decoder is highly optimized, processing frames in just 0.001 seconds on a Titan X GPU. This speed is twice that of competing artifact-removal networks, making it ideal for low-latency applications.

Key Insights and Limitations

A major takeaway is that a hybrid approach is superior to a pure deep-learning codec. H.264 remains highly optimized for temporal motion, while deep learning is best utilized for correcting the specific non-linear errors H.264 leaves behind. Furthermore, binarization is essential; without it, the latent representations would be too large for practical transmission. By pre-transmitting the lightweight decoder weights once (e.g., during a game's loading screen), the system maintains low latency during actual playback.

Despite its advantages, the system has limitations. It is domain-dependent, requiring a specific autoencoder to be trained for each individual game or scenario. Additionally, there is a small initial overhead as the client must download decoder weights and Huffman tables before streaming begins. Future work may focus on optimizing these networks for mobile devices and exploring temporal dependencies within the residual autoencoder.

Summary updated

BibTeX
@inproceedings{tsai2018learning,
  author = {Yi-Hsuan Tsai and Ming-Yu Liu and Deqing Sun and Ming-Hsuan Yang and Jan Kautz},
  title = {{Learning Binary Residual Representations for Domain-specific Video Streaming}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12259},
}
Figure