Convolutional Block Design for Learned Fractional Downsampling

Li-Heng Chen, Christos G. Bampis, Zhi Li, Chao Chen, Alan C. Bovik

TL;DR

Researchers from UT Austin and Netflix have developed a new "conv-resize" convolutional block that enables neural networks to perform fractional downsampling (e.g., resizing a video by 1.5x). By replacing standard integer-only strided convolutions with a combination of a stride-1 convolution and a differentiable resizer, they created a deep learning-based downsampler that significantly improves video compression efficiency for adaptive streaming services.

What Problem Does This Paper Address?

In the world of adaptive bitrate (ABR) streaming—the technology behind Netflix and YouTube—video is stored at multiple resolutions (1080p, 720p, 540p, etc.) to accommodate different internet speeds. As shown in the general flow diagram below, a source video must be downsampled before encoding.

General flow of adaptive video streaming. The blue box indicates where the proposed downsampling block is applied.

While Convolutional Neural Networks (CNNs) are state-of-the-art for image processing, they have a major limitation: their standard downsampling layers (strided convolutions or pooling) only support integer scaling factors (e.g., 2x, 3x). However, many industry-standard resolutions require fractional scaling. For example, converting 1080p to 720p requires a scaling factor of $M=1.5$.

Until now, engineers had to rely on "legacy" signal processing algorithms like Lanczos for these fractional steps, which aren't optimized for the specific content or the subsequent encoding process.

Methods

The authors propose a simple yet elegant architectural change. Instead of using a single strided convolution layer to handle both feature extraction and resolution reduction, they split these tasks into a two-step "conv-resize" block:

  1. Convolutional Layer ($s=1$): A standard learnable convolution that preserves the input resolution.
  2. Differentiable Resizer: A bilinear resizing operation that can scale the image by any rational factor $M \in \mathbb{Q}_+$.
Comparison of downsampling blocks. (a) and (b) are standard integer-only methods. (c) is the proposed fractional "conv-resize" block.

By making the resizer differentiable, the entire network can be trained end-to-end using backpropagation. They integrated this block into a 10-layer residual network (CNN-CR) as the backbone for their downsampler.

To train the system for video streaming, they used a framework where the network learns to downsample an image such that, after it is upscaled back to the original resolution on the "client side," the loss (Mean Squared Error) is minimized.

Datasets

  • Training: The DIV2K dataset, which contains 1,000 high-quality 2K resolution images.
  • Evaluation: 20 test videos (1080p) from the Xiph Video Test Media and 25 1080p video sources from the Netflix library to ensure a diverse range of realistic content.

Key Insights and Ideas

  • Anti-Aliasing by Design: The authors discovered that the order of operations matters immensely. Placing the learnable convolution before the resizer allows the network to act as an adaptive anti-aliasing filter. This preserves more relevant information before the "decimation" (data loss) of resizing occurs.
  • The "Aha" Moment on Ordering: When they tested the reverse order ("resize-conv"), performance plummeted as the scaling factor $M$ increased. This confirms that neural networks, like classical filters, perform best when they can "prepare" the signal for downsampling.
  • Codec Agnosticism: Even though the training process didn't include a real video encoder (because standard encoders like H.264 aren't differentiable), the learned downsampler still generalized well, providing gains across different upscaling algorithms used during playback.

Key Results

The researchers measured success using BD-rate, a metric that quantifies the bitrate savings for a given level of quality (measured by PSNR, SSIM, and VMAF). A negative percentage indicates better efficiency.

  • Fractional Success: For the common $M=1.5$ (1080p to 720p) scenario, the proposed model achieved a -4.06% BD-rate (PSNR) improvement over the standard Lanczos algorithm when using bilinear upsampling.
  • Competitive at Integer Scales: At integer scales where standard CNNs can operate (like $M=2$ or $M=3$), the proposed fractional block performed just as well as, and often better than, traditional strided CNNs.
  • Perceptual Gains: The model showed significant improvements in VMAF (a perceptual quality metric developed by Netflix), reaching up to -24.17% BD-rate at $M=5$ for bilinear upsampling.

Limitations and Future Work

  • Encoder Modeling: The current training pipeline skips the actual video encoding step because it is non-differentiable. The authors note that at very high scaling factors (like $M=5$), the lack of "awareness" of encoding artifacts might lead to suboptimal results.
  • Future Directions: The team plans to extend this fractional design to other areas, such as developing better quality assessment metrics for images viewed at different scales or distances.

Reference: Chen, L. H., et al. "Convolutional Block Design for Learned Fractional Downsampling." arXiv:2105.09999v1, 2021.

Summary updated

Abstract

The layers of convolutional neural networks (CNNs) can be used to alter the resolution of their inputs, but the scaling factors are limited to integer values. However, in many image and video processing applications, the ability to resize by a fractional factor would be advantageous. One example is conversion between resolutions standardized for video compression, such as from 1080p to 720p. To solve this problem, we propose an alternative building block, formulated as a conventional convolutional layer followed by a differentiable resizer. More concretely, the convolutional layer preserves the resolution of the input, while the resizing operation is fully handled by the resizer. In this way, any CNN architecture can be adapted for non-integer resizing. As an application, we replace the resizing convolutional layer of a modern deep downsampling model by the proposed building block, and apply it to an adaptive bitrate video streaming scenario. Our experimental results show that an improvement in coding efficiency over the conventional Lanczos algorithm is attained, in terms of PSNR, SSIM, and VMAF on test videos.
BibTeX
@inproceedings{chen2022convolutional,
  author = {Li-Heng Chen and Christos G. Bampis and Zhi Li and Chao Chen and Alan C. Bovik},
  title = {{Convolutional Block Design for Learned Fractional Downsampling}},
  booktitle = {IEEECONF},
  year = {2022},
  url = {https://openreview.net/forum?id=oLeRUPrUe0},
}

Loading related content...

Figure