Convolutional Block Design for Learned Fractional Downsampling

Li-Heng Chen, Christos G. Bampis, Zhi Li, Chao Chen, Alan C. Bovik

Abstract

The layers of convolutional neural networks (CNNs) can be used to alter the resolution of their inputs, but the scaling factors are limited to integer values. However, in many image and video processing applications, the ability to resize by a fractional factor would be advantageous. One example is conversion between resolutions standardized for video compression, such as from 1080p to 720p. To solve this problem, we propose an alternative building block, formulated as a conventional convolutional layer followed by a differentiable resizer. More concretely, the convolutional layer preserves the resolution of the input, while the resizing operation is fully handled by the resizer. In this way, any CNN architecture can be adapted for non-integer resizing. As an application, we replace the resizing convolutional layer of a modern deep downsampling model by the proposed building block, and apply it to an adaptive bitrate video streaming scenario. Our experimental results show that an improvement in coding efficiency over the conventional Lanczos algorithm is attained, in terms of PSNR, SSIM, and VMAF on test videos.

Introduction

Adaptive Bitrate (ABR) streaming is the backbone of modern video platforms like Netflix and YouTube. To ensure smooth playback across varying network conditions, source videos are encoded and stored at multiple resolutions (e.g., 1080p, 720p, 540p). This process requires downsampling the source video before it enters the encoder.

General flow of adaptive video streaming showing the role of downsampling.

While Convolutional Neural Networks (CNNs) have revolutionized image processing, they face a significant hurdle in this domain: standard downsampling layers, such as strided convolutions or pooling, are mathematically constrained to integer scaling factors (e.g., 2x or 3x). However, industry-standard resolution transitions often require fractional scaling. For instance, downsampling from 1080p to 720p requires a scaling factor of $M=1.5$. Historically, this has forced a reliance on legacy signal processing algorithms like Lanczos, which are not optimized for specific content or modern neural-based upscaling.

Proposed Architecture: The Conv-Resize Block

To bridge the gap between deep learning and fractional scaling, we propose a "conv-resize" block. Instead of a single layer attempting to perform both feature extraction and decimation, we decouple these tasks into two distinct components:

  1. Convolutional Layer ($s=1$): A learnable layer that performs feature extraction while maintaining the original input resolution.
  2. Differentiable Resizer: A bilinear resizing operation that can scale the feature maps by any rational factor $M \in \mathbb{Q}_+$.
Comparison of downsampling blocks: (a) strided convolution, (b) pooling, and (c) the proposed conv-resize block.

By ensuring the resizer is differentiable, the entire block can be integrated into larger architectures and trained end-to-end via backpropagation. We integrated this block into a 10-layer residual network (CNN-CR) to serve as a high-performance downsampler.

A critical insight from our design is the ordering of operations. By placing the learnable convolution before the resizer, the network can function as an adaptive anti-aliasing filter. This allows the model to intelligently preserve information that would otherwise be lost during the decimation process. Our tests showed that reversing this order ("resize-conv") led to significantly degraded performance as the scaling factor increased.

Experimental Setup and Training

The model was trained using the DIV2K dataset, which consists of 1,000 high-quality 2K images. The training objective was to minimize the Mean Squared Error (MSE) between the original source and the version that had been downsampled by our network and subsequently upscaled back to the original resolution.

For evaluation, we utilized 20 test videos from the Xiph Video Test Media and 25 1080p sources from the Netflix library. This diversity ensured the model was tested against a wide variety of realistic textures and motion patterns.

Key Results

We measured performance using BD-rate, which calculates the bitrate savings required to maintain a specific quality level. A negative BD-rate indicates improved efficiency.

  • Fractional Scaling Performance: In the $M=1.5$ (1080p to 720p) scenario, our model achieved a -4.06% BD-rate (PSNR) improvement over the standard Lanczos algorithm when using bilinear upsampling.
  • Integer Scale Parity: At integer scales (e.g., $M=2$), where standard strided CNNs are applicable, our fractional block performed as well as or better than traditional strided architectures.
  • Perceptual Quality: The gains were even more pronounced when measured by VMAF (Video Multi-Method Assessment Fusion), a perceptual metric. At a scaling factor of $M=5$, we observed up to a -24.17% BD-rate improvement, suggesting that the learned downsampler preserves details that are highly relevant to human visual perception.

Conclusion and Future Work

The proposed conv-resize block enables CNNs to handle the fractional scaling requirements of modern video pipelines without sacrificing the benefits of end-to-end learning. While the current framework does not explicitly model the non-differentiable video encoding step (e.g., H.264/AVC), the results demonstrate that the learned downsampler generalizes remarkably well to standard compression workflows. Future research will focus on incorporating differentiable encoder proxies to further optimize performance at extreme scaling factors and applying these fractional designs to advanced quality assessment metrics.

Summary updated

Abstract

The layers of convolutional neural networks (CNNs) can be used to alter the resolution of their inputs, but the scaling factors are limited to integer values. However, in many image and video processing applications, the ability to resize by a fractional factor would be advantageous. One example is conversion between resolutions standardized for video compression, such as from 1080p to 720p. To solve this problem, we propose an alternative building block, formulated as a conventional convolutional layer followed by a differentiable resizer. More concretely, the convolutional layer preserves the resolution of the input, while the resizing operation is fully handled by the resizer. In this way, any CNN architecture can be adapted for non-integer resizing. As an application, we replace the resizing convolutional layer of a modern deep downsampling model by the proposed building block, and apply it to an adaptive bitrate video streaming scenario. Our experimental results show that an improvement in coding efficiency over the conventional Lanczos algorithm is attained, in terms of PSNR, SSIM, and VMAF on test videos.
BibTeX
@inproceedings{chen2022convolutional,
  author = {Li-Heng Chen and Christos G. Bampis and Zhi Li and Chao Chen and Alan C. Bovik},
  title = {{Convolutional Block Design for Learned Fractional Downsampling}},
  booktitle = {IEEECONF},
  year = {2022},
  url = {https://openreview.net/forum?id=oLeRUPrUe0},
}
Figure