Characterizing and Taming Resolution in Convolutional Neural Networks

Eddie Q. Yan, Liang Luo, Luis Ceze

TL;DR

This paper tackles the significant but often overlooked impact of image resolution on the efficiency and accuracy of convolutional neural networks (CNNs). The authors propose a dynamic resolution approach that intelligently selects the optimal input resolution for each image during inference, leading to substantial improvements in latency and data storage/bandwidth without sacrificing accuracy.

What Problem Does This Paper Address?

Modern computer vision models typically operate at a fixed input resolution. While this resolution is often chosen to maximize accuracy, it comes with substantial costs: higher computational demands (scaling quadratically with resolution), increased storage requirements, and greater bandwidth usage. These costs become particularly problematic in large-scale inference systems. The problem is that choosing the "right" resolution isn't straightforward; it's tightly coupled with other factors like image crop size and the level of detail (image quality), and the optimal choice is highly dataset-dependent. Existing solutions often involve fine-tuning models for specific object scales, which assumes a known and fixed distribution of scales at test time, a scenario not always realistic. This paper aims to bridge this gap by providing a more flexible and efficient way to handle image resolution.

Methods

The core of the paper's contribution lies in a dynamic resolution mechanism. This mechanism involves a two-stage pipeline:

  1. Scale Model: A lightweight CNN (e.g., MobileNet) is trained to predict the appropriate inference resolution for an input image. This model uses a multi-label classification objective to estimate the likelihood of a given resolution yielding a correct prediction from a larger "backbone" model. This allows for coarse-grained decisions about resolution without significant computational overhead.
  2. Backbone Model: This is the main CNN responsible for the actual computer vision task (e.g., classification). It's trained independently and can handle different input resolutions.

To further optimize efficiency, the authors also employ:

  • Progressive Image Encoding: They leverage progressive JPEG, which allows for reading image data in stages, enabling the selection of image detail based on the chosen resolution and quality requirements.
  • Storage Calibration: Using image quality metrics like SSIM, they determine the minimum amount of image data (scans) needed to maintain a target accuracy at a given resolution, thereby reducing storage and bandwidth.
  • Operator Autotuning: They use automatic tensor program optimization techniques to generate specialized, high-performance kernels for convolution operations at various resolutions. This addresses the issue of hardware underutilization when running at non-standard resolutions.

Datasets

The paper uses the following datasets for training and evaluation:

  • ImageNet [24]: A large-scale dataset commonly used for image classification.
  • Stanford Cars [15]: A dataset focused on fine-grained categorization of car models.

The authors note that these datasets differ in resolution distribution and image characteristics, which helps demonstrate the generality of their approach.

Key Insights and Ideas

  • Resolution is a Tightly Coupled Hyperparameter: The choice of image resolution isn't isolated; it directly influences crop size, image detail, and the efficiency of compute kernels.
  • Dynamic Resolution Outperforms Static: Instead of fixing a resolution, dynamically choosing it per image can achieve a better balance between accuracy and efficiency.
  • Scale Model is Efficient and Effective: A small, specialized model can accurately predict optimal resolutions, adding minimal overhead to the overall inference pipeline.
  • Hardware Optimization is Crucial: Standard libraries often don't optimize for all resolutions, leading to underutilization. Autotuning kernels is essential to realize the full benefits of dynamic resolution.
  • Data Savings are Significant: By intelligently selecting image detail and resolution, substantial reductions in data access (up to 20-30%) are achievable without accuracy loss.
  • Scale Equivariance is Key: CNNs' lack of scale equivariance makes them sensitive to object scales, and dynamic resolution can help mitigate this by adapting to different crop sizes.

Key Results

  • Inference Latency Improvement: The dynamic resolution approach improves inference latency by 1.2x to 1.7x.
  • Data Access Reduction: Data access volume is reduced by up to 20-30% without affecting accuracy.
  • Accuracy Preservation: The dynamic resolution approach maintains accuracy comparable to static approaches, often achieving near-optimal accuracy across various crop sizes and resolutions.
  • Operator Autotuning Benefits: Tuned inference at 280x280 is 1.2x to 1.7x faster than a hardware-specific library implementation at 224x224.
  • Storage Calibration: Achieves significant read savings (e.g., 11.2% on ImageNet, up to 48.8% on Cars) with minimal accuracy loss (<0.05%).

Limitations and Future Work

  • Scale Model Overhead: While minimized, the scale model still introduces some runtime overhead (around 9.7ms on a 4790K). The authors suggest this can be hidden through pipelining or further optimized with autotuning.
  • Image Quality Metric: The use of SSIM is a proxy for perceptual quality. More sophisticated or reference-free metrics could potentially yield even greater bandwidth savings.
  • Scale Model Calibration: The scale model is currently calibrated for accuracy. Future work could involve calibrating it for inference costs (latency) as well.
  • Progressive Encoding Dependency: While the approach is general, the current implementation relies on progressive JPEG. Exploring other progressive encodings could be beneficial.
  • Future Work: The authors suggest calibrating image quality for the scale model itself to further break data read limits and exploring alternative loss functions for the scale model that incorporate inference costs.

Summary updated

Abstract

Image resolution has a significant effect on the accuracy and computational, storage, and bandwidth costs of computer vision model inference. These costs are exacerbated when scaling out models to large inference serving systems and make image resolution an attractive target for optimization. However, the choice of resolution inherently introduces additional tightly coupled choices, such as image crop size, image detail, and compute kernel implementation that impact computational, storage, and bandwidth costs. Further complicating this setting, the optimal choices from the perspective of these metrics are highly dependent on the dataset and problem scenario. We characterize this tradeoff space, quantitatively studying the accuracy and efficiency tradeoff via systematic and automated tuning of image resolution, image quality and convolutional neural network operators. With the insights from this study, we propose a dynamic resolution mechanism that removes the need to statically choose a resolution ahead of time. Our evaluation shows that our dynamic resolution approach improves inference latency by 1.2×-1.7×, reduces data access volume by up to 20–30%, without affecting accuracy. We establish the dynamic resolution approach as a viable alternative to fine-tuning for a specific object scale to compensate for unknown crop sizes, which is the current state of the art.
BibTeX
@inproceedings{yan2021characterizing,
  author = {Eddie Q. Yan and Liang Luo and Luis Ceze},
  title = {{Characterizing and Taming Resolution in Convolutional Neural Networks}},
  booktitle = {IISWC},
  year = {2021},
  url = {https://openreview.net/forum?id=Xg7IImSLq8r},
}
Figure