DySR: Adaptive Super-Resolution via Algorithm and System Co-design

Syed Zawad, Cheng Li, Zhewei Yao, Elton Zheng, Yuxiong He, Feng YanICLR 2023 · OpenReview

TL;DR

DySR is a dynamic super-resolution framework that uses algorithm-system co-design to adapt to fluctuating mobile resources in real-time. By combining an adaptation-aware one-shot Neural Architecture Search (NAS) with a low-overhead incremental model adaptation system, it maintains steady frame rates for video streaming while using 40% less memory than traditional ensemble methods.

What Problem Does This Paper Address?

Deep Learning-based Super-Resolution (SR) is a game-changer for mobile streaming, allowing low-resolution data to be upscaled into high-definition visuals. However, SR models are computationally "heavy." On a mobile device, available CPU and GPU resources are constantly shifting as other apps run in the background.

Existing solutions typically fall into two traps:

1. Static Models: They perform well under ideal conditions but cause massive frame rate drops or stuttering when the device gets busy. As system utilization drops, the static model cannot adapt, leading to a proportional drop in frames per second (FPS).

FPS degradation of static models as system utilization drops across different hardware.

2. Naive Adaptation: Switching between different-sized models to match available resources usually requires loading new weights from disk. This causes a 1–3 second "freeze" in the video where the FPS drops to zero. Alternatively, keeping all models in RAM (an ensemble) consumes far too much memory for a mobile device.

The "freeze" effect: FPS drops to zero during model reloading across various devices, alongside the memory cost of loading multiple models.

The authors tackle the challenge of how to dynamically adapt an SR model to available resources within milliseconds without sacrificing memory or quality.

Methods

The authors propose a two-pronged "Co-design" approach to solve the adaptation bottleneck:

1. Algorithm: Adaptation-aware One-Shot NAS

Instead of searching for a single "best" model, DySR uses Neural Architecture Search to build a Meta-Graph. This meta-graph contains many possible "sub-graphs" (smaller models). Crucially, these sub-graphs are trained to share weights for common operations.

  • The Goal: Maximize image quality (PSNR) and the amount of shared operations while staying above a minimum frame rate (FPS) threshold.
  • The Architecture: A cell-based design incorporating efficient blocks from well-known models like CARN, RCAB, and WDSR.

2. System: Incremental Model Adaptation

To avoid the "loading freeze," DySR keeps the entire meta-graph in memory. Because the sub-graphs share weights, the total memory footprint is much smaller than maintaining separate models.

  • Dynamic Rerouting: During inference, the system monitors resource availability. To "switch" models, it simply changes the pointers (variable references) that define the execution path.
  • Zero Disk I/O: Since the weights are already in RAM, the switch happens at a per-batch granularity with near-zero overhead.

Datasets

The framework was trained and evaluated on standard SR benchmarks:

  • Training: Div2K (images) and Vimeo90k (video).
  • Evaluation: Urban100, Set5, and Vimeo90k.
  • Preprocessing: Images were downsampled using bi-cubic interpolation at 2x and 4x scales.

Key Insights and Ideas

  • The Weight-Sharing Synergy: The authors discovered that the number of total operations in the meta-graph and the number of path re-routes are inversely proportional to the number of shared operations. By maximizing sharing, they simultaneously reduced memory usage and adaptation time.
  • Pareto-Frontier Awareness: DySR doesn't just find one efficient model; it finds a family of models that represent the best possible trade-off between speed and quality across a wide spectrum of hardware (from a budget Snapdragon 855 to a server-grade A100).
  • System-Level Agility: Treating a neural network as a mutable graph where execution paths can be rerouted on-the-fly is a significant departure from the "static graph" approach used in most deployment frameworks.

Key Results

  • Memory Efficiency: DySR achieves a memory footprint approximately 40% smaller than an assembled baseline of 10 individual models.
  • Steady Throughput: In "bursty" resource scenarios (where background tasks suddenly spike), DySR maintained a steady frame rate by instantly downshifting to a smaller sub-graph, whereas static models saw their FPS plummet.
  • Competitive Quality: Despite the constraints of weight sharing, DySR sub-graphs remain close to the Pareto frontier, matching or exceeding the performance of state-of-the-art static models (like FALSR or CARN) at similar speeds.

Limitations and Future Work

  • Search Complexity: The architecture search space is massive ($5.8 \times 10^{23}$ possible networks), requiring significant offline training time (21 GPU days in their setup).
  • Metric Focus: The evaluation primarily uses PSNR and SSIM. While standard, these don't always capture "perceptual" quality as well as GAN-based metrics.
  • Resource Types: The current work focuses on CPU/GPU utilization. Future iterations could incorporate battery level or thermal throttling states as triggers for model adaptation.

Summary updated

Abstract

Super resolution (SR) is a promising approach for improving the quality of low resolution steaming services on mobile devices. On mobile devices, the available computing and memory resources change dynamically depending on other running applications. Due to the high computation and memory demands of SR models, it is essential to adapt the model according to available resources to harvest the best possible model performance while maintaining quality of service (QoS), such as meeting a minimum framerate and avoiding interruptions. Nevertheless, there is no SR model or machine learning system that supports adaptive SR, and enabling adaptive SR model on mobile devices is challenging because adapting model can cause significant framerate drop or even service interruption. To address this challenge, we take an algorithm and system co-design approach and propose DySR that maintains QoS while maximizing the model performance. During the training stage, DySR employs an adaption-aware one-shot Neural Architecture Search to produce sub-graphs that share kernel operation weights for low model adaption overhead while striking a balance between performance and framerate. During the inference stage, an incremental model adaption method is developed for further reducing the model adaption overhead. We evaluate on a diverse set of hardware and datasets to show that DySR can generate models close to the Pareto frontier while maintaining a steady framerate throughput with a memory footprint of around 40\% less compared to baseline methods.
BibTeX
@inproceedings{zawad2023dysr,
  author = {Syed Zawad and Cheng Li and Zhewei Yao and Elton Zheng and Yuxiong He and Feng Yan},
  title = {{DySR: Adaptive Super-Resolution via Algorithm and System Co-design}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Pgtn4l6eKjv},
}
Figure