TL;DR
DySR is a dynamic super-resolution framework that uses algorithm-system co-design to adapt to fluctuating mobile resources in real-time. By combining an adaptation-aware one-shot Neural Architecture Search (NAS) with a low-overhead incremental model adaptation system, it maintains steady frame rates for video streaming while using 40% less memory than traditional ensemble methods.
What Problem Does This Paper Address?
Deep Learning-based Super-Resolution (SR) is a game-changer for mobile streaming, allowing low-resolution data to be upscaled into high-definition visuals. However, SR models are computationally "heavy." On a mobile device, available CPU and GPU resources are constantly shifting as other apps run in the background.
Existing solutions typically fall into two traps:
1. Static Models: They perform well under ideal conditions but cause massive frame rate drops or stuttering when the device gets busy. As system utilization drops, the static model cannot adapt, leading to a proportional drop in frames per second (FPS).
2. Naive Adaptation: Switching between different-sized models to match available resources usually requires loading new weights from disk. This causes a 1–3 second "freeze" in the video where the FPS drops to zero. Alternatively, keeping all models in RAM (an ensemble) consumes far too much memory for a mobile device.
The authors tackle the challenge of how to dynamically adapt an SR model to available resources within milliseconds without sacrificing memory or quality.
Methods
The authors propose a two-pronged "Co-design" approach to solve the adaptation bottleneck:
1. Algorithm: Adaptation-aware One-Shot NAS
Instead of searching for a single "best" model, DySR uses Neural Architecture Search to build a Meta-Graph. This meta-graph contains many possible "sub-graphs" (smaller models). Crucially, these sub-graphs are trained to share weights for common operations.
- The Goal: Maximize image quality (PSNR) and the amount of shared operations while staying above a minimum frame rate (FPS) threshold.
- The Architecture: A cell-based design incorporating efficient blocks from well-known models like CARN, RCAB, and WDSR.
2. System: Incremental Model Adaptation
To avoid the "loading freeze," DySR keeps the entire meta-graph in memory. Because the sub-graphs share weights, the total memory footprint is much smaller than maintaining separate models.
- Dynamic Rerouting: During inference, the system monitors resource availability. To "switch" models, it simply changes the pointers (variable references) that define the execution path.
- Zero Disk I/O: Since the weights are already in RAM, the switch happens at a per-batch granularity with near-zero overhead.
Datasets
The framework was trained and evaluated on standard SR benchmarks:
- Training: Div2K (images) and Vimeo90k (video).
- Evaluation: Urban100, Set5, and Vimeo90k.
- Preprocessing: Images were downsampled using bi-cubic interpolation at 2x and 4x scales.
Key Insights and Ideas
- The Weight-Sharing Synergy: The authors discovered that the number of total operations in the meta-graph and the number of path re-routes are inversely proportional to the number of shared operations. By maximizing sharing, they simultaneously reduced memory usage and adaptation time.
- Pareto-Frontier Awareness: DySR doesn't just find one efficient model; it finds a family of models that represent the best possible trade-off between speed and quality across a wide spectrum of hardware (from a budget Snapdragon 855 to a server-grade A100).
- System-Level Agility: Treating a neural network as a mutable graph where execution paths can be rerouted on-the-fly is a significant departure from the "static graph" approach used in most deployment frameworks.
Key Results
- Memory Efficiency: DySR achieves a memory footprint approximately 40% smaller than an assembled baseline of 10 individual models.
- Steady Throughput: In "bursty" resource scenarios (where background tasks suddenly spike), DySR maintained a steady frame rate by instantly downshifting to a smaller sub-graph, whereas static models saw their FPS plummet.
- Competitive Quality: Despite the constraints of weight sharing, DySR sub-graphs remain close to the Pareto frontier, matching or exceeding the performance of state-of-the-art static models (like FALSR or CARN) at similar speeds.
Limitations and Future Work
- Search Complexity: The architecture search space is massive ($5.8 \times 10^{23}$ possible networks), requiring significant offline training time (21 GPU days in their setup).
- Metric Focus: The evaluation primarily uses PSNR and SSIM. While standard, these don't always capture "perceptual" quality as well as GAN-based metrics.
- Resource Types: The current work focuses on CPU/GPU utilization. Future iterations could incorporate battery level or thermal throttling states as triggers for model adaptation.
Summary updated