TL;DR
The PIRM 2018 challenge tackles the computational bottleneck of deploying deep learning-based image enhancement and super-resolution models directly on smartphones. By evaluating submissions on actual mobile hardware using a balanced metric of speed, fidelity, and perceptual quality, the challenge established new state-of-the-art architectures that run up to 8x faster than standard baselines.
What Problem Does This Paper Address?
Most deep learning research for image restoration focuses purely on maximizing quantitative metrics like Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). To win standard competitions, researchers typically build massive, deep networks trained on clusters of power-hungry GPUs.
However, these monstrous models are far too computationally expensive and memory-intensive to run on edge devices like smartphones or tablets. The PIRM 2018 challenge addresses this gap by shifting the focus toward resource-efficient architectures that can deliver high-quality perceptual results under strict mobile runtime and memory constraints.
Methods
The challenge featured two tracks: Track A (4x Image Super-Resolution) and Track B (Real-world Image Enhancement). To achieve mobile efficiency, the top-performing teams relied heavily on processing images at a lower spatial resolution before upscaling them at the very end of the network.
-
TEAM_ALEX (Track A Winner): Introduced a "desubpixel" block that reversibly rearranges spatial features into channels. This reduces spatial dimensions without losing information, allowing the bulk of the 20-block residual network to run efficiently at a lower resolution.
Figure 3: Desubpixel block and the CNN architecture proposed by TEAM ALEX. -
KAIST-VICLAB (Track A Speed Winner): Proposed a highly compact 8-layer CNN utilizing pixel shuffles and an error-feedback scheme. For Track B, they used an encoder-decoder architecture where spatial sizes were reduced using a space-to-batch technique, paired with a Relativistic GAN (RGAN) discriminator during training.
-
Mt.Phoenix (Track B Winner): Utilized a U-Net style architecture augmented with global features calculated via average pooling from its bottleneck layer. They also introduced a global transform layer performing element-wise multiplication to effectively map low-quality mobile photos to DSLR quality.
U-net architecture proposed by Mt.Phoenix
Datasets
- DIV2K: Used for Track A (Super-Resolution), containing 800 diverse high-resolution internet images for training.
- DPED (Digital Photo Enhancement Dataset): Used for Track B (Image Enhancement), featuring thousands of photos taken simultaneously with smartphones and a high-end DSLR. The challenge specifically focused on mapping low-quality photos from an old iPhone 3GS to a Canon 70D DSLR.
Key Insights and Ideas
- Early Downsampling is Essential: Nearly all successful teams avoided operating on full-resolution images throughout the network. Reversible downsampling (like space-to-depth or desubpixel blocks) drastically cuts down floating-point operations (FLOPs) while preserving high-frequency details.
- The Perceptual Metric Disconnect: In the real-world enhancement track, high PSNR scores did not guarantee the best visual results. Furthermore, standard automated metrics like MS-SSIM correlated poorly with human Mean Opinion Scores (MOS), highlighting the ongoing need for better automated perceptual evaluation.
- Human Indistinguishability in SR: For Track A, because the models optimized for standard pixel-wise (MSE/L1) or VGG losses rather than aggressive adversarial losses, human evaluators in a 2,000+ participant user study found the visual outputs of the top models virtually indistinguishable, despite major differences in network architectures.
Key Results
All final submissions significantly outperformed the baseline SRCNN network, running 3x to 8x faster while delivering vastly superior visual quality.
- Track A (Super-Resolution): TEAM_ALEX achieved the highest overall balanced scores across all validation tracks with a PSNR of 28.21 dB. Meanwhile, KAIST-VICLAB proved to be the speed champion, processing HD images in under a second on actual mobile devices (812 ms on a Razer Phone and 985 ms on a Huawei P20).
- Track B (Image Enhancement): Mt.Phoenix won the track by achieving the top human MOS (2.6804) and the fastest smartphone runtimes among the top contenders (1.47 seconds on the Razer Phone).
Limitations and Future Work
- Deployment Framework Constraints: A major hurdle highlighted by the challenge was mobile framework compatibility. For instance, the Rainbow team utilized a specific contrast adjustment operation (
tf.image.adjust_contrast) that was not yet supported by TensorFlow Mobile, preventing their model from running on Android during testing. - Memory Bottlenecks: Peak memory consumption remains a critical issue for mobile deployment. Several high-performing models (such as the EdS submission and the DPED baseline) suffered from "Out of Memory" errors on the Huawei P20 smartphone.
- Future Directions: The organizers emphasize the need for future architectures to better exploit hardware-specific mobile acceleration (like NPUs) and to develop automated perceptual loss functions that align more closely with human judgment without requiring massive network depths.
Summary updated