PIRM Challenge on Perceptual Image Enhancement on Smartphones: Report

Andrey Ignatov, Radu Timofte, Thang Van Vu, Tung Minh Luu, Trung X. Pham, Cao Van Nguyen, Yongwoo Kim, Jae-Seok Choi, Munchurl Kim, Jie Huang, Jiewen Ran, Chen Xing, Xingguang Zhou, Pengfei Zhu, Mingrui Geng, Yawei Li, Eirikur Agustsson, Shuhang Gu, Luc Van Gool, Etienne de Stoutz, Nikolay Kobyshev, Kehui Nie, Yan Zhao, Gen Li, Tong Tong, Qinquan Gao, Hanwen Liu, Pablo Navarrete Michelini, Dan Zhu, Hu Fengshuo, Zheng Hui, Xiumei Wang, Lirui Deng, Rang Meng, Jinghui Qin, Yukai Shi, Wushao Wen, Liang Lin, Ruicheng Feng, Shixiang Wu, Chao Dong, Yu Qiao, Subeesh Vasu, Thekke Madam Nimisha, Praveen Kandula, A. N. Rajagopalan, Jie Liu, Cheolkon Jung

TL;DR

The PIRM 2018 challenge tackles the computational bottleneck of deploying deep learning-based image enhancement and super-resolution models directly on smartphones. By evaluating submissions on actual mobile hardware using a balanced metric of speed, fidelity, and perceptual quality, the challenge established new state-of-the-art architectures that run up to 8x faster than standard baselines.

What Problem Does This Paper Address?

Most deep learning research for image restoration focuses purely on maximizing quantitative metrics like Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). To win standard competitions, researchers typically build massive, deep networks trained on clusters of power-hungry GPUs.

However, these monstrous models are far too computationally expensive and memory-intensive to run on edge devices like smartphones or tablets. The PIRM 2018 challenge addresses this gap by shifting the focus toward resource-efficient architectures that can deliver high-quality perceptual results under strict mobile runtime and memory constraints.

Methods

The challenge featured two tracks: Track A (4x Image Super-Resolution) and Track B (Real-world Image Enhancement). To achieve mobile efficiency, the top-performing teams relied heavily on processing images at a lower spatial resolution before upscaling them at the very end of the network.

  • TEAM_ALEX (Track A Winner): Introduced a "desubpixel" block that reversibly rearranges spatial features into channels. This reduces spatial dimensions without losing information, allowing the bulk of the 20-block residual network to run efficiently at a lower resolution.

    Figure 3: Desubpixel block and the CNN architecture proposed by TEAM ALEX.
  • KAIST-VICLAB (Track A Speed Winner): Proposed a highly compact 8-layer CNN utilizing pixel shuffles and an error-feedback scheme. For Track B, they used an encoder-decoder architecture where spatial sizes were reduced using a space-to-batch technique, paired with a Relativistic GAN (RGAN) discriminator during training.

  • Mt.Phoenix (Track B Winner): Utilized a U-Net style architecture augmented with global features calculated via average pooling from its bottleneck layer. They also introduced a global transform layer performing element-wise multiplication to effectively map low-quality mobile photos to DSLR quality.

    U-net architecture proposed by Mt.Phoenix

Datasets

  • DIV2K: Used for Track A (Super-Resolution), containing 800 diverse high-resolution internet images for training.
  • DPED (Digital Photo Enhancement Dataset): Used for Track B (Image Enhancement), featuring thousands of photos taken simultaneously with smartphones and a high-end DSLR. The challenge specifically focused on mapping low-quality photos from an old iPhone 3GS to a Canon 70D DSLR.

Key Insights and Ideas

  • Early Downsampling is Essential: Nearly all successful teams avoided operating on full-resolution images throughout the network. Reversible downsampling (like space-to-depth or desubpixel blocks) drastically cuts down floating-point operations (FLOPs) while preserving high-frequency details.
  • The Perceptual Metric Disconnect: In the real-world enhancement track, high PSNR scores did not guarantee the best visual results. Furthermore, standard automated metrics like MS-SSIM correlated poorly with human Mean Opinion Scores (MOS), highlighting the ongoing need for better automated perceptual evaluation.
  • Human Indistinguishability in SR: For Track A, because the models optimized for standard pixel-wise (MSE/L1) or VGG losses rather than aggressive adversarial losses, human evaluators in a 2,000+ participant user study found the visual outputs of the top models virtually indistinguishable, despite major differences in network architectures.

Key Results

All final submissions significantly outperformed the baseline SRCNN network, running 3x to 8x faster while delivering vastly superior visual quality.

  • Track A (Super-Resolution): TEAM_ALEX achieved the highest overall balanced scores across all validation tracks with a PSNR of 28.21 dB. Meanwhile, KAIST-VICLAB proved to be the speed champion, processing HD images in under a second on actual mobile devices (812 ms on a Razer Phone and 985 ms on a Huawei P20).
  • Track B (Image Enhancement): Mt.Phoenix won the track by achieving the top human MOS (2.6804) and the fastest smartphone runtimes among the top contenders (1.47 seconds on the Razer Phone).

Limitations and Future Work

  • Deployment Framework Constraints: A major hurdle highlighted by the challenge was mobile framework compatibility. For instance, the Rainbow team utilized a specific contrast adjustment operation (tf.image.adjust_contrast) that was not yet supported by TensorFlow Mobile, preventing their model from running on Android during testing.
  • Memory Bottlenecks: Peak memory consumption remains a critical issue for mobile deployment. Several high-performing models (such as the EdS submission and the DPED baseline) suffered from "Out of Memory" errors on the Huawei P20 smartphone.
  • Future Directions: The organizers emphasize the need for future architectures to better exploit hardware-specific mobile acceleration (like NPUs) and to develop automated perceptual loss functions that align more closely with human judgment without requiring massive network depths.

Summary updated

Abstract

This paper reviews the first challenge on efficient perceptual image enhancement with the focus on deploying deep learning models on smartphones. The challenge consisted of two tracks. In the first one, participants were solving the classical image super-resolution problem with a bicubic downscaling factor of 4. The second track was aimed at real-world photo enhancement, and the goal was to map low-quality photos from the iPhone 3GS device to the same photos captured with a DSLR camera. The target metric used in this challenge combined the runtime, PSNR scores and solutions' perceptual results measured in the user study. To ensure the efficiency of the submitted models, we additionally measured their runtime and memory requirements on Android smartphones. The proposed solutions significantly improved baseline results defining the state-of-the-art for image enhancement on smartphones.
BibTeX
@misc{ignatov2018pirm,
  author = {Andrey Ignatov and Radu Timofte and Thang Van Vu and Tung Minh Luu and Trung X. Pham and Cao Van Nguyen and Yongwoo Kim and Jae-Seok Choi and Munchurl Kim and Jie Huang and Jiewen Ran and Chen Xing and Xingguang Zhou and Pengfei Zhu and Mingrui Geng and Yawei Li and Eirikur Agustsson and Shuhang Gu and Luc Van Gool and Etienne de Stoutz and Nikolay Kobyshev and Kehui Nie and Yan Zhao and Gen Li and Tong Tong and Qinquan Gao and Hanwen Liu and Pablo Navarrete Michelini and Dan Zhu and Hu Fengshuo and Zheng Hui and Xiumei Wang and Lirui Deng and Rang Meng and Jinghui Qin and Yukai Shi and Wushao Wen and Liang Lin and Ruicheng Feng and Shixiang Wu and Chao Dong and Yu Qiao and Subeesh Vasu and Thekke Madam Nimisha and Praveen Kandula and A. N. Rajagopalan and Jie Liu and Cheolkon Jung},
  title = {{PIRM Challenge on Perceptual Image Enhancement on Smartphones: Report}},
  year = {2018},
  eprint = {1810.01641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.01641},
}
Figure