SYENet: A Simple Yet Effective Network for Multiple Low-Level Vision Tasks with Real-time Performance on Mobile Device

Weiran Gou, Ziyao Yi, Yan Xiang, Shaoqing Li, Zibin Liu, Dehui Kong, Ke XuICCV 2023

Introduction

Low-level vision tasks, such as Image Signal Processing (ISP), Low-Light Enhancement (LLE), and Super-Resolution (SR), are essential for high-quality mobile photography. However, most state-of-the-art models are designed for high-end GPUs and possess millions of parameters, making them unsuitable for the strict real-time constraints and limited memory bandwidth of mobile devices. SYENet (Simple Yet Effective Network) addresses these limitations with an ultra-lightweight design of only 6,000 parameters, enabling 2K60FPS throughput on modern mobile SoCs.

Architecture and Methodology

SYENet's design focuses on maximizing representational power within a minimal parameter budget through several innovative components.

Asymmetric Multi-Branch Architecture

The network splits the processing workload into two distinct sub-tasks: Texture Generation and Pattern Classification. By using a deeper branch for regression and a shallower branch for pixel grouping, the model allocates its limited parameters more effectively than a standard monolithic architecture.

Quadratic Connection Unit (QCU)

Standard linear feature fusion (like addition) often lacks the expressiveness needed for tiny models. SYENet introduces the QCU, which uses element-wise multiplication to create a quadratic feature space: $QCU(F_1, F_2) = (F_1 \odot F_2) \oplus B$. This non-linear fusion significantly boosts the model's ability to handle complex visual patterns.

Revised Re-parameterization

To improve performance without increasing inference latency, SYENet employs structural re-parameterization. During training, the model uses a complex block featuring multiple convolution kernels (e.g., $1\times1$, $3\times3$, $5\times5$) and Batch Normalization (BN). At inference, these branches are mathematically merged into a single $5\times5$ convolution.

The multi-branch training structure is re-parameterized into a single convolution for efficient inference.

Outlier-Aware Loss ($L_{OA}$)

To prevent the "regression to the mean" effect that often causes blurriness in lightweight models, the authors developed an Outlier-Aware Loss. This loss function uses global error statistics to prioritize "hard" pixels (outliers) during training, ensuring the model preserves sharp details and edges.

Experimental Results

SYENet was benchmarked across ISP, Super-Resolution, and Low-Light Enhancement tasks, demonstrating a superior efficiency-to-quality ratio.

ISP and Super-Resolution Performance

In the MAI 2022 Learned Smartphone ISP challenge, SYENet achieved the highest overall score, effectively balancing image quality (PSNR) with extreme processing speed. In Super-Resolution benchmarks (Set5, Set14, etc.), SYENet outperformed other mobile-oriented lightweight models by 1–7dB PSNR while using only a fraction of the parameters.

Performance comparison of SYENet against other methods in ISP (a) and Super-Resolution (b) tasks. Bubble size represents processing speed.

Real-time Mobile Throughput

On the Qualcomm Snapdragon 8 Gen 1 SoC, SYENet achieves a throughput of 2K60FPS. This performance is roughly 100x faster than many "lightweight" competitors, making it viable for real-time video enhancement on consumer mobile devices.

Conclusion

SYENet demonstrates that ultra-lightweight neural networks can achieve competitive results for low-level vision tasks through clever architectural choices like quadratic fusion and re-parameterization. While currently limited in tasks requiring complex temporal alignment like Video Super-Resolution, SYENet sets a new benchmark for on-device real-time image processing.

Summary updated

BibTeX
@inproceedings{gou2023syenet,
  author = {Weiran Gou and Ziyao Yi and Yan Xiang and Shaoqing Li and Zibin Liu and Dehui Kong and Ke Xu},
  title = {{SYENet: A Simple Yet Effective Network for Multiple Low-Level Vision Tasks with Real-time Performance on Mobile Device}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure