Expectation-Maximization Attention Cross Residual Network for Single Image Super-Resolution

Xiaobiao Du, Jie Niu, Chongjin LiuWorkshops @ CVPR 2021 · OpenReview

Abstract

Single Image Super-Resolution (SISR) remains a challenging ill-posed problem in computer vision. While deep learning has significantly advanced the field, existing models often struggle with high computational costs and optimization difficulties. This paper introduces the Expectation-Maximization Attention Cross Residual Network (EACRN), which utilizes a hierarchical "cross-residual" architecture and an iterative Expectation-Maximization (EM) attention mechanism. EACRN efficiently captures long-range dependencies and multi-scale features, achieving state-of-the-art accuracy with a fraction of the training updates required by previous models.

Introduction

The goal of SISR is to reconstruct a high-resolution (HR) image from a single low-resolution (LR) input. Deep Convolutional Neural Networks (CNNs) have become the standard approach, but they face two primary hurdles:

  1. Optimization Complexity: Increasing network depth often leads to vanishing gradients, making training difficult.
  2. Computational Inefficiency: Global context is typically captured via self-attention, which has quadratic complexity $O(N^2)$ relative to pixel count, leading to high memory and compute demands.

Proposed Method: EACRN

The EACRN architecture addresses these challenges through three key innovations:

Cross Residual in Cross Residual (CRICR)

Instead of standard linear residual connections, EACRN employs a hierarchical structure. Multiple Cross Residual Groups (CRG) are linked via global skip connections. Within each CRG, Cross Residual Blocks (CRB) utilize short skip connections. This design creates "highways" for low-frequency information, allowing the network to focus its learning capacity on high-frequency details like edges and textures.

Figure 1: The overall architecture of EACRN, highlighting the hierarchical Cross Residual in Cross Residual (CRICR) structure and the flow from LR to HR.

Multi-Scale Feature Extraction

To capture patterns of varying sizes, the residual blocks incorporate convolutions with different kernel sizes (e.g., $3\times3$ and $5\times5$). This allows the model to adaptively process features at multiple scales simultaneously.

Expectation-Maximization Attention (EMA)

The core innovation is the EMA module. Unlike standard self-attention that computes a full pixel-to-pixel similarity matrix, EMA treats attention as an iterative estimation problem. It uses the EM algorithm to find a compact set of representative "bases." This approach is robust to input variance and significantly reduces the computational burden, making it highly efficient for GPU processing.

Experimental Results

The model was evaluated on standard benchmarks including Set5, Set14, BSD100, and Urban100, using the DIV2K dataset for training.

Quantitative and Qualitative Performance

EACRN consistently outperformed established models like RCAN, EDSR, and SAN. On the Set5 benchmark ($2\times$ scale), EACRN achieved a PSNR of 38.42 dB, surpassing RCAN's 38.27 dB. Visually, EACRN produces sharper edges and more accurate textures, particularly in complex urban scenes where other models often introduce blurring or aliasing.

Training Efficiency

A standout feature of EACRN is its convergence speed. As shown in the comparative analysis, while models like RCAN require approximately $4 \times 10^5$ updates to reach peak performance, EACRN achieves superior results in just $1 \times 10^3$ updates—a massive reduction in training time and resource consumption.

Figure 6: Visual comparison of super-resolution results on benchmark datasets and a specification comparison with RCAN, demonstrating EACRN's superior texture reconstruction and training efficiency.

Conclusion

EACRN represents a significant step forward in efficient SISR. By combining a hierarchical cross-residual structure with an iterative EM-based attention mechanism, the model achieves high-quality reconstructions with remarkable training efficiency. While the performance gains are most notable on complex textures, future work may focus on optimizing the model for simpler, repetitive geometric patterns and extending the architecture to video super-resolution.

Summary updated

BibTeX
@inproceedings{du2021expectation,
  author = {Xiaobiao Du and Jie Niu and Chongjin Liu},
  title = {{Expectation-Maximization Attention Cross Residual Network for Single Image Super-Resolution}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=GMQmHurlo1B},
}
Figure