Multi-Memory Convolutional Neural Network for Video Super-Resolution

Zhongyuan Wang, Peng Yi, Kui Jiang, Junjun Jiang, Zhen Han, Tao Lu, Jiayi Ma

Abstract

Video super-resolution (SR) is focused on reconstructing high-resolution frames from consecutive low-resolution (LR) frames. Most previous video SR methods based on convolutional neural networks (CNN) use a direct connection and single-memory module within the network, and thus, they fail to make full use of spatio-temporal complementary information from LR observed frames. To fully exploit spatio-temporal correlations between adjacent LR frames and reveal more realistic details, this paper proposes a multi-memory CNN (MMCNN) for video SR, cascading an optical flow network and an image-reconstruction network. A series of residual blocks engaged in utilizing intra-frame spatial correlations is proposed for feature extraction and reconstruction. Particularly, instead of using a single-memory module, we embed convolutional long short-term memory into the residual block, thus forming a multi-memory residual block to progressively extract and retain inter-frame temporal correlations between the consecutive LR frames. We conduct extensive experiments on numerous testing datasets with respect to different scaling factors. Our proposed MMCNN shows superiority over the state-of-the-art methods in terms of PSNR and visual quality and surpasses the best counterpart method by 1 dB at most.
BibTeX
@article{wang2019multi,
  author = {Zhongyuan Wang and Peng Yi and Kui Jiang and Junjun Jiang and Zhen Han and Tao Lu and Jiayi Ma},
  title = {{Multi-Memory Convolutional Neural Network for Video Super-Resolution}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  doi = {10.1109/tip.2018.2887017},
}

Loading related content...

Figure