M3Cam: Extreme Super-resolution via Multi-Modal Optical Flow for Mobile Cameras
Lists
Only you see this
Abstract
The demand for ultra-high-resolution imaging in mobile phone photography is continuously increasing. However, the image resolution of mobile devices is typically constrained by the size of the CMOS sensor. Although deep learning-based super-resolution (SR) techniques have the potential to overcome this limitation, existing SR neural network models require large computational resources, making them unsuitable for real-time SR imaging on current mobile devices. Additionally, cloud-based SR systems pose privacy leakage risks. In this paper, we propose M3Cam, an innovative and lightweight SR imaging system for mobile phones. M3Cam can ensure high-quality 16× SR image (4× in both height and width) visualization with almost negligible latency. In detail, we utilize an optical image stabilization (OIS) module for lens control and introduce a new modality of data, namely gyroscope readings, to achieve high-precision and compact optical flow estimation modules. Building upon this concept, we design a multi-frame-based SR model utilizing the Swin Transformer. Our proposed system can generate a 16× SR image from four captured low-resolution images in real-time, with low computational load, low inference latency, and minimal reliance on runtime RAM. Through extensive experiments, we demonstrate that our proposed multi-modal optical flow model significantly enhances pixel alignment accuracy between multiple frames and delivers outstanding 16× SR imaging results under various shooting scenarios. Code and dataset are available at: https://github.com/liangjindeamo-yuer/M3CAM
BibTeX
@inproceedings{lu2024m,
author = {Yu Lu and Dian Ding and Hao Pan and Yongjian Fu and Liyun Zhang and Feitong Tan and Ran Wang and Yi-Chao Chen and Guangtao Xue and Ju Ren},
title = {{M3Cam: Extreme Super-resolution via Multi-Modal Optical Flow for Mobile Cameras}},
booktitle = {SenSys},
year = {2024},
url = {https://openreview.net/forum?id=UZRn75L51U},
}Loading related content...