MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices

Andranik Sargsyan, Shant Navasardyan, Xingqian Xu, Humphrey Shi

Abstract

Mobile Inpainting GAN (MI-GAN) is a lightweight generative baseline designed for high-quality image inpainting directly on mobile devices. By integrating structural re-parameterization and knowledge distillation from larger "teacher" models, MI-GAN achieves performance comparable to state-of-the-art (SOTA) approaches while remaining significantly smaller and faster. This work addresses the gap between heavy server-side models and the need for efficient, privacy-preserving on-device processing.

Introduction

Image inpainting—the process of reconstructing missing or removed regions of an image—has advanced significantly with the advent of Large Diffusion Models (LDMs) and complex Generative Adversarial Networks (GANs). However, these "heavyweight" models typically require substantial computational resources, often necessitating server-side execution. This dependency introduces latency, increases operational costs, and raises user privacy concerns.

Existing mobile-optimized models often struggle to balance efficiency with the "hallucination" capability required to synthesize complex textures and structures. MI-GAN is proposed as a solution that runs locally on smartphone hardware without sacrificing output quality.

Methodology

MI-GAN utilizes three core strategies to optimize the trade-off between generative power and computational efficiency:

1. Dual-Branch Architecture

The model features a Main Branch based on a U-Net style encoder-decoder using depthwise separable convolutions, and a Painting Branch. Inspired by StyleGAN, the painting branch iteratively generates missing regions in RGB space, progressing from coarse structures to fine details. This approach ensures global consistency while maintaining sharp local textures.

2. Structural Re-parameterization

To enhance model capacity during training without impacting inference speed, MI-GAN employs structural re-parameterization. The model is trained with multiple parallel convolution kernels (over-parameterization), which are mathematically fused into a single, efficient kernel for deployment. This allows the model to "act big" during training but "stay small" during execution.

3. Knowledge Distillation (KD)

MI-GAN leverages a pre-trained Co-Mod-GAN as a "teacher" model. During training, the "student" MI-GAN minimizes the discrepancy between its intermediate results and those of the teacher. This process allows the lightweight student to inherit the generative prowess of the much larger teacher model.

Experiments and Results

The model was evaluated on standard benchmarks, including Places2 for diverse scenes, FFHQ for facial features, and OpenImages for real-world user studies.

Qualitative inpainting results and quantitative comparison of FID vs. FLOPs across various models.

Performance Metrics

MI-GAN demonstrates significant improvements in efficiency:

  • Efficiency: MI-GAN is approximately 13x smaller (5.95M parameters) and 8x faster than its teacher model, Co-Mod-GAN.
  • On-Device Speed: On an iPhone 14 Pro Max, MI-GAN processes a 256x256 image in 296ms, compared to over 1.3 seconds for heavier SOTA models.
  • Quality: On the Places2 dataset, MI-GAN achieved an FID (Fréchet Inception Distance) of 11.83, outperforming larger models like LaMa (FID 22.00) and ZITS (FID 16.78).

User Preference

In blind studies, MI-GAN's results were preferred over the native object removal tools found in flagship devices like the Google Pixel 6 and Samsung Galaxy S22.

Conclusion and Future Work

MI-GAN establishes a strong baseline for on-device inpainting. While highly effective for general object removal, it can encounter difficulties with complex 3D structures where perspective is critical. Future work may involve incorporating additional structural priors, such as edge maps or 3D geometry cues, to improve architectural reconstruction. Furthermore, scaling the model to handle ultra-high-resolution photography remains an open challenge for mobile generative AI.

Summary updated

Abstract

In recent years, many deep learning based image inpainting methods have been developed by the research community. Some of those methods have shown impressive image completion abilities. Yet, to the best of our knowledge, there is no image inpainting model designed to run on mobile devices. In this paper we present a simple image inpainting baseline, Mobile Inpainting GAN (MI-GAN), which is approximately one order of magnitude computationally cheaper and smaller than existing state-of-the-art inpainting models, and can be efficiently deployed on mobile devices. Excessive quantitative and qualitative evaluations show that MI-GAN performs comparable or, in some cases, better than recent state-of-the-art approaches. Moreover, we perform a user study comparing MI-GAN results with results from several commercial mobile inpainting applications, which clearly shows the advantage of MI-GAN in comparison to existing apps. With the purpose of high quality and efficient inpainting, we utilize an effective combination of adversarial training, model reparametrization, and knowledge distillation. Our models and code are publicly available at https://github.com/Picsart-AI-Research/MI-GAN.
BibTeX
@inproceedings{sargsyan2023mi,
  author = {Andranik Sargsyan and Shant Navasardyan and Xingqian Xu and Humphrey Shi},
  title = {{MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00674},
}
Figure