Abstract
Mobile Inpainting GAN (MI-GAN) is a lightweight generative baseline designed for high-quality image inpainting directly on mobile devices. By integrating structural re-parameterization and knowledge distillation from larger "teacher" models, MI-GAN achieves performance comparable to state-of-the-art (SOTA) approaches while remaining significantly smaller and faster. This work addresses the gap between heavy server-side models and the need for efficient, privacy-preserving on-device processing.
Introduction
Image inpainting—the process of reconstructing missing or removed regions of an image—has advanced significantly with the advent of Large Diffusion Models (LDMs) and complex Generative Adversarial Networks (GANs). However, these "heavyweight" models typically require substantial computational resources, often necessitating server-side execution. This dependency introduces latency, increases operational costs, and raises user privacy concerns.
Existing mobile-optimized models often struggle to balance efficiency with the "hallucination" capability required to synthesize complex textures and structures. MI-GAN is proposed as a solution that runs locally on smartphone hardware without sacrificing output quality.
Methodology
MI-GAN utilizes three core strategies to optimize the trade-off between generative power and computational efficiency:
1. Dual-Branch Architecture
The model features a Main Branch based on a U-Net style encoder-decoder using depthwise separable convolutions, and a Painting Branch. Inspired by StyleGAN, the painting branch iteratively generates missing regions in RGB space, progressing from coarse structures to fine details. This approach ensures global consistency while maintaining sharp local textures.
2. Structural Re-parameterization
To enhance model capacity during training without impacting inference speed, MI-GAN employs structural re-parameterization. The model is trained with multiple parallel convolution kernels (over-parameterization), which are mathematically fused into a single, efficient kernel for deployment. This allows the model to "act big" during training but "stay small" during execution.
3. Knowledge Distillation (KD)
MI-GAN leverages a pre-trained Co-Mod-GAN as a "teacher" model. During training, the "student" MI-GAN minimizes the discrepancy between its intermediate results and those of the teacher. This process allows the lightweight student to inherit the generative prowess of the much larger teacher model.
Experiments and Results
The model was evaluated on standard benchmarks, including Places2 for diverse scenes, FFHQ for facial features, and OpenImages for real-world user studies.
Performance Metrics
MI-GAN demonstrates significant improvements in efficiency:
- Efficiency: MI-GAN is approximately 13x smaller (5.95M parameters) and 8x faster than its teacher model, Co-Mod-GAN.
- On-Device Speed: On an iPhone 14 Pro Max, MI-GAN processes a 256x256 image in 296ms, compared to over 1.3 seconds for heavier SOTA models.
- Quality: On the Places2 dataset, MI-GAN achieved an FID (Fréchet Inception Distance) of 11.83, outperforming larger models like LaMa (FID 22.00) and ZITS (FID 16.78).
User Preference
In blind studies, MI-GAN's results were preferred over the native object removal tools found in flagship devices like the Google Pixel 6 and Samsung Galaxy S22.
Conclusion and Future Work
MI-GAN establishes a strong baseline for on-device inpainting. While highly effective for general object removal, it can encounter difficulties with complex 3D structures where perspective is critical. Future work may involve incorporating additional structural priors, such as edge maps or 3D geometry cues, to improve architectural reconstruction. Furthermore, scaling the model to handle ultra-high-resolution photography remains an open challenge for mobile generative AI.
Summary updated