Efficient Layout-Guided Image Inpainting for Mobile Use

Wenbo Li, Yi Wei, Yilin Shen, Hongxia JinWACV 2024 · OpenReview

Abstract

Layout guidance, which specifies pixel-wise object distribution, is essential for preserving object boundaries in image inpainting without compromising a model’s generalization capability. This paper presents an efficient and robust layout-guided image inpainting method designed for mobile environments. Our approach utilizes two sub-models that restore pixel information from coarse to fine, specifically addressing the challenges of "mixed scenes" where delicate object shapes reside adjacent to image holes. By leveraging category-agnostic layout masks, our method achieves high structural integrity and efficiency, as demonstrated through extensive experiments and a mobile demonstration.

Introduction: The Challenge of Mixed Scenes

Image inpainting—the process of filling missing or damaged parts of an image—often fails in "mixed scenes." These are complex scenarios where an object boundary (such as an airplane wing) intersects with a missing region. Traditional inpainting methods frequently "smear" these boundaries or fail to reconstruct delicate shapes accurately.

While some existing solutions use semantic masks (labels like "dog" or "sky") to guide reconstruction, they suffer from two primary drawbacks: poor generalization to unseen object classes and appearance bias, where the model memorizes training data (e.g., always filling a sky with blue even if the context is a sunset). Furthermore, high-performing models are typically too computationally intensive for mobile hardware, leading to high latency and battery drain.

Methodology: A Two-Stage Layout-Guided Pipeline

To address these issues, we propose a lightweight, two-stage pipeline that prioritizes efficiency and structural integrity. The core innovation lies in using category-agnostic layout masks rather than rigid semantic labels.

Input image with hole and corresponding layout mask used for guidance.

1. Layout-Guided Diffusion (Coarse Stage)

This stage uses a non-parametric iterative process to propagate textures from known areas into the hole. Critically, this propagation is restricted to the same layout class. By using a layout mask that marks boundaries without naming objects, the model avoids the "memorization" trap of semantic labels and can generalize to any object shape.

2. Refinement with Top-X Attention (Fine Stage)

This stage sharpens the coarse result using a novel Top-X Attention mechanism. Unlike standard contextual attention that searches the entire image—a process that is slow and prone to artifacts—Top-X attention restricts the search to the most relevant patches within the same layout region. This simultaneously improves visual quality and reduces computational load for mobile GPUs.

3. Adaptive Dropout

To prevent unnatural "grid artifacts," we implement an adaptive dropout mechanism. This varies based on the color saturation of the image, ensuring texture diversity and natural-looking results in the filled areas.

Key Results and Performance

Our method was evaluated using custom benchmarks derived from COCO and ADE20K datasets, featuring approximately 53.4K training images and realistic object-shaped holes. The results demonstrate significant improvements over "heavyweight" competitors like SGE-Net and MAT.

Qualitative comparison showing our method's superior boundary preservation compared to SGE-Net and SPG-Net.
  • Efficiency: The model is 83.2% smaller in parameter count than previous state-of-the-art methods. On a Snapdragon 865+ mobile processor, it achieves an average latency of approximately 2 seconds.
  • Quality: Our method achieved the best (lowest) FID score (11.79) and highest SSIM (0.9587), indicating superior statistical similarity to real images and high structural accuracy.
  • Robustness: Qualitative analysis shows the model successfully preserving sharp edges in complex scenes where other models produced blurry or "leaky" boundaries.

Conclusion

By replacing semantic labels with flexible layout masks and introducing Top-X attention, we have developed a robust inpainting solution optimized for mobile devices. While the method currently depends on the accuracy of the input layout mask, it offers users creative control—allowing them to guide the inpainting process with simple sketches. Future work will explore integrating this layout-guidance into advanced generative frameworks like Latent Diffusion Models to further enhance mobile image editing capabilities.

Summary updated

Abstract

The layout guidance, which specifies the pixel-wise object distribution, is beneficial to preserving the object boundaries in image inpainting while not hurting model’s generalization capability. We aim to design an efficient and robust layout-guided image inpainting method for mobile use, which can achieve the robustness in presence of the mixed scenes where objects with the delicate shape reside next to the hole. Our method is made up of two sub-models, which restore the pixel-information for the hole from coarse to fine, and support each other to overcome the practical challenges encountered when making the whole method lightweight. The layout mask guides the two sub-models, which thus enables the robustness of our method in mixed scenes. We demonstrate the efficiency and robustness of our method via both the experiments and a mobile demo.
BibTeX
@inproceedings{li2024efficient,
  author = {Wenbo Li and Yi Wei and Yilin Shen and Hongxia Jin},
  title = {{Efficient Layout-Guided Image Inpainting for Mobile Use}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=PoCIucwmSP},
}
Figure