Abstract
Layout guidance, which specifies pixel-wise object distribution, is essential for preserving object boundaries in image inpainting without compromising a model’s generalization capability. This paper presents an efficient and robust layout-guided image inpainting method designed for mobile environments. Our approach utilizes two sub-models that restore pixel information from coarse to fine, specifically addressing the challenges of "mixed scenes" where delicate object shapes reside adjacent to image holes. By leveraging category-agnostic layout masks, our method achieves high structural integrity and efficiency, as demonstrated through extensive experiments and a mobile demonstration.
Introduction: The Challenge of Mixed Scenes
Image inpainting—the process of filling missing or damaged parts of an image—often fails in "mixed scenes." These are complex scenarios where an object boundary (such as an airplane wing) intersects with a missing region. Traditional inpainting methods frequently "smear" these boundaries or fail to reconstruct delicate shapes accurately.
While some existing solutions use semantic masks (labels like "dog" or "sky") to guide reconstruction, they suffer from two primary drawbacks: poor generalization to unseen object classes and appearance bias, where the model memorizes training data (e.g., always filling a sky with blue even if the context is a sunset). Furthermore, high-performing models are typically too computationally intensive for mobile hardware, leading to high latency and battery drain.
Methodology: A Two-Stage Layout-Guided Pipeline
To address these issues, we propose a lightweight, two-stage pipeline that prioritizes efficiency and structural integrity. The core innovation lies in using category-agnostic layout masks rather than rigid semantic labels.
1. Layout-Guided Diffusion (Coarse Stage)
This stage uses a non-parametric iterative process to propagate textures from known areas into the hole. Critically, this propagation is restricted to the same layout class. By using a layout mask that marks boundaries without naming objects, the model avoids the "memorization" trap of semantic labels and can generalize to any object shape.
2. Refinement with Top-X Attention (Fine Stage)
This stage sharpens the coarse result using a novel Top-X Attention mechanism. Unlike standard contextual attention that searches the entire image—a process that is slow and prone to artifacts—Top-X attention restricts the search to the most relevant patches within the same layout region. This simultaneously improves visual quality and reduces computational load for mobile GPUs.
3. Adaptive Dropout
To prevent unnatural "grid artifacts," we implement an adaptive dropout mechanism. This varies based on the color saturation of the image, ensuring texture diversity and natural-looking results in the filled areas.
Key Results and Performance
Our method was evaluated using custom benchmarks derived from COCO and ADE20K datasets, featuring approximately 53.4K training images and realistic object-shaped holes. The results demonstrate significant improvements over "heavyweight" competitors like SGE-Net and MAT.
- Efficiency: The model is 83.2% smaller in parameter count than previous state-of-the-art methods. On a Snapdragon 865+ mobile processor, it achieves an average latency of approximately 2 seconds.
- Quality: Our method achieved the best (lowest) FID score (11.79) and highest SSIM (0.9587), indicating superior statistical similarity to real images and high structural accuracy.
- Robustness: Qualitative analysis shows the model successfully preserving sharp edges in complex scenes where other models produced blurry or "leaky" boundaries.
Conclusion
By replacing semantic labels with flexible layout masks and introducing Top-X attention, we have developed a robust inpainting solution optimized for mobile devices. While the method currently depends on the accuracy of the input layout mask, it offers users creative control—allowing them to guide the inpainting process with simple sketches. Future work will explore integrating this layout-guidance into advanced generative frameworks like Latent Diffusion Models to further enhance mobile image editing capabilities.
Summary updated