Abstract
Image inpainting is an ill-posed problem that often relies on auxiliary structures like edges or segmentation maps to guide the reconstruction of missing pixels. However, traditional multi-stage pipelines suffer from "biased priors," where errors in initial structure prediction propagate to the final output. This paper presents an end-to-end Multi-Modality Guided Transformer that jointly learns from RGB images, semantic segmentation, and edge textures. By utilizing a mutual decoder and novel normalization techniques, the model allows modalities to compensate for each other's inaccuracies, resulting in state-of-the-art visual realism and significantly improved computational efficiency.
Introduction
Image inpainting involves "hallucinating" missing regions in damaged or masked images. To achieve realistic results, modern deep learning approaches frequently incorporate auxiliary information—such as edge maps or semantic segmentation—to provide structural guidance. Despite their success, most existing methods employ a multi-stage approach: they first predict the structural priors and then use these predictions to guide the inpainting process.
This sequential dependency creates a significant bottleneck. If the initial stage fails to correctly identify a structural element (e.g., the frame of a pair of glasses), the subsequent inpainting stage will faithfully reproduce that error, leading to visible artifacts. Furthermore, stacking multiple complex networks for different tasks results in high computational overhead, making these models unsuitable for real-time applications. This paper addresses these issues by proposing a single-pass, multi-modality framework that treats auxiliary tasks as collaborative rather than dictatorial.
Methodology
The proposed architecture is a Multi-Modality Guided Transformer designed for end-to-end training. It introduces three primary innovations to handle the interplay between different data types.
Adaptive Contextual Bottlenecks (ACB)
Located within the encoder, the ACB modules address the limitations of standard dilated convolutions, which often produce "gridding" artifacts. ACB utilizes dynamic gating to weight different dilated convolutions, allowing the network to automatically adjust its receptive field based on the specific shape and size of the masked area. This flexibility is crucial for handling the unpredictable nature of irregular masks.
Multi-modal Mutual Decoder
Instead of a linear pipeline, the decoder features three parallel branches for Inpainting, Segmentation, and Edge detection. These branches interact through a Multi-Scale Spatial-aware Attention (MSSA) module. This structure ensures that no single modality dictates the final output; instead, they share features across scales to maintain structural and semantic consistency.
Auxiliary DeNormalization (ADN)
To merge features within the attention module, the authors introduce ADN. Unlike previous techniques like SPADE or AdaIN that rely on a single predicted prior, ADN learns affine transformation parameters from the collective interplay of all three modalities. This "unbiased" approach allows the model to ignore noise in one modality (e.g., a poorly predicted edge) if the other modalities (e.g., semantic context) provide stronger evidence for a different structure.
Experimental Setup
The model was evaluated on four diverse benchmarks to test its robustness and generalization:
- CelebA-HQ: High-resolution facial images for testing fine-grained detail.
- OST (Outdoor Scene): Images across eight categories of outdoor environments.
- Cityscapes: Urban street-view images requiring strong structural consistency.
- Places2: A large-scale dataset of diverse scenes used to evaluate generalization with off-the-shelf segmentation tools.
Results and Discussion
Quantitative and Qualitative Performance
The model achieved state-of-the-art results across all benchmarks. Notably, it produced a ~2-point improvement in Fréchet Inception Distance (FID) over the previous leading method, CTSDG, on the CelebA-HQ dataset. Qualitatively, the model excels at restoring complex structures, such as facial features and architectural details, where previous methods often produced blurred or structurally disjointed patches.
Efficiency
By integrating segmentation and edge detection as auxiliary tasks within a single network, the model achieves significant speed gains. While some two-stage U-Net competitors operate as slowly as 1.37 FPS, the proposed transformer runs at approximately 30-40 FPS, making it viable for high-speed applications.
The "Unbiased" Advantage
The core insight of this work is that auxiliary priors should not be treated as absolute truth. By allowing RGB, edge, and segmentation features to "negotiate" via the MSSA and ADN modules, the model can effectively filter out errors from individual branches. This collaborative approach leads to more coherent and visually pleasing reconstructions.
Conclusion
This paper introduces an efficient, end-to-end transformer for image inpainting that overcomes the bias inherent in multi-stage structural guidance. While the model shows some limitations in high-level semantic recognition for extremely large masks, its ability to maintain structural consistency and high frame rates represents a significant step forward. Future research will explore extending this multi-modality interplay to video inpainting, where temporal consistency remains a major challenge.
Summary updated