Region-aware Attention for Image Inpainting

Zhilin Huang, Chujun Qin, Zhenyu Weng, Yuesheng ZhuarXiv 2022 · DOI · arXiv

Abstract: Traditional attention mechanisms in image inpainting often suffer from blurry artifacts due to poor initial feature guesses and a lack of global dataset context. Researchers from Peking University propose Region-aware Attention (RA), a novel module that maps pixels to learnable regions and utilizes a global dictionary of prototypical features. This approach ensures semantic consistency and produces sharper, more realistic reconstructions across diverse datasets like CelebA and Places2.

Introduction

Image inpainting—the process of reconstructing missing or damaged parts of an image—has increasingly relied on attention mechanisms to borrow information from distant, uncorrupted areas. For instance, a model might look at a visible left eye to reconstruct a missing right eye. However, existing state-of-the-art modules, such as the Contextual Attention Module (CAM), face two primary challenges:

  1. The "Garbage In, Garbage Out" Loop: Standard attention calculates similarity between known pixels and the "guessed" features inside the hole. Because these initial guesses are often poor, the mechanism is misled, resulting in blurry or semantically nonsensical textures.
  2. Single-Sample Myopia: Most models operate on a per-image basis, ignoring the vast "prototypical" information (e.g., the standard structure of a nose or a brick wall) present across the entire training dataset.

Methodology: Region-aware Attention (RA)

To address these issues, the authors introduce a single-stage encoder-decoder architecture featuring a Local-Global Attention (LGA) layer. The centerpiece of this layer is the Region-aware Attention (RA) module, which departs from traditional pixel-to-pixel mapping in favor of a pixel-to-region approach.

Region Mask Generator (RMG)

Instead of comparing every pixel to every other pixel ($N$-to-$N$), the RMG maps each pixel to a small number of semantic "regions" ($n$). By using a shared-weight linear layer, the model generates a coarse, global estimate of the image structure. This effectively bypasses the volatile, ill-predicted local features that typically degrade attention performance.

Figure 4: Visualization of the Region Mask Generator, which maps pixels to semantic regions to guide the attention mechanism.

Learnable Region Dictionary (LRD)

The LRD acts as a "memory bank" that stores the most important feature representations from the entire dataset. When filling a hole, the model retrieves information from this dictionary based on the region mask. For example, if the RMG identifies a "face region," the model pulls "face-like" features learned from thousands of other images, ensuring semantic consistency even when local context is missing.

Experimental Results

The proposed method was evaluated on the CelebA, Places2, and Paris StreetView benchmarks against irregular masks covering 10% to 50% of the image area.

Quantitative Performance

On the CelebA dataset with 40-50% missing area, the model achieved a Fréchet Inception Distance (FID) of 19.47, significantly outperforming the 20.23 achieved by standard Contextual Attention (CA). It also demonstrated superior performance in PSNR and SSIM metrics across nearly all test categories.

Qualitative Analysis

Visually, the RA-based model produces much sharper boundaries and more realistic textures than its competitors (such as LBAM and CTSDG). While other models often leave "ghosting" artifacts or blurry patches in large holes, this method synthesizes semantically plausible features, such as consistent eye shapes and architectural lines.

Figure 3: A high-quality reconstruction showing sharp, semantically consistent facial features generated by the RA module.

Conclusion and Future Work

By shifting from local pixel-to-pixel correlation to a global, region-aware framework, the authors have significantly improved the stability and quality of image inpainting. While the model excels at structured objects like human faces, the authors noted that chaotic natural scenes remain a challenge. Future work may involve incorporating explicit semantic segmentation labels to help the Region Mask Generator better distinguish between complex objects in non-stationary environments.

Summary updated

Abstract

Recent attention-based image inpainting methods have made inspiring progress by modeling long-range dependencies within a single image. However, they tend to generate blurry contents since the correlation between each pixel pairs is always misled by ill-predicted features in holes. To handle this problem, we propose a novel region-aware attention (RA) module. By avoiding the directly calculating corralation between each pixel pair in a single samples and considering the correlation between different samples, the misleading of invalid information in holes can be avoided. Meanwhile, a learnable region dictionary (LRD) is introduced to store important information in the entire dataset, which not only simplifies correlation modeling, but also avoids information redundancy. By applying RA in our architecture, our methodscan generate semantically plausible results with realistic details. Extensive experiments on CelebA, Places2 and Paris StreetView datasets validate the superiority of our method compared with existing methods.
BibTeX
@misc{huang2022region,
  author = {Zhilin Huang and Chujun Qin and Zhenyu Weng and Yuesheng Zhu},
  title = {{Region-aware Attention for Image Inpainting}},
  year = {2022},
  eprint = {2204.01004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.01004},
}
Figure