Abstract: Traditional attention mechanisms in image inpainting often suffer from blurry artifacts due to poor initial feature guesses and a lack of global dataset context. Researchers from Peking University propose Region-aware Attention (RA), a novel module that maps pixels to learnable regions and utilizes a global dictionary of prototypical features. This approach ensures semantic consistency and produces sharper, more realistic reconstructions across diverse datasets like CelebA and Places2.
Introduction
Image inpainting—the process of reconstructing missing or damaged parts of an image—has increasingly relied on attention mechanisms to borrow information from distant, uncorrupted areas. For instance, a model might look at a visible left eye to reconstruct a missing right eye. However, existing state-of-the-art modules, such as the Contextual Attention Module (CAM), face two primary challenges:
- The "Garbage In, Garbage Out" Loop: Standard attention calculates similarity between known pixels and the "guessed" features inside the hole. Because these initial guesses are often poor, the mechanism is misled, resulting in blurry or semantically nonsensical textures.
- Single-Sample Myopia: Most models operate on a per-image basis, ignoring the vast "prototypical" information (e.g., the standard structure of a nose or a brick wall) present across the entire training dataset.
Methodology: Region-aware Attention (RA)
To address these issues, the authors introduce a single-stage encoder-decoder architecture featuring a Local-Global Attention (LGA) layer. The centerpiece of this layer is the Region-aware Attention (RA) module, which departs from traditional pixel-to-pixel mapping in favor of a pixel-to-region approach.
Region Mask Generator (RMG)
Instead of comparing every pixel to every other pixel ($N$-to-$N$), the RMG maps each pixel to a small number of semantic "regions" ($n$). By using a shared-weight linear layer, the model generates a coarse, global estimate of the image structure. This effectively bypasses the volatile, ill-predicted local features that typically degrade attention performance.
Learnable Region Dictionary (LRD)
The LRD acts as a "memory bank" that stores the most important feature representations from the entire dataset. When filling a hole, the model retrieves information from this dictionary based on the region mask. For example, if the RMG identifies a "face region," the model pulls "face-like" features learned from thousands of other images, ensuring semantic consistency even when local context is missing.
Experimental Results
The proposed method was evaluated on the CelebA, Places2, and Paris StreetView benchmarks against irregular masks covering 10% to 50% of the image area.
Quantitative Performance
On the CelebA dataset with 40-50% missing area, the model achieved a Fréchet Inception Distance (FID) of 19.47, significantly outperforming the 20.23 achieved by standard Contextual Attention (CA). It also demonstrated superior performance in PSNR and SSIM metrics across nearly all test categories.
Qualitative Analysis
Visually, the RA-based model produces much sharper boundaries and more realistic textures than its competitors (such as LBAM and CTSDG). While other models often leave "ghosting" artifacts or blurry patches in large holes, this method synthesizes semantically plausible features, such as consistent eye shapes and architectural lines.
Conclusion and Future Work
By shifting from local pixel-to-pixel correlation to a global, region-aware framework, the authors have significantly improved the stability and quality of image inpainting. While the model excels at structured objects like human faces, the authors noted that chaotic natural scenes remain a challenge. Future work may involve incorporating explicit semantic segmentation labels to help the Region Mask Generator better distinguish between complex objects in non-stationary environments.
Summary updated