SEM-Net: Efficient Pixel Modelling for Image Inpainting with Spatially Enhanced SSM

Shuang Chen, Haozheng Zhang, Amir Atapour-Abarghouei, Hubert P. H. ShumWACV 2025

TL;DR

This paper introduces SEM-Net, a novel image inpainting model that leverages State Space Models (SSMs) to efficiently capture long-range dependencies (LRDs) at the pixel level. By introducing a "Snake Mamba Block" and a "Spatially-Enhanced Feedforward Network," SEM-Net overcomes the limitations of existing methods in handling complex image structures and achieves state-of-the-art performance on inpainting tasks.

What Problem Does This Paper Address?

Image inpainting, the task of filling in missing or damaged regions of an image, is notoriously challenging due to the need for semantic plausibility and pixel-level consistency. Traditional Convolutional Neural Networks (CNNs) struggle with capturing long-range dependencies (LRDs) because their receptive fields grow slowly. While Transformer-based methods can capture LRDs, they suffer from quadratic computational complexity and often operate on patches rather than pixels, limiting their ability to reconstruct fine spatial details. Existing State Space Models (SSMs), like Mamba, show promise for LRDs but lack inherent 2D spatial awareness, leading to issues like positional drifting in inpainted regions. This paper aims to bridge this gap by developing an efficient SSM-based model that effectively captures both LRDs and crucial spatial information at the pixel level.

Methods

The core of SEM-Net is a U-shaped encoder-decoder architecture built upon novel components:

  • Snake Mamba Block (SMB): This block is designed to address the limitations of vanilla SSMs. It comprises two key sub-modules:
  • Snake Bi-Directional Modelling (SBDM): This module reorders pixels in a "snake-like" manner to maintain pixel adjacency continuity when flattening spatial features into a sequence. It also incorporates bi-directional traversal (horizontal and vertical) to capture more comprehensive pixel dependencies.
  • Position Enhancement Layer (PE Layer): This layer explicitly injects 1D positional embeddings into the sequences to enhance positional awareness, crucial for understanding the relative locations of pixels.
  • Spatially-Enhanced Feedforward Network (SEFN): This network complements the SMB by incorporating local spatial information. It uses features from before and after the Mamba block, applies average pooling to expand the receptive field, and then uses a gating mechanism to modulate the features, thereby enhancing spatial awareness.

The overall architecture uses four stages of these SEM blocks, with skip connections to preserve information across scales.

Datasets

The paper evaluates SEM-Net on two widely used inpainting datasets:

  • CelebA-HQ: A dataset of high-quality celebrity faces.
  • Places2-Standard: A dataset of scene images.

The paper also mentions evaluating SEM-Net on two motion deblurring datasets (GoPro and HIDE) to demonstrate its generalizability.

Key Insights and Ideas

  • Snake-like traversal is crucial for pixel-level SSMs: Flattening image features into a sequence for SSMs can break spatial adjacency. A snake-like traversal preserves this continuity, allowing SSMs to better learn pixel-level dependencies.
  • Bi-directional processing enhances spatial context: Combining horizontal and vertical traversals in the SBDM module provides a more holistic understanding of spatial relationships than a single direction.
  • Explicit positional encoding is vital for SSMs: While SSMs are good at sequential modeling, they can lack awareness of absolute positions. Explicitly adding positional embeddings helps the model understand where pixels are located in the image.
  • SEFN effectively injects local spatial context: SSMs excel at LRDs, but local details can be lost. SEFN acts as a crucial post-processing step to refine features with local spatial information.
  • SSMs can be efficient and powerful for vision tasks: SEM-Net demonstrates that SSMs, when properly adapted for spatial data, can achieve state-of-the-art results with competitive efficiency compared to Transformers.

Key Results

  • State-of-the-art Inpainting Performance: SEM-Net outperforms existing CNN, Transformer, and diffusion-based methods on CelebA-HQ and Places2 datasets across various mask ratios, showing significant improvements in PSNR, SSIM, and L1 metrics. For example, on CelebA-HQ, it achieved substantial gains in PSNR and significant reductions in L1.
  • Improved Spatial Consistency and LRD Capture: Qualitative results show SEM-Net generates more coherent structures, sharper textures, and better semantic details, particularly in challenging cases like replicating eye patterns.
  • Efficient LRD Capture: The proposed SMB block is shown to be more efficient than Transformer blocks (like CSA and SSA) for capturing LRDs, enabling training on higher resolution images.
  • Generalization Ability: SEM-Net demonstrates strong generalization to unseen high-resolution images and performs competitively on the image motion deblurring task without task-specific modifications.

Limitations and Future Work

  • Limitations: The paper notes that while SEM-Net performs well, the vanilla SSM's inherent lack of 2D spatial awareness was a significant challenge that required novel architectural designs (SMB and SEFN) to overcome.
  • Future Work: The authors suggest building a controllable image inpainting model based on the proposed SMB to handle large-resolution images more effectively.

Summary updated

BibTeX
@inproceedings{chen2025sem,
  author = {Shuang Chen and Haozheng Zhang and Amir Atapour-Abarghouei and Hubert P. H. Shum},
  title = {{SEM-Net: Efficient Pixel Modelling for Image Inpainting with Spatially Enhanced SSM}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure