SPAN: Spatial Pyramid Attention Network for Image Manipulation Localization

Xuefeng Hu, Zhihan Zhang, Zhenye Jiang, Syomantak Chaudhuri, Zhenheng Yang, Ram Nevatia

Abstract

We present a novel framework, Spatial Pyramid Attention Network (SPAN) for detection and localization of multiple types of image manipulations. The proposed architecture efficiently and effectively models the relationship between image patches at multiple scales by constructing a pyramid of local self-attention blocks. The design includes a novel position projection to encode the spatial positions of the patches. SPAN is trained on a generic, synthetic dataset but can also be fine tuned for specific datasets; The proposed method shows significant gains in performance on standard datasets over previous state-of-the-art methods.

Introduction

In an era where sophisticated image editing tools increasingly threaten the principle of "seeing is believing," the need for robust image forensics has never been greater. While human observers often fail to detect subtle object removals or clever copy-paste operations, these manipulations often leave behind pixel-level inconsistencies that tell a different story.

Existing image forensics tools frequently suffer from two primary limitations. First, many are specialized for a single type of manipulation (e.g., splicing) and fail to generalize to others like copy-move or object removal. Second, they often lack "spatial awareness," failing to model complex relationships between distant image patches. For example, models like ManTra-Net analyze features vertically across scales but often ignore horizontal spatial dependencies. SPAN (Spatial Pyramid Attention Network) addresses these challenges by modeling spatial relationships across multiple scales with high precision.

Methods

The SPAN framework is designed to localize image forgeries at the pixel level through a three-stage process: feature extraction, hierarchical attention propagation, and a final decision module.

The SPAN architecture featuring hierarchical local self-attention propagation.

Feature Extraction

The model utilizes a pre-trained backbone (derived from ManTra-Net) optimized for extracting features related to visual artifacts and noise patterns. These features serve as the foundation for identifying tampered regions.

Spatial Pyramid Attention (SPA) Block

The core innovation of SPAN is the use of a hierarchy of local self-attention blocks rather than standard global attention. Global attention is often computationally prohibitive and can obscure fine local details. SPAN employs "dilated" distances—similar to dilated convolutions—allowing the network to expand its receptive field across layers. This pyramid structure enables the comparison of distant patches while preserving the high-resolution details necessary for accurate masking.

Furthermore, SPAN introduces Position Projection (PP). Unlike standard Transformers that use additive positional embeddings, SPAN utilizes learnable matrix projections to explicitly represent relative spatial relationships. This approach is better suited for the geometric nature of image localization tasks.

Decision Module

The final stage consists of 2D convolutional layers that process the attention-rich features to generate a binary mask, predicting the probability of manipulation for every pixel.

Results and Discussion

SPAN was evaluated on several benchmark datasets, including NIST16, Columbia, Coverage, and CASIA, covering various manipulation types such as splicing, copy-move, and object removal.

Qualitative comparison of SPAN predictions against ground truth masks for various manipulation types.

Performance Gains

The model demonstrated superior generalization and accuracy compared to previous state-of-the-art methods:

  • Generalization: Without dataset-specific fine-tuning, SPAN achieved an 11.21% improvement in AUC over ManTra-Net on the Columbia dataset.
  • Fine-tuning: When fine-tuned, SPAN reached near-perfect scores, such as a 96.1% AUC on the challenging NIST16 dataset.
  • Robustness: The architecture proved resilient against common post-processing techniques like JPEG compression and Gaussian noise.

Key Insights

The researchers found that 3x3 local attention blocks provide an optimal balance between memory efficiency and full-image coverage. By explicitly modeling the "mutual relationship" between pixels through self-attention—rather than simple summation as in standard convolutions—SPAN effectively identifies the boundaries where authentic image distributions shift into tampered ones.

Conclusion

SPAN represents a significant advancement in digital forensics by combining hierarchical local attention with a novel position projection technique. While its effectiveness can decrease on very low-resolution images where subtle artifacts are lost, it provides a powerful, general-purpose tool for pixel-level manipulation localization. Future work may focus on optimizing the architecture for ultra-high-resolution imagery and real-time applications.

Summary updated

Abstract

We present a novel framework, Spatial Pyramid Attention Network (SPAN) for detection and localization of multiple types of image manipulations. The proposed architecture efficiently and effectively models the relationship between image patches at multiple scales by constructing a pyramid of local self-attention blocks. The design includes a novel position projection to encode the spatial positions of the patches. SPAN is trained on a generic, synthetic dataset but can also be fine tuned for specific datasets; The proposed method shows significant gains in performance on standard datasets over previous state-of-the-art methods.
BibTeX
@inproceedings{hu2020span,
  author = {Xuefeng Hu and Zhihan Zhang and Zhenye Jiang and Syomantak Chaudhuri and Zhenheng Yang and Ram Nevatia},
  title = {{SPAN: Spatial Pyramid Attention Network for Image Manipulation Localization}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=52A2Z_C6Wwm},
}
Figure