A Multi-Stream Fusion Network for Image Splicing Localization

Maria Siopi, Giorgos Kordopatis-Zilos, Polychronis Charitidis, Ioannis Kompatsiaris, Symeon PapadopoulosZenodo 2022 · DOI · OpenAlex

Introduction

Image splicing—the process of pasting a region from one image into another—is a fundamental technique in digital forgery. As editing tools become more sophisticated, these manipulations are increasingly difficult to detect with the naked eye. While deep learning has significantly advanced image forensics, many existing models rely exclusively on raw RGB data. This reliance creates a vulnerability: when images are resized or recompressed for the web, the subtle visual traces of an edit are often "washed out."

Traditional forensic methods utilize handcrafted signals, such as noise patterns or frequency analysis, to identify mathematical inconsistencies. However, simply stacking these signals as extra channels in a standard neural network (early fusion) often fails to capture the nuanced relationship between visual content and forensic artifacts. This paper introduces a Multi-Stream Fusion Network designed to process these distinct data types in parallel, allowing for more precise pixel-level localization of spliced regions.

Methodology

The proposed architecture is built on an encoder-decoder framework, characterized by three primary innovations:

Parallel Multi-Stream Encoders

Instead of a single input pipeline, the model employs multiple independent encoder streams. One stream is dedicated to the raw RGB image, while others are specialized for handcrafted signals:

  • Discrete Cosine Transform (DCT): Used to identify frequency inconsistencies often introduced during JPEG compression.
  • Splicebuster: A noise-based feature extractor that identifies statistical anomalies in the image grain.

By providing dedicated encoders for these signals, the network learns the specific "language" of frequency and noise before attempting to relate them to the visual scene.

Late Fusion and Learnable Decoding

Features from the independent streams are only combined (concatenated) at the "bottleneck" of the network—the deepest point of feature extraction. This late fusion approach prevents the dominant RGB signals from drowning out more subtle forensic traces early in the process. Following fusion, a learnable decoder uses trainable transpose convolutional layers to upsample the data, producing a high-resolution "heat map" that pinpoints the tampered pixels.

Selective Skip Connections

To maintain sharp localization, the authors implemented skip connections. Crucially, these connections are only applied to the RGB encoder. The researchers discovered that including skip connections for handcrafted streams actually degraded performance, likely because those signals represent global statistical anomalies rather than the fine-grained spatial edges required for sharp mask reconstruction.

Experimental Results

The model was evaluated against several industry-standard benchmarks, including CASIA (v1 and v2), IFS-TC, and the Columbia dataset.

Performance Metrics

The Multi-Stream (MS) approach consistently outperformed traditional multi-channel (MC) architectures and existing state-of-the-art (SOTA) models:

  • CASIA Dataset: The model achieved an AUC of 0.898, significantly surpassing other high-end models such as ManTra-Net (0.665) and SPAN.
  • IFS-TC Dataset: The configuration utilizing the Splicebuster signal (MS-SB) reached an AUC of 0.776, maintaining a substantial lead over the nearest competitor (0.680).

Key Insights

  • Noise vs. Frequency: The Splicebuster (noise-based) signal generally proved more effective than the DCT (frequency-based) signal. This suggests that while modern software is adept at hiding frequency artifacts, it struggles to unify the underlying "noise" of two disparate images.
  • The Necessity of Fine-Tuning: The research highlighted a significant performance jump when the model was fine-tuned on specific datasets. This indicates that forensic traces are highly dependent on the source camera and post-processing history, making a universal "one-size-fits-all" detector a continuing challenge.

Conclusion

The Multi-Stream Fusion Network demonstrates that specialized, parallel processing of forensic signals is superior to simple channel stacking. By allowing the network to independently analyze RGB, noise, and frequency data before fusion, the model achieves high-precision localization even in challenging datasets. Future work will focus on improving the model's generalizability to ensure robust performance across diverse image sources without the need for extensive fine-tuning.

Summary updated

Abstract

In this paper, we address the problem of image splicing localization with a multi-stream network architecture that processes the raw RGB image in parallel with other handcrafted forensic signals. Unlike previous methods that either use only the RGB images or stack several signals in a channel-wise manner, we propose an encoder-decoder architecture that consists of multiple encoder streams. Each stream is fed with either the tampered image or handcrafted signals and processes them separately to capture relevant information from each one independently. Finally, the extracted features from the multiple streams are fused in the bottleneck of the architecture and propagated to the decoder network that generates the output localization map. We experiment with two handcrafted algorithms, i.e., DCT and Splicebuster. Our proposed approach is benchmarked on three public forensics datasets, demonstrating competitive performance against several competing methods and achieving state-of-the-art results, e.g., 0.898 AUC on CASIA.
BibTeX
@inproceedings{siopi2022multi,
  author = {Maria Siopi and Giorgos Kordopatis-Zilos and Polychronis Charitidis and Ioannis Kompatsiaris and Symeon Papadopoulos},
  title = {{A Multi-Stream Fusion Network for Image Splicing Localization}},
  booktitle = {Zenodo},
  year = {2022},
  doi = {10.5281/zenodo.7322720},
}

Loading related content...

Figure