Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries

Jawadul H. Bappy, Cody Simons, Lakshmanan Nataraj, B. S. Manjunath, Amit K. Roy-Chowdhury

TL;DR

This paper introduces a novel deep learning architecture for precisely locating manipulated regions within images at a pixel level. By combining spatial features from an encoder-decoder network with frequency-domain resampling features processed by an LSTM, the model achieves high accuracy in detecting various image forgeries, outperforming existing methods.

What Problem Does This Paper Address?

The proliferation of sophisticated image editing tools makes it increasingly difficult to distinguish between authentic and manipulated images. Content-changing manipulations like copy-move, splicing, and object removal can significantly alter an image's meaning, potentially spreading misinformation. While classifying an image as tampered is one challenge, precisely identifying where the manipulation occurred at a pixel level is even more difficult, as these alterations often leave subtle or no visible clues. Existing methods often focus on specific manipulation types or struggle to generalize, highlighting the need for a robust, general-purpose localization framework.

Methods

The proposed "Hybrid LSTM and Encoder-Decoder Architecture" tackles this by integrating two key types of information:

  • Spatial Features (Encoder-Decoder): A convolutional encoder-decoder network is used to capture the visual context, shapes, and spatial relationships within an image. This is a standard approach in semantic segmentation, but here it's adapted to focus on identifying manipulated regions.
  • Frequency-Domain Features (LSTM with Resampling Features): To detect subtle artifacts introduced by image manipulations (like JPEG compression, upsampling, rotation, or shearing), the paper extracts "resampling features" from image patches. These features are then fed into a Long Short-Term Memory (LSTM) network. The LSTM is crucial for learning the temporal correlations and transitions between these features across different image patches, effectively analyzing frequency-domain characteristics that are often indicative of tampering.

These two streams of information are fused before being processed by a decoder, which then generates a pixel-wise probability map, ultimately leading to a binary mask indicating manipulated regions. The use of Hilbert curves to order patches for the LSTM is a novel technique to preserve spatial locality.

Datasets

  • Synthesized Dataset: The authors created a large dataset of approximately 65,000 manipulated images, primarily focusing on object splicing. They leveraged the MS-COCO dataset for segmented objects and combined them with images from DRESDEN and NIST'16 to generate these synthetic forgeries. This dataset is noted as being significantly larger and higher resolution than many existing benchmarks.
  • NIST'16 [2]: A challenging dataset containing copy-clone, removal, and splicing manipulations.
  • IEEE Forensics Challenge [1]: Another dataset used for evaluation.
  • COVERAGE [86]: Used for copy-move manipulation evaluation.

Key Insights and Ideas

  • Fusion is Key: The paper strongly advocates for combining both spatial (visual context) and frequency-domain (resampling artifacts) information. Neither alone is sufficient for robust, general-purpose forgery detection.
  • LSTM for Temporal Artifacts: Using LSTMs to process resampling features from image patches is an effective way to capture subtle, non-visual artifacts that signal manipulation. The LSTM's ability to learn sequential dependencies is vital here.
  • Global Context Matters: Processing the entire image as input, rather than just patches, allows the network to leverage global context, leading to more precise localization, especially for smaller manipulated regions.
  • Hilbert Curves for Spatial Coherence: Employing Hilbert curves to order image patches for the LSTM is an innovative way to maintain spatial relationships, improving the LSTM's ability to learn meaningful correlations.

Key Results

  • High Accuracy: The proposed "LSTM-EnDec" model achieved state-of-the-art pixel-wise accuracy, reaching 94.80% on the NIST'16 dataset and 91.19% on the IEEE Forensics Challenge dataset.
  • Outperforms Baselines: The model significantly outperformed baseline methods like FCN and SegNet, showing a 20.52% and 11.84% improvement on NIST'16, respectively.
  • Robustness: The model demonstrated strong performance across different datasets and manipulation types, including copy-clone, splicing, and removal.
  • AUC Scores: Achieved AUC scores of 0.7936 (NIST'16), 0.7577 (IEEE Forensics), and 0.7124 (COVERAGE), indicating good discriminative ability.

Limitations and Future Work

  • Data Hungry: Deep learning models, including this one, are data-intensive. The creation of a large synthesized dataset was a significant effort to address this.
  • Patch Size Trade-off: The choice of patch size for resampling feature extraction involves a trade-off between detecting subtle artifacts (larger patches) and localizing small manipulated regions (smaller patches).
  • Future Work: The paper suggests that the contour approximation method used to derive bounding boxes from segmentation masks could be further refined. The authors also imply that their approach could be extended to other types of manipulations or more complex forgery scenarios.

Summary updated

Abstract

With advanced image journaling tools, one can easily alter the semantic meaning of an image by exploiting certain manipulation techniques such as copy clone, object splicing, and removal, which mislead the viewers. In contrast, the identification of these manipulations becomes a very challenging task as manipulated regions are not visually apparent. This paper proposes a high-confidence manipulation localization architecture that utilizes resampling features, long short-term memory (LSTM) cells, and an encoder–decoder network to segment out manipulated regions from non-manipulated ones. Resampling features are used to capture artifacts, such as JPEG quality loss, upsampling, downsampling, rotation, and shearing. The proposed network exploits larger receptive fields (spatial maps) and frequency-domain correlation to analyze the discriminative characteristics between the manipulated and non-manipulated regions by incorporating the encoder and LSTM network. Finally, the decoder network learns the mapping from low-resolution feature maps to pixel-wise predictions for image tamper localization. With the predicted mask provided by the final layer (softmax) of the proposed architecture, end-to-end training is performed to learn the network parameters through back-propagation using the ground-truth masks. Furthermore, a large image splicing dataset is introduced to guide the training process. The proposed method is capable of localizing image manipulations at the pixel level with high precision, which is demonstrated through rigorous experimentation on three diverse datasets.
BibTeX
@article{bappy2019hybrid,
  author = {Jawadul H. Bappy and Cody Simons and Lakshmanan Nataraj and B. S. Manjunath and Amit K. Roy-Chowdhury},
  title = {{Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=OIYybJm-OW9},
}
Figure