Tampering Detection and Localization in Images from Social Networks: A CBIR Approach

Cédric Maigrot, Ewa Kijak, Ronan Sicre, Vincent ClaveauICIAP 2017 · DOI

TL;DR

This paper tackles the challenge of detecting and localizing image tampering in social networks by proposing a novel Content-Based Image Retrieval (CBIR) approach. By leveraging large online image databases, their system retrieves similar images and then compares them to identify and pinpoint manipulated regions, offering a more robust solution for diverse forgery types than traditional methods.

What Problem Does This Paper Address?

The proliferation of social networks has led to an unprecedented volume of information sharing, including images. Unfortunately, this also means a surge in the dissemination of misinformation, often through manipulated images. These manipulations can range from subtle aesthetic edits to outright fabrications, making it crucial to verify image authenticity. Traditional image forensics methods often struggle with the sheer variety of tampering techniques and the lack of original image metadata (like EXIF data) in social media contexts. This paper addresses the need for a generic and effective system that can not only detect but also precisely locate these manipulations, even when dealing with complex forgeries and limited prior information.

Methods

The proposed method is a two-step process:

  1. Content-Based Image Retrieval (CBIR) for Initial Ranking: The core idea is to leverage external information by searching for similar images. The system uses a Convolutional Neural Network (CNN), specifically the VGG vd19, to extract deep features from the query image. These features are then used to query a large database of images, aiming to retrieve candidates that are visually similar, even if tampered. A cosine similarity measure is employed, and a filtering step using geometric verification (SURF features and RANSAC) is applied to refine the retrieved candidates and select the most promising match.
  2. Tampering Localization: Once a potential match is found, the system focuses on identifying inconsistencies between the query image and the retrieved candidate. This is achieved by matching local descriptors (SURF) between the two images. Outlier matches, which are spatially clustered, are identified as indicators of tampering. A Kernel Density Estimation (KDE) technique is used to create a density map of these outliers, which is then thresholded to generate a binary mask highlighting the suspicious regions.

Datasets

The authors introduce two new datasets and utilize existing ones for evaluation:

  • Reddit: A dataset built from real-world images from a Photoshop challenge on Reddit, featuring a variety of forgeries including copy-paste and other manipulations not commonly found in existing datasets.
  • Synthetic: An artificially generated dataset designed to test the system's robustness to specific forgery types and parameters (size, rotation, blur, etc.).
  • MICC-F600: A dataset from image forensics containing original and forged images with realistic copy-move attacks.
  • MediaEval (ME): A dataset derived from tweets, including challenging fake images with their original counterparts, as well as images with typical collage, cropping, or text insertions.
  • Holidays: Used as a source of negative queries (pristine images) to evaluate the system's ability to avoid false positives.
  • Distractors: A large collection of images from hoax detection websites and Twitter to further challenge the CBIR system.

Key Insights and Ideas

  • Leveraging External Data is Key: The paper strongly advocates for using external image databases (via CBIR) as a crucial component for social network image forensics, moving beyond traditional methods that rely solely on the image itself.
  • Generic Approach for Diverse Forgeries: The CBIR-based approach offers a more generic solution compared to specialized methods that target only specific types of tampering (e.g., copy-move).
  • Clustering Outlier Matches for Localization: Identifying and clustering outlier local feature matches between a query and a similar image is an effective way to pinpoint tampered regions, even for complex manipulations.
  • Social Network Images are Challenging: The authors highlight that social network images, with their inherent noise, compression, and diverse editing styles, pose significant challenges for standard CBIR systems.

Key Results

  • The proposed CBIR system achieved a high accuracy of 91.91% with filtering and 81.08% without filtering at a cosine similarity threshold of T=0.9.
  • The tampering localization achieved near-perfect results on the Reddit and ME datasets, with TPR of 81.37% and FPR of 5.14% across all datasets.
  • The system demonstrated robustness to various transformations like rotation and illumination changes.
  • Performance was notably affected by large forged areas and blurring, especially in the CBIR retrieval step.

Limitations and Future Work

  • CBIR Sensitivity to Large Forgeries: The CBIR system struggled when the tampered area was very large relative to the image size, leading to missed original images in some cases.
  • Blurring Impact: Blurring of the entire image, even if original, could lead to misclassification by the localization step.
  • Future Work: The authors suggest improving the CBIR system's performance on social network images, which are inherently challenging due to their diverse nature. Further research could focus on enhancing robustness to these specific image characteristics.

Summary updated

BibTeX
@inproceedings{maigrot2017tampering,
  author = {Cédric Maigrot and Ewa Kijak and Ronan Sicre and Vincent Claveau},
  title = {{Tampering Detection and Localization in Images from Social Networks: A CBIR Approach}},
  booktitle = {ICIAP},
  year = {2017},
  doi = {10.1007/978-3-319-68560-1\_67},
}
Figure