Abstract
We present a novel architecture for comparing a pair of images to identify image regions that have been subjected to editorial manipulation. We first describe a robust near-duplicate search, for matching a potentially manipulated image circulating online to an image within a trusted database of originals. We then describe a novel architecture for comparing that image pair, to localize regions that have been manipulated to differ from the retrieved original. The localization ignores discrepancies due to benign image transformations that commonly occur during online redistribution. These include artifacts due to noise and recompression degradation, as well as out-of-place transformations due to image padding, warping, and changes in size and shape. Robustness towards out-of-place transformations is achieved via the end-to-end training of a differentiable warping module within the comparator architecture. We demonstrate effective retrieval and comparison of benign transformed and manipulated images, over a dataset of millions of photographs.
Introduction
In the contemporary digital landscape, identifying image manipulations is a critical task for combating misinformation. Images shared online frequently undergo "benign transformations"—resizing, cropping, JPEG compression, or padding—which complicate the detection of actual editorial changes, such as the addition or removal of objects. Traditional forensic tools often fail to distinguish between these technical artifacts and intentional manipulations. Furthermore, identifying the original source image from a massive database is a significant challenge when the query image has been warped or degraded. This paper introduces a two-stage system designed to retrieve original images and accurately localize editorial changes while remaining robust to benign transformations.
Methodology
The proposed pipeline consists of two primary stages: a robust near-duplicate search and a deep image comparator.
Robust Near-Duplicate Search
To find the original version of a suspicious image, the system encodes images into compact 256-dimensional embeddings using a ResNet50 backbone. This model is trained with a modified contrastive loss, treating both benignly transformed and editorially manipulated versions of an image as positive matches. This ensures that the search remains effective even when the query image has been significantly altered. For efficiency at scale, these embeddings are binarized into 128-bit hashes, allowing for rapid searching across millions of images.
Deep Image Comparator (ICN)
Once a candidate original is retrieved, the Image Comparator Network (ICN) performs a detailed analysis to localize changes. The architecture, illustrated in Figure 1, comprises two main components:
- Geometric Alignment Module: This module utilizes a modified RAFT (optical flow) architecture and a Differentiable Warping Unit (DWU) to align the query image with the retrieved original. This step is crucial for "undoing" any geometric distortions like padding or warping, ensuring that subsequent comparisons are spatially accurate.
- Prediction Module: A two-stream CNN extracts features from the aligned image pair. These features are fused to produce a 3-way classification (benignly changed, manipulated, or distinct) and a 7x7 manipulation heatmap that highlights edited regions.
Experimental Results
The system was evaluated using the PSBattles dataset, which contains approximately 10,000 original images and over 100,000 community-created manipulations. To test scalability, 2 million distractor images from Adobe Stock were added to the search database.
Retrieval and Localization Performance
The integrated two-stage approach significantly outperformed standalone search methods. The system achieved an IR@1 (Image Retrieval at rank 1) of 0.8745, compared to 0.7138 for search alone. In terms of localization, the ICN achieved an Intersection over Union (IoU) of 0.551 for its heatmaps, a substantial improvement over baseline methods (e.g., a ResNet-based difference model achieved only 0.243).
Qualitatively, the heatmaps produced by the ICN are highly interpretable. As shown in Figure 2, the model successfully identifies manipulated regions while ignoring the benign blurring and warping present in the "Manipulated" query image.
Efficiency and Human Study
The comparison stage is highly efficient, taking approximately 4ms per pair on a standard GPU. A user study further validated the model's utility, with participants preferring the ICN's heatmaps 85.2% of the time over competing methods, citing their clarity and robustness to noise.
Conclusion and Future Work
The Deep Image Comparator provides a robust framework for identifying editorial manipulations in real-world scenarios where images are frequently degraded by benign transformations. By integrating geometric alignment directly into the forensic pipeline, the model achieves superior localization accuracy. Future research will focus on increasing the resolution of the manipulation heatmaps to detect smaller edits and extending the system to handle generative AI content and video sequences.
Summary updated