Deep Image Comparator: Learning To Visualize Editorial Change

Alexander Black, Tu Bui, Hailin Jin, Vishy Swaminathan, John P. CollomosseWorkshops @ CVPR 2021 · OpenReview

Abstract

We present a novel architecture for comparing a pair of images to identify image regions that have been subjected to editorial manipulation. We first describe a robust near-duplicate search, for matching a potentially manipulated image circulating online to an image within a trusted database of originals. We then describe a novel architecture for comparing that image pair, to localize regions that have been manipulated to differ from the retrieved original. The localization ignores discrepancies due to benign image transformations that commonly occur during online redistribution. These include artifacts due to noise and recompression degradation, as well as out-of-place transformations due to image padding, warping, and changes in size and shape. Robustness towards out-of-place transformations is achieved via the end-to-end training of a differentiable warping module within the comparator architecture. We demonstrate effective retrieval and comparison of benign transformed and manipulated images, over a dataset of millions of photographs.

Introduction

In the contemporary digital landscape, identifying image manipulations is a critical task for combating misinformation. Images shared online frequently undergo "benign transformations"—resizing, cropping, JPEG compression, or padding—which complicate the detection of actual editorial changes, such as the addition or removal of objects. Traditional forensic tools often fail to distinguish between these technical artifacts and intentional manipulations. Furthermore, identifying the original source image from a massive database is a significant challenge when the query image has been warped or degraded. This paper introduces a two-stage system designed to retrieve original images and accurately localize editorial changes while remaining robust to benign transformations.

Methodology

The proposed pipeline consists of two primary stages: a robust near-duplicate search and a deep image comparator.

To find the original version of a suspicious image, the system encodes images into compact 256-dimensional embeddings using a ResNet50 backbone. This model is trained with a modified contrastive loss, treating both benignly transformed and editorially manipulated versions of an image as positive matches. This ensures that the search remains effective even when the query image has been significantly altered. For efficiency at scale, these embeddings are binarized into 128-bit hashes, allowing for rapid searching across millions of images.

Deep Image Comparator (ICN)

Once a candidate original is retrieved, the Image Comparator Network (ICN) performs a detailed analysis to localize changes. The architecture, illustrated in Figure 1, comprises two main components:

  1. Geometric Alignment Module: This module utilizes a modified RAFT (optical flow) architecture and a Differentiable Warping Unit (DWU) to align the query image with the retrieved original. This step is crucial for "undoing" any geometric distortions like padding or warping, ensuring that subsequent comparisons are spatially accurate.
  2. Prediction Module: A two-stream CNN extracts features from the aligned image pair. These features are fused to produce a 3-way classification (benignly changed, manipulated, or distinct) and a 7x7 manipulation heatmap that highlights edited regions.
The Deep Image Comparator architecture, featuring a Geometric Alignment Module and a Prediction Module for localization and classification.

Experimental Results

The system was evaluated using the PSBattles dataset, which contains approximately 10,000 original images and over 100,000 community-created manipulations. To test scalability, 2 million distractor images from Adobe Stock were added to the search database.

Retrieval and Localization Performance

The integrated two-stage approach significantly outperformed standalone search methods. The system achieved an IR@1 (Image Retrieval at rank 1) of 0.8745, compared to 0.7138 for search alone. In terms of localization, the ICN achieved an Intersection over Union (IoU) of 0.551 for its heatmaps, a substantial improvement over baseline methods (e.g., a ResNet-based difference model achieved only 0.243).

Qualitatively, the heatmaps produced by the ICN are highly interpretable. As shown in Figure 2, the model successfully identifies manipulated regions while ignoring the benign blurring and warping present in the "Manipulated" query image.

Comparison of a manipulated query image against the original, showing the ICN's ability to localize editorial changes despite benign transformations.

Efficiency and Human Study

The comparison stage is highly efficient, taking approximately 4ms per pair on a standard GPU. A user study further validated the model's utility, with participants preferring the ICN's heatmaps 85.2% of the time over competing methods, citing their clarity and robustness to noise.

Conclusion and Future Work

The Deep Image Comparator provides a robust framework for identifying editorial manipulations in real-world scenarios where images are frequently degraded by benign transformations. By integrating geometric alignment directly into the forensic pipeline, the model achieves superior localization accuracy. Future research will focus on increasing the resolution of the manipulation heatmaps to detect smaller edits and extending the system to handle generative AI content and video sequences.

Summary updated

Abstract

We present a novel architecture for comparing a pair of images to identify image regions that have been subjected to editorial manipulation. We first describe a robust near-duplicate search, for matching a potentially manipulated image circulating online to an image within a trusted database of originals. We then describe a novel architecture for comparing that image pair, to localize regions that have been manipulated to differ from the retrieved original. The localization ignores discrepancies due to benign image transformations that commonly occur during online redistribution. These include artifacts due to noise and recompression degradation, as well as out-of-place transformations due to image padding, warping, and changes in size and shape. Robustness towards out-of-place transformations is achieved via the end-to-end training of a differentiable warping module within the comparator architecture. We demonstrate effective retrieval and comparison of benign transformed and manipulated images, over a dataset of millions of photographs.
BibTeX
@inproceedings{black2021deep,
  author = {Alexander Black and Tu Bui and Hailin Jin and Vishy Swaminathan and John P. Collomosse},
  title = {{Deep Image Comparator: Learning To Visualize Editorial Change}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=9yz1AfkXjsTc},
}
Figure