Investigating Human Factors in Image Forgery Detection

Parag S. Chandakkar, Baoxin Li

Abstract

As image manipulation techniques become increasingly sophisticated, understanding the human capacity for forgery detection is critical. This study investigates the "human factors" involved in identifying tampered images using eye-tracking technology. By analyzing gaze patterns and visual saliency, researchers from Arizona State University identified key drivers of human success and failure. The findings reveal that while humans are proficient at detecting unsophisticated "spliced" images, they are significantly outperformed by automated algorithms when faced with skillful forgeries integrated into naturally salient regions.

Introduction

In the current digital landscape, the proliferation of deepfakes and advanced image editing has made the "gold standard" of human visual perception more important than ever. However, despite our reliance on human judgment, there is a lack of quantitative data regarding the cognitive and visual processes involved in forgery detection. This research addresses this gap by examining how visual saliency and gaze patterns influence a person's ability to distinguish between authentic and tampered content.

Methodology

The study involved 24 participants who viewed 73 images from the CASIA v1.0 and v2.0 benchmarks. These datasets include "spliced" images (simple cut-and-paste) and more complex "tampered" images involving blurring and resizing.

Examples of image forgeries used in the study, including (a) spliced compositions and (b) tampered facial images.

Researchers categorized the images based on saliency changes:

  • Class 01: A previously non-salient region becomes salient in the forgery (e.g., adding a bright object to a dull background).
  • Class 11: A naturally salient region (like a face) is tampered with.

Eye-tracking data was used to calculate "Gaze-metrics," comparing fixations in forged versus naturally salient regions. A K-means clustering algorithm was then applied to these metrics to predict the difficulty level of each image based on how humans scanned them.

Key Insights: The Role of Saliency

The research highlights that visual saliency is the primary "tell" for human observers. If a forgery creates a new point of high contrast or interest (Class 01), humans are highly likely to detect it. Conversely, when a forgery is skillfully hidden within an existing salient region (Class 11), human accuracy drops precipitously.

The study also found that gaze patterns are task-driven. When specifically looking for forgeries, participants' eye movements differed significantly from natural viewing, with fixation duration and order serving as strong indicators of an image's perceived difficulty.

Results and Comparison

The overall human accuracy was recorded at 68.5%. However, a stark contrast emerged in the "difficult" category (Class 11), where human accuracy was only 22.22%, compared to 77.78% for an automated baseline algorithm. This suggests that while humans have intuitive advantages, machines are better at detecting subtle inconsistencies in spatial and contextual continuity.

Furthermore, the gaze-based K-means algorithm successfully categorized images into "easy" and "difficult" groups with 63.87% accuracy, demonstrating the predictive power of human eye movements in assessing forgery complexity.

Conclusion

This research provides a quantitative foundation for understanding human limitations in image forensics. The findings suggest that future automated detection systems could be improved by incorporating human-like gaze patterns, creating hybrid models that leverage both human intuition and machine precision.

Summary updated

BibTeX
@inproceedings{chandakkar2014investigating,
  author = {Parag S. Chandakkar and Baoxin Li},
  title = {{Investigating Human Factors in Image Forgery Detection}},
  booktitle = {Huevent@mm},
  year = {2014},
  doi = {10.1145/2660505.2660510},
}
Figure