Abstract
Image forgery detection faces significant challenges as generative AI and diffusion models evolve, often rendering traditional forensic fingerprints obsolete. We present Forgery MAE (FMAE), a novel framework that shifts the detection paradigm from hunting for specific forgery artifacts to identifying deviations from "natural" image characteristics. By applying the Lottery Ticket Hypothesis to a Masked Autoencoder (MAE), we identify a subnetwork sensitive to forgeries, allowing for sparse fine-tuning that preserves the model's inherent knowledge of natural images. Combined with a Mixture of Experts Noise Extractor (MoENE), FMAE achieves superior generalization across diverse and unseen forgery techniques, including sophisticated AI-generated fakes.
Introduction
Image forgery detection is a perpetual cat-and-mouse game. Traditional forensic tools are designed to find specific fingerprints—such as JPEG compression artifacts or pixel inconsistencies—left behind by editing software. However, as generative models advance, these fingerprints are increasingly suppressed or absent. Most existing models struggle with out-of-distribution (OOD) performance; they excel at detecting known forgery types but fail when faced with novel techniques.
The authors argue that instead of focusing solely on the "signals of faking," models should leverage the "signals of authenticity." By learning the universal characteristics of natural, untouched images, a system can more effectively identify anomalies that indicate tampering, regardless of the specific forgery method used.
Methods
The FMAE framework introduces a multi-stage approach to robust forgery detection and localization:
- MAE Backbone: The system utilizes a Masked Autoencoder (MAE) pre-trained on ImageNet. MAEs are inherently adept at understanding image semantics and high-frequency details due to their reconstruction-based training objective.
- Lottery Ticket Sparse Fine-Tuning (LTSFT): To avoid "forgetting" general natural image features during task-specific training, the authors apply the Lottery Ticket Hypothesis. They identify a specific subnetwork (the "winning ticket") within the MAE that is most sensitive to forgery by comparing parameter changes when exposed to forged versus authentic images. Only these parameters (approximately 70%) are updated during fine-tuning.
- Mixture of Experts Noise Extractor (MoENE): To capture subtle noise traces at various scales, FMAE integrates three distinct noise extraction algorithms: SRM filters, Bayar convolutions, and NoisePrint++. A self-attention mechanism adaptively weights these "experts" based on the input image.
- Multi-Source Forgery Awareness (MSFFT): Extracted noise features are injected into specific layers of the MAE encoder, amplifying the model's sensitivity to tampering.
Datasets and Evaluation
The model was rigorously evaluated on a comprehensive suite of benchmarks. Training was conducted on Casiav2.0 and FantasticReality. Testing spanned traditional sets focusing on splicing and cloning (e.g., Casiav1.0, NIST16, IMD2020) and modern Deep Generative Model (DGM) sets like AutoSplice and OpenForensic, which feature AI-generated and diffusion-based forgeries.
Key Results and Insights
FMAE demonstrated superior performance and generalization capabilities:
- Localization Performance: On the IMD2020 dataset, which contains real-world forgeries, FMAE achieved an F1 score of 0.491, significantly outperforming previous state-of-the-art methods like DiffForensics (0.415).
- Robustness to AI Fakes: The model reached a 0.950 AUC on the AutoSplice dataset, proving its efficacy against modern generative AI tampering.
- Preservation of Priors: The core insight is that sparse fine-tuning prevents the model from losing its "common sense" about natural images. This preservation acts as a critical benchmark for spotting anomalies, as evidenced by t-SNE visualizations showing cleaner separation between real and fake feature clusters.
Conclusion and Future Work
FMAE represents a significant step forward in forgery detection by prioritizing the preservation of natural image priors. While highly effective, future work could focus on automating the selection of the optimal fine-tuning ratio for different image domains (e.g., satellite or medical imagery) and extending this "naturalness prior" approach to the temporal domain for video deepfake detection. The current two-phase training process also presents opportunities for computational optimization.
Summary updated