Abstract
As synthetic imagery evolves from 2D Generative Adversarial Networks (GANs) and Diffusion Models (DMs) toward 3D Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), traditional fake detection systems are becoming increasingly obsolete. This paper introduces FFiT (Fourier Frequency-based image Transformer), a multimodal detector designed to identify these sophisticated fakes. By leveraging a novel unsupervised pre-training strategy that accounts for the centrosymmetric properties of the Fourier spectrum, FFiT achieves state-of-the-art performance, effectively bridging the domain gap between traditional generative models and modern neural rendering techniques.
Introduction
The landscape of synthetic imagery is shifting. While artifacts from GANs and DMs are becoming well-understood, new technologies like NeRF and 3DGS present a fresh challenge. Unlike traditional models that synthesize pixels in 2D, these neural rendering methods reconstruct 3D scenes from real images and project them back into 2D. This process avoids many logical and semantic inconsistencies, such as "six-fingered hands," making the resulting images nearly imperceptible to existing detection systems.
The core issue is a significant "domain gap": detectors trained on GANs fail to generalize to the subtle spectral imprints left by 3D neural rendering. Research indicates that while these images appear perfect in the spatial domain, the rendering process leaves distinct signatures in the frequency domain that FFiT is designed to exploit.
Methods: The FFiT Architecture
FFiT utilizes a dual-branch architecture that fuses spatial and spectral information. The spatial branch employs a ViT-L14, while the frequency branch uses a ViT-B16 to analyze the Fourier magnitude spectrum of images.
Addressing Spectral Centrosymmetry
A key challenge in training the frequency branch is that standard Masked Autoencoders (MAE) fail on spectral data. The Fourier spectrum is centrosymmetric, meaning amplitudes at positive frequencies match those at negative frequencies. A standard MAE would allow the model to "cheat" by simply copying unmasked symmetric patches.
To overcome this, the authors introduced:
- A Novel Loss Function: A focal loss variant that balances different masking scenarios (e.g., when both a patch and its symmetric counterpart are masked).
- Dynamic Masking Ratios: Varying the amount of masking during training to force the model to learn global spectral structures rather than local patterns.
Multimodal Fusion and Adaptation
The spatial and frequency branches are fused using a Gated-Multimodal-Unit (GMU). To ensure parameter-efficient fine-tuning, the model employs AdaLoRA, allowing it to adapt to new types of fakes with minimal training data.
Datasets and Experimental Results
To evaluate FFiT, the authors constructed a large-scale database comprising nearly 300,000 images generated via NeRF and 3DGS, over 330,000 3D scene editing images, and frames from OpenAI’s Sora.
Performance Metrics
FFiT achieved an average precision (AP) of 92.81% and an AUROC of 91.19% across 11 different types of 3D scene generators, outperforming previous state-of-the-art models like FatFormer.
Generalization and Scaling
The study found that as a model’s representation capability increases, the distance between real and fake features in high-dimensional space grows. Larger models (like ViT) proved less sensitive to the difficulty of training samples compared to smaller architectures like ResNet. Furthermore, FFiT demonstrated superior "backward compatibility," maintaining high accuracy on traditional GAN and DM fakes while excelling at new neural rendering detection.
Conclusion
FFiT represents a significant step forward in securing the digital landscape against increasingly realistic synthetic media. By focusing on spectral imprints and addressing the unique symmetries of frequency data, the model provides a robust defense against 3D neural rendering artifacts. Future work will focus on reducing the computational cost of generating high-quality 3D datasets and ensuring robustness against adversarial attacks and heavy image compression common on social media platforms.
Summary updated