Abstract
Deepfake detection in real-world scenarios faces significant challenges due to severe class imbalance and rapid distribution shifts in generative techniques. This paper presents a robust framework that leverages the CLIP foundation model for feature extraction, combined with a novel composite loss function—integrating Conditional Value-at-Risk (CVaR) and Area Under the Curve (AUC) loss—and Sharpness-Aware Minimization (SAM). Our approach prioritizes challenging minority-class samples and improves generalization across unseen generative models. Evaluated on the DFWild-Cup dataset, the proposed method achieves an AUC of 0.969 and an accuracy of 91.37%, significantly outperforming standard cross-entropy baselines.
Introduction
As AI-generated media becomes increasingly sophisticated, the need for reliable detection tools grows. However, most existing detectors are trained on balanced datasets, failing to account for the "real world" where authentic content vastly outnumbers fakes. This class imbalance often leads models to over-predict the majority class. Furthermore, the rapid evolution of generative models, such as Latent Diffusion Models, creates distribution shifts that render detectors trained on older GAN-based methods ineffective. We propose a framework specifically designed to handle these imbalances and shifts by focusing on the most difficult and rare examples.
Methodology
The proposed framework consists of four integrated components designed for robustness and generalization.
Feature Extraction and Augmentation
Instead of training a vision backbone from scratch, we utilize a frozen CLIP ViT-L/14 image encoder. CLIP’s pre-training on diverse image-text pairs provides a rich semantic understanding that generalizes well to various generative artifacts. To prevent overfitting, we apply Latent Feature Augmentation by injecting controlled Gaussian noise into the extracted CLIP features, forcing the classifier to learn more robust characteristics.
Composite Loss Function and Optimization
The core of our method is a composite loss function combining Conditional Value-at-Risk (CVaR) and Weighted AUC Loss.
- CVaR: This component focuses training on the "hardest" samples—those with the highest loss—ensuring the model learns to distinguish subtle fakes from real images.
- Weighted AUC Loss: This optimizes the model to rank fake images higher than real ones, a strategy more effective for imbalanced data than simple classification.
For optimization, we employ Sharpness-Aware Minimization (SAM). SAM identifies "flat" regions in the loss landscape, ensuring that small perturbations in model parameters do not significantly degrade performance, thereby enhancing generalization to unseen data.
Experimental Results
We evaluated the framework on the DFWild-Cup Dataset, which aggregates eight standard benchmarks including Celeb-DF, FaceForensics++, and DFDC. The training set featured 42,690 real and 219,470 fake samples.
As shown in the results, our method significantly outperforms the baseline across all metrics. The high AUC (0.969) and F1-score (0.913) demonstrate the model's ability to maintain a strong balance between precision and recall despite the data skew. Ablation studies confirmed that the combination of CVaR and AUC loss is critical for this performance gain. Notably, we found that a simple 3-layer MLP classifier outperformed deeper architectures when paired with the powerful CLIP features, suggesting that minimizing classifier complexity helps prevent overfitting.
Conclusion
This work demonstrates a principled approach to deepfake detection in imbalanced, real-world environments. By combining foundation models with ranking-based loss functions and sharpness-aware optimization, we achieve state-of-the-art performance and improved robustness. Future work will explore adapting these methods to AI-generated audio detection and addressing scenarios where the class distribution is unknown or dynamic.
Code available at: https://github.com/Purdue-M2/SP_CUP
Summary updated