DeCLIP: Decoding CLIP Representations for Deepfake Localization

Stefan Smeu, Elisabeta Oneata, Dan OneataWACV 2025

TL;DR

This paper introduces DeCLIP, a novel method for pinpointing manipulated regions in images by leveraging the powerful, pre-trained representations from CLIP. DeCLIP demonstrates significant improvements in generalizing to unseen deepfake generation methods, particularly excelling in the challenging domain of latent diffusion models.

What Problem Does This Paper Address?

The proliferation of generative AI has made it increasingly difficult to distinguish between real and manipulated images. While detecting fully synthetic images has seen progress, accurately localizing partial manipulations (e.g., altering a person's mouth in a video) remains a significant challenge. Existing deepfake detectors often struggle to generalize to new, unseen generative models because they rely on subtle "fingerprints" left by the generator, which can vary greatly depending on the model's architecture, training data, and even random seeds. This lack of generalization is a critical vulnerability, as attackers can exploit it by using novel generation techniques.

Methods

DeCLIP builds upon the success of self-supervised models like CLIP, which have shown remarkable generalization capabilities for image-level deepfake detection. The core idea is to utilize the rich visual features extracted by a frozen CLIP image encoder and then train a convolutional decoder to map these features to a manipulation mask.

Here's a breakdown of the approach:

  • Encoder: The paper explores using pre-trained CLIP image encoders (specifically ViT-L/14 and ResNet-50) at various layers to extract feature maps.
  • Decoder: A convolutional neural network (CNN) is employed as a decoder. This decoder takes the CLIP features and progressively upscales them to produce a pixel-wise prediction of the manipulated region. The authors investigate different decoder architectures (linear, attention-based, and convolutional) and sizes, finding that larger convolutional decoders yield better results.
  • Training: The model is trained using a binary cross-entropy loss to predict a mask where values close to 1 indicate a manipulated pixel and values close to 0 indicate an authentic pixel.

Datasets

The primary dataset used for training and evaluation is the Dolos dataset. Dolos is specifically designed for studying local manipulations and includes images inpainted using four different methods: two diffusion models (P2 and LDM) and two GANs. This dataset is crucial because it disentangles various aspects of image generation, allowing for a thorough analysis of generalization across different manipulation types and generators. The authors also mention using a subset of Dolos for specific experiments, like the LDM-inpainted images. For broader generalization testing, they also utilize the MS COCO dataset (inpainted with Stable Diffusion) and the AutoSplice dataset (manipulated with DALL-E 2).

Key Insights and Ideas

  • CLIP's potential for localization: While CLIP features are known for image-level detection, this paper shows they can be effectively adapted for localization when paired with a suitable decoder.
  • The challenge of local manipulations: Local manipulations are harder for CLIP features to capture compared to global ones, as the subtle changes might be diluted in the broader feature representation.
  • Decoder matters: The choice and size of the decoder are critical. A simple linear decoder is insufficient; a more powerful convolutional decoder is necessary to effectively leverage CLIP's features for precise localization.
  • LDM's unique fingerprint: Latent Diffusion Models (LDMs) present a particularly challenging case. Their inpainting process, which occurs in the latent space and involves a VAE upscaling step, leaves artifacts throughout the image. This makes localization harder but also offers a unique opportunity for robust generalization.
  • LDM training for generalization: Surprisingly, training DeCLIP on LDM-generated data leads to superior out-of-domain generalization compared to training on other generator types. This suggests that the LDM's inherent "fingerprint" or the way it forces the model to focus on semantic information is more transferable.
  • Backbone complementarity: Different CLIP backbones (ViT-L/14 and ResNet-50) exhibit complementary strengths, with ViT-L/14 excelling at higher layers for OOD performance and ResNet-50 at lower layers. Combining features from both can further boost performance.

Key Results

  • Improved generalization: DeCLIP significantly outperforms existing methods in out-of-domain (OOD) generalization, achieving a 50% relative improvement in OOD IoU compared to Patch Forensics.
  • High accuracy on LDM: DeCLIP successfully localizes manipulations in LDM-inpainted images, a task where other methods struggle.
  • Competitive performance: DeCLIP achieves competitive in-domain (ID) performance, often matching or exceeding specialized methods like Patch Forensics.
  • Decoder impact: Larger convolutional decoders (e.g., conv-20) lead to more precise and smoother localization masks compared to linear or attention-based decoders.
  • Backbone insights: ViT-L/14 and ResNet-50 backbones show distinct layer preferences for ID vs. OOD performance, with ViT-L/14 being better for OOD at higher layers and ResNet-50 at lower layers.

Limitations and Future Work

  • Dataset bias: While Dolos is a well-constructed dataset, it primarily focuses on faces. The authors acknowledge the need to test generalization on more diverse image content, which they begin to address with experiments on MS COCO.
  • Computational cost: Larger decoders, while beneficial, increase the model's parameter count and computational cost.
  • Future directions: The paper suggests exploring different self-supervised models beyond CLIP, investigating the impact of various data augmentation strategies (though LDM's inherent artifacts seem to provide a strong form of augmentation), and further analyzing the transferability of LDM-specific fingerprints. The complementarity of different backbones also opens avenues for ensemble methods.

Summary updated

BibTeX
@inproceedings{smeu2025declip,
  author = {Stefan Smeu and Elisabeta Oneata and Dan Oneata},
  title = {{DeCLIP: Decoding CLIP Representations for Deepfake Localization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure