TL;DR
Instead of just flagging a deepfake as "Fake," what if we could reverse the process to see the original face underneath? This paper introduces IDRetracor, a novel visual forensics framework designed for "face retracing"—reconstructing the original target identity from a face-swapped deepfake image. By dynamically adapting to multiple unknown face-swapping methods, IDRetracor achieves high-fidelity identity recovery, providing actionable, traceable evidence for digital forensics.
What Problem Does This Paper Address?
Current deepfake countermeasures are heavily reliant on binary detection: they tell you whether an image is real or fake, but nothing more. For high-profile victims of malicious face-swapping (like celebrities or politicians), a simple "Fake" label is insufficient. It lacks traceability and fails to provide concrete evidence needed for accountability or legal enforcement.
To bridge this gap, the authors propose a new forensic task: face retracing. The goal is to take a forged face-swapped image and map it back to the original "target" face (the victim whose body/attributes were used).
Prior attempts at this inverse mapping problem (such as DRNet) suffered from two fatal flaws: they could only handle target identities they had already seen during training, and they were tailored to a single, specific face-swapping algorithm. In the real world, forensic investigators don't know which deepfake tool a perpetrator used, and they routinely encounter entirely unseen victims.
Methods
IDRetracor overcomes these limitations using a dynamic, mapping-aware architecture that adapts to the unique artifacts left behind by different deepfake generators.
The framework consists of two primary components:
- Mapping Resolver: Built on a ResNet18 backbone, this component analyzes the input fake face to estimate the underlying face-swapping method. Instead of performing a hard classification, it outputs a continuous weight vector ($\Omega$). It utilizes a novel fusion regularization technique (inspired by label smoothing) to ensure the weights are non-binary, allowing the model to blend strategies for mixed or unknown generation methods.
- Mapping-aware Convolutions: Instead of standard vanilla convolutions, the reconstruction network (a modified 3-layer UNet) uses dynamically recombined kernels. It contains $n$ groups of candidate kernels. Guided by the weight vector from the Mapping Resolver, these kernels are linearly combined on the fly ($\delta_{final} = \sum \omega_i \delta_i$) to handle both the common and unique features of various face-swapping methods.
The network is trained using a joint loss function:
- Pixel-wise Reconstruction Loss ($L_{rec}$): Ensures structural and color consistency with the ground truth target face.
- Identity Loss ($L_{id}$): Leverages a pre-trained ArcFace model to maximize the cosine similarity between the feature vectors of the retraced face and the true target face.
- Resolving Loss ($L_{res}$): Optimizes the Mapping Resolver.
Datasets
Standard deepfake benchmarks like Celeb-DF-v2 and FaceForensics++ are poorly suited for face retracing because they contain too few target identities paired with any single source identity, and their exact generation methods are often undisclosed.
To train IDRetracor, the authors crafted a massive new target-fake dataset using VGGFace2 as the base. They selected specific source subjects and generated face-swaps against thousands of distinct target identities using four state-of-the-art methods: SimSwap, InfoSwap, HifiFace, and E4S.
- Training set: 150,000 target-fake pairs per source subject (spanning 4,636 target IDs).
- Testing set: 10,000 pairs (spanning 1,301 target IDs). Crucially, there is zero overlap between the target identities used in training and testing.
- Identity Preservation: They also mixed in real "source-source" pairs to ensure that if the network is fed a real image, it simply reconstructs the input without altering it.
Key Insights and Ideas
- Identity Residuals are Inevitable: The core premise of the paper is that existing deepfake models are "black boxes" that fail to perfectly erase the target's identity. Subtle, implicit identity information (residuals) always remains embedded in the facial artifacts.
- Soft Blending over Hard Decisions: Perpetrators often use hybrid or unknown techniques. By predicting a continuous weight vector rather than a single discrete class, IDRetracor can interpolate between known inversion spaces to tackle completely unseen face-swapping variations.
Key Results
IDRetracor significantly outperforms vanilla UNet baselines trained on single methods (VU-S) or multiple methods (VU-M):
- Identity Similarity: On the testing set of entirely unseen target identities, IDRetracor achieved an ArcFace similarity score of 0.6520 and a CosFace score of 0.7011, outperforming the multi-method baseline by over 20%.
- Forensic Retrieval: In an identity retrieval test against a gallery of 1,000 faces, the retraced images achieved a Top-1 matching accuracy of 85.70% and a Top-5 accuracy of 96.00%.
- Semantic Consistency: When the authors took the retraced faces and face-swapped them back using the original source faces, the resulting images matched the initial deepfakes with a mean ArcFace similarity of 0.88, proving high semantic fidelity.
Limitations and Future Work
Despite its strength, IDRetracor has notable limitations:
- Source Identity Dependence: The model must be trained specifically for a designated source identity. If a new celebrity needs protection, a new target-fake dataset must be generated and the model retrained for that specific individual.
- Extreme Information Loss: If a face-swapping method completely obliterates the target attributes or leaves zero identity residuals (violating the paper's core premises), the model's performance degrades significantly.
Future Directions: The authors aim to explore more robust backbone architectures and find adaptive methods to make the framework entirely source-agnostic, removing the need to retrain the model for each unique source subject.
Summary updated