Abstract
With the rise of AI-enabled Real-Time Deepfakes (RTDFs), the integrity of online video interactions has become a growing concern. RTDFs have now made it feasible to replace an imposter's face with their victim in live video interactions. Such advancement in deepfakes also coaxes detection to rise to the same standard. However, existing deepfake detection techniques are asynchronous and hence ill-suited for RTDFs. To bridge this gap, we propose a challenge-response approach that establishes authenticity in live settings. We focus on talking-head style video interaction and present a taxonomy of challenges that specifically target inherent limitations of RTDF generation pipelines. We evaluate representative examples from the taxonomy by collecting a unique dataset comprising eight challenges, which consistently and visibly degrades the quality of state-of-the-art deepfake generators. These results are corroborated both by humans and a new automated scoring function, leading to 88.6% and 80.1% AUC, respectively. The findings underscore the promising potential of challenge-response systems for explainable and scalable real-time deepfake detection in practical scenarios. We provide access to data and code at https://github.com/mittalgovind/GOTCHA-Deepfakes
1. Introduction: The Threat of Real-Time Deepfakes
As AI-enabled "Real-Time Deepfakes" (RTDFs) become more sophisticated, they pose a massive threat to the integrity of live video calls, from job interviews to high-stakes financial transfers. Traditional deepfake detection methods are largely passive and asynchronous; they analyze a video after it has been recorded and assume no interaction between the detector and the imposter.
This creates a "threat model" that favors the attacker, who can use offline resources to perfectly tune a model for a specific target. In a live setting, however, the imposter must maintain high-quality synthesis with zero latency. There is currently no established way to interactively "stress test" a live video feed to see if the face on the other end is real or a synthetic overlay.
2. The GOTCHA Framework
The proposed GOTCHA (a play on CAPTCHA) system shifts the burden of proof onto the imposter through a series of interactive challenges. By introducing a challenge, the "onus of consistency" shifts to the imposter. They must now solve complex computer vision problems—such as hand-face occlusion or extreme facial deformation—in milliseconds.
The framework exploits the inherent architectural weaknesses of deepfake generation pipelines. As shown in the pipeline above, RTDFs rely on a sequence of face detection, alignment, encoding/decoding, and blending. Challenges are designed to break specific components of this chain.
3. A Taxonomy of Challenges
The researchers developed a taxonomy of tasks designed to break specific components of the RTDF pipeline:
- Face Occlusion: Placing a hand or object (like a tissue or sunglasses) in front of the face. This often breaks the segmentation and alignment modules, causing the digital mask to flicker or disappear.
- Head Movement: Rapidly turning or nodding. This tests temporal coherence and the model's ability to handle extreme angles not present in the training data.
- Facial Deformation: Poking a cheek or stretching lips. Current auto-encoders struggle to map these physical distortions onto a target identity.
- Illumination: Using a phone flash or changing screen brightness. This forces the color correction and blending modules to recalculate in real-time, often resulting in visible boundary artifacts.
4. Experimental Setup and Dataset
Because no existing deepfake dataset included these specific interactive challenges, the authors created the GOTCHA Dataset, consisting of 56,247 videos derived from 47 unique participants. They tested the challenges against three major RTDF pipelines: LIA (reenactment), FSGAN (subject-agnostic swapping), and DFL (DeepFaceLab), the current state-of-the-art for high-fidelity swaps.
To move beyond human intuition, the authors built an automated detector using a 3D-ResNet18 architecture. It uses a contrastive loss function to learn a latent space where "real" video embeddings are clustered together, while "fake" videos are pushed far away. They also implemented "Compliance Detectors" to ensure the user actually performed the requested challenge.
5. Key Results and Insights
The study found that occlusion-based challenges (especially using hands) are the most effective. They consistently induce the most "human-visible" artifacts because they disrupt the fundamental geometry the deepfake relies on.
- Human Detection: In a study with 43 qualified evaluators, humans achieved an AUC of 88.62% and an accuracy of 81.2% when using the GOTCHA metric.
- Automated Detection: The in-house ML model achieved an AUC of 80.1%, significantly outperforming existing offline detectors (like SBI or FTCN) which hovered around 50-53% AUC on this specific dataset.
- Efficacy Ranking: There is a clear hierarchy in challenge effectiveness: No Challenge < Illumination < Head Movement < Facial Deformation < Occlusions.
- The Power of the Sequence: While a high-end model might pass a single challenge, a sequence of 4+ challenges acts as a monotonic boost to detection. Even an "adaptive" adversary saw its failure rate escalate to nearly 90% when faced with a sequence of tasks.
6. Conclusion
GOTCHA provides an inherently explainable detection method. Unlike "black-box" detectors, if a hand passes "through" a face or a mask flickers during a cheek-poke, a human auditor can immediately see why the system flagged the video as fake.
While the current dataset is heavily skewed toward the 21-29 age bracket, future work will focus on validating these challenges across a broader spectrum of skin tones and ages. Ultimately, GOTCHA demonstrates that a single 15-second engagement is often enough to unmask even sophisticated RTDFs.
Summary updated