Abstract
Recent advances in deep learning and computer vision have spawned a new class of media forgeries known as deepfakes, which typically consist of artificially generated human faces or voices. The creation and distribution of deepfakes raise many legal and ethical concerns. As a result, the ability to distinguish between deepfakes and authentic media is vital. While deepfakes can create plausible video and audio, it may be challenging for them to generate content that is consistent in terms of high-level semantic features, such as emotions. Unnatural displays of emotion, measured by features such as valence and arousal, can provide significant evidence that a video has been synthesized. In this paper, we propose a novel method for detecting deepfakes of a human speaker using the emotion predicted from the speaker's face and voice. The proposed technique leverages LSTM networks that predict emotion from audio and video LLDs. Predicted emotion in time is used to classify videos as authentic or deepfakes through an additional supervised classifier.
Introduction
Most deepfake detection methods focus on low-level digital artifacts—things like unusual eye blinking, inconsistent head poses, or "warping" traces left by generative models. However, as synthesis technology evolves, these technical glitches are being ironed out, leading to an arms race between forgers and forensic researchers.
The authors argue that we should look at semantic features instead. Humans communicate emotion through a complex, synchronized dance of facial muscles and vocal inflection. Deepfake generators, which often operate on a frame-by-frame basis or treat audio and video as separate streams, struggle to maintain this emotional coherence. This research investigates whether these "unnatural" displays of emotion can serve as a robust signature for forgery.
Methodology
The researchers developed a three-stage pipeline to turn emotional cues into a binary "Real or Fake" classification.
- Feature Extraction: Standard tools are used to extract Low-Level Descriptors (LLDs). For video, OpenFace tracks facial action units (e.g., brow raising). For audio, OpenSmile extracts acoustic parameters like Mel-Frequency Cepstrum Coefficients (MFCCs).
- Emotion Prediction: These LLDs are fed into a Long Short-Term Memory (LSTM) network trained to predict emotion in a continuous space using the Valence-Arousal model. Valence measures how positive or negative an emotion is, while Arousal measures intensity.
- Deepfake Detection: Finally, the predicted emotional time-series are analyzed by a classifier. The authors tested two approaches: a statistical method using handcrafted features (mean, std dev, correlation) and a "learned" method using a second, lightweight LSTM to analyze temporal evolution directly.
Key Insights and Results
The "Learned" LSTM approach significantly outperformed traditional statistical methods, achieving a staggering 99.5% accuracy with the multi-modal (Audio + Video) classifier. At a strict 5% False Alarm Rate (FAR), the system maintained a 100% detection rate.
Several key observations emerged from the data:
- The "Stateless" Flaw: Many deepfake algorithms transform faces frame-by-frame without temporal context, leading to "choppy" emotional transitions that LSTMs easily detect.
- Audio is the Weak Link: Current audio-faking techniques are significantly worse at mimicking emotional range than video-faking techniques. As shown in Figure 5(a), fake audio tends to have a much narrower "emotional span" compared to authentic speech.
- Identity vs. Expression: Deepfakes often transfer a target's identity while preserving the source's expression. This facial emotion might look "real" in isolation, but it often becomes decoupled from the synthesized voice, creating a detectable semantic mismatch.
Conclusion and Limitations
While highly effective, the authors highlight a significant hurdle: Bias. The datasets used (SEMAINE and DFDC) consist primarily of Caucasian subjects with British or American accents. Because emotional expression is deeply tied to culture and language, models trained on this data might not generalize well to other populations. Future work will focus on expanding these datasets to include more diverse cultural expressions and investigating "anti-forensic" measures where forgers specifically attempt to mimic emotional consistency.
Summary updated