The Affective Bridge: Unifying Feature Representations for Speech Deepfake Detection

Yupei Li, Chenyang Lyu, Longyue Wang, Weihua Luo, Kaifu Zhang, Björn W. Schuller

Abstract

Speech deepfake detection has been widely explored using low-level acoustic descriptors. However, each study tends to select different feature sets, making it difficult to establish a unified representation for the task. Moreover, such features are not intuitive for humans to perceive, as the distinction between bona fide and synthesized speech becomes increasingly subtle with the advancement of deepfake generation techniques. Emotion, on the other hand, remains a unique human attribute that current deepfake generator struggles to fully replicate, reflecting the gap toward true artificial general intelligence. Interestingly, many existing acoustic and semantic features have implicit correlations with emotion. For instance, speech features recognized by automatic speech recognition systems often varies naturally with emotional expression. Based on this insight, we propose a novel training framework that leverages emotion as a bridge between conventional deepfake features and emotion-oriented representations. Experiments on the widely used FakeOrReal and In-the-Wild datasets demonstrate consistent and substantial improvements in accuracy, up to approximately 6% and 2% increases, respectively, and in equal error rate (EER), showing reductions of up to about 4% and 1%, respectively, while achieving comparable results on ASVspoof2019. This approach provides a unified training strategy for all features and interpretable feature direction for deepfake detection while improving model performance through emotion-informed learning.

Introduction

The rapid advancement of generative AI has made distinguishing between bona fide and synthesized speech increasingly difficult. While current deepfake detection (DFD) methods achieve high performance, they often face a trade-off between specialization and versatility. Standard approaches typically involve either full fine-tuning, which can "break" a model's original utility (such as speech-to-text), or using frozen features, which may fail to capture the subtle artifacts of synthetic speech.

This research introduces EmoBridge, a training framework that leverages emotion as a "bridge" to enhance DFD. The core insight is that while deepfake generators can mimic voices, they struggle to authentically reproduce the complex acoustic nuances of human emotion. By aligning speech representations with emotional cues, EmoBridge improves detection accuracy without destroying the model's original semantic utility.

Methodology: The EmoBridge Framework

The EmoBridge pipeline consists of two distinct phases designed to be feature-agnostic and non-destructive to the underlying pre-trained models.

The two-phase EmoBridge framework for emotion-guided representation alignment and deepfake detection.

Phase I: Emotion-Guided Representation Alignment

Instead of training for deepfake detection immediately, a lightweight emotion prediction head is attached to various speech encoders (e.g., Whisper for low-level features, SpeechT5 for semantic features, and WavLM for speaker identity). This head is trained on emotion recognition tasks using established datasets. This process forces the encoders to align their internal representations toward emotional cues, effectively "tuning" them to recognize human affective patterns without altering their fundamental structure.

Phase II: Deepfake Detection

Once aligned, the emotion-guided encoders are frozen. Their outputs are then fed into a simple downstream classifier (such as an SVM or MLP) to perform the final classification between bona fide and synthetic speech. This approach treats emotion recognition as a representation alignment objective rather than just an auxiliary task.

Experimental Results and Analysis

The framework was evaluated across several benchmark datasets, including ASVSpoof2019 LA, FakeOrReal (FoR), and In-the-Wild (ITW).

Performance Gains

The results demonstrate consistent improvements across all tested architectures. On the FoR dataset, accuracy increased by up to 6%. Notably, using ASR-based features, the Equal Error Rate (EER) dropped significantly from 0.082 to 0.045. The framework proved to be truly feature-agnostic, benefiting both hand-crafted features (openSMILE) and deep representations alike.

Explainability and Attention

Analysis of the model's internal mechanics reveals why EmoBridge is effective. Attention map visualizations show that "bridged" models develop new focus regions where affective cues are likely captured, while still retaining the original attention patterns of the pre-trained model.

Mean attention comparison between the original model and the EmoBridge-enhanced model.

Furthermore, t-SNE visualizations confirmed that after the EmoBridge process, speech samples clustered more distinctly by emotion while maintaining speaker-identity groupings. This suggests that the model successfully learned to prioritize emotional authenticity as a key discriminator for deepfake detection.

Conclusion

EmoBridge provides a unified training strategy that improves DFD performance through emotion-informed learning. Importantly, the method is non-destructive; tests on Speaker Verification (SV) showed negligible performance drops, meaning the features remain useful for their original purposes. While gains on highly imbalanced datasets like ASVSpoof2019 were more modest, the overall success of EmoBridge highlights the "generative gap" in emotional authenticity as a critical frontier for securing audio communications.

Summary updated

Abstract

Speech deepfake detection has been widely explored using low-level acoustic descriptors. However, each study tends to select different feature sets, making it difficult to establish a unified representation for the task. Moreover, such features are not intuitive for humans to perceive, as the distinction between bona fide and synthesized speech becomes increasingly subtle with the advancement of deepfake generation techniques. Emotion, on the other hand, remains a unique human attribute that current deepfake generator struggles to fully replicate, reflecting the gap toward true artificial general intelligence. Interestingly, many existing acoustic and semantic features have implicit correlations with emotion. For instance, speech features recognized by automatic speech recognition systems often varies naturally with emotional expression. Based on this insight, we propose a novel training framework that leverages emotion as a bridge between conventional deepfake features and emotion-oriented representations. Experiments on the widely used FakeOrReal and In-the-Wild datasets demonstrate consistent and substantial improvements in accuracy, up to approximately 6% and 2% increases, respectively, and in equal error rate (EER), showing reductions of up to about 4% and 1%, respectively, while achieving comparable results on ASVspoof2019. This approach provides a unified training strategy for all features and interpretable feature direction for deepfake detection while improving model performance through emotion-informed learning.
BibTeX
@misc{li2025affective,
  author = {Yupei Li and Chenyang Lyu and Longyue Wang and Weihua Luo and Kaifu Zhang and Björn W. Schuller},
  title = {{The Affective Bridge: Unifying Feature Representations for Speech Deepfake Detection}},
  year = {2025},
  eprint = {2512.11241},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.11241},
}
Figure