Abstract
Different from traditional sentence-level audio deepfake detection (ADD), partial audio deepfake detection (PADD) requires frame-level positioning of the location of fake speech. While some progress has been made in this area, leveraging semantic information from audio, especially named entities, remains an underexplored aspect. To this end, we propose NE-PADD, a novel method for Partial Audio Deepfake Detection (PADD) that leverages named entity knowledge through two parallel branches: Speech Name Entity Recognition (SpeechNER) and PADD. The approach incorporates two attention aggregation mechanisms: Attention Fusion (AF) for combining attention weights and Attention Transfer (AT) for guiding PADD with named entity semantics using an auxiliary loss. Built on the PartialSpoof-NER dataset, experiments show our method outperforms existing baselines, proving the effectiveness of integrating named entity knowledge in PADD.
Introduction
Audio deepfake detection (ADD) has traditionally focused on utterance-level classification. However, Partial Audio Deepfake Detection (PADD) presents a more granular challenge: identifying specific segments within an otherwise authentic recording that have been replaced with synthetic speech. These "spliced" segments often target high-value semantic information, such as named entities (names, locations, dates), to alter the meaning of a statement while maintaining acoustic consistency.
Current PADD models primarily rely on detecting acoustic artifacts at segment boundaries. NE-PADD addresses the limitation of ignoring semantic context by integrating Named Entity Recognition (NER) knowledge into the detection process, hypothesizing that semantic "high-value targets" are more likely to be tampered with.
Methodology
The NE-PADD framework utilizes a dual-branch architecture to bridge acoustic detection and semantic knowledge.
Dual-Branch Architecture
- SpeechNER Branch: This branch acts as a semantic guide, identifying named entities directly from the audio signal using an encoder and self-attention mechanism to produce entity-level embeddings.
- PADD Branch: This branch focuses on acoustic feature extraction, utilizing a pre-trained Wav2Vec 2.0 front-end and a ResNet-1D to generate frame-level representations.
Attention Aggregation Mechanisms
To integrate these branches, the authors propose two mechanisms:
- Attention Fusion (AF): A gated attention mechanism that combines the attention weights from the SpeechNER and PADD branches. This tells the model to prioritize acoustic analysis in regions identified as semantically significant.
- Attention Transfer (AT): A teacher-student paradigm where the SpeechNER branch guides the PADD branch during training via a KL-Divergence loss, encouraging the detection model to learn semantic distributions.
The final backend classifier, comprising Transformer encoders and a BiLSTM, processes these aggregated features to predict the probability of spoofing for each 20ms frame.
Experimental Results
The researchers introduced the PartialSpoof-NER dataset, augmenting the original PartialSpoof dataset with transcriptions from OpenAI's Whisper and entity labels from Stanford’s Stanza.
As shown in the results, NE-PADD-AF achieves a state-of-the-art Equal Error Rate (EER) of 7.89%, a substantial improvement over previous benchmarks like WBD (11.59%) and TDL (14.64%). Ablation studies further confirmed that the attention aggregation mechanisms consistently improved the performance of other baseline models, such as BAM, reducing its EER from 18.21% to 10.61%.
Conclusion
NE-PADD demonstrates that semantic information is a powerful prior for partial audio deepfake detection. By aligning "what" is being said with "how" it sounds through attention aggregation, the model achieves superior robustness. Future work aims to extend this framework to multilingual contexts and improve detection granularity for sub-word level manipulations.
Reference: Xian, H., Liu, R., Sisman, B., & Li, H. (2025). NE-PADD: Leveraging Named Entity Knowledge for Robust Partial Audio Deepfake Detection via Attention Aggregation. APSIPA ASC.
Summary updated