HierCon: Hierarchical Contrastive Attention for Audio Deepfake Detection

Zhili Nicholas Liang, Soyeon Caren Han, Qizhou Wang, Christopher Leckie

Abstract

Audio deepfakes generated by modern TTS and voice conversion systems are increasingly difficult to distinguish from real speech, raising serious risks for security and online trust. While state-of-the-art self-supervised models provide rich multi-layer representations, existing detectors treat layers independently and overlook temporal and hierarchical dependencies critical for identifying synthetic artefacts. We propose HierCon, a hierarchical layer attention framework combined with margin-based contrastive learning that models dependencies across temporal frames, neighbouring layers, and layer groups, while encouraging domain-invariant embeddings. Evaluated on ASVspoof 2021 DF and In-the-Wild datasets, our method achieves state-of-the-art performance (1.93% and 6.87% EER), improving over independent layer weighting by 36.6% and 22.5% respectively. The results and attention visualisations confirm that hierarchical modelling enhances generalisation to cross-domain generation techniques and recording conditions.

Introduction

Modern Text-to-Speech (TTS) and Voice Conversion (VC) systems have reached a level of sophistication where synthetic speech is nearly indistinguishable from human voices. While self-supervised learning (SSL) models like XLS-R provide powerful multi-layer representations, current detection methods often treat these layers as independent silos, typically employing a simple weighted average. This approach suffers from two primary limitations: temporal blindness, which ignores that synthesis artifacts are often localized, and hierarchical neglect, which fails to model the relationships between acoustic, phonetic, and semantic layers.

Methodology

HierCon (Hierarchical Contrastive Attention) addresses these gaps through a multi-stage architecture designed to capture complex dependencies within SSL representations.

Three-Stage Hierarchical Attention

The framework processes the 24 layers of the XLS-R backbone through three distinct stages:

  1. Stage 1 (Temporal Attention): Learns to identify discriminative frames within each layer.
  2. Stage 2 (Intra-Group Attention): Models dependencies between neighboring layers within clusters (e.g., layers 1-8, 9-16, 17-24).
  3. Stage 3 (Inter-Group Attention): Aggregates these clusters to prioritize different levels of the hierarchy, such as low-level acoustics versus high-level semantics.
The HierCon architecture, illustrating the three-stage hierarchical attention mechanism and the contrastive learning heads.

Margin-Based Contrastive Learning

To enhance generalization across diverse deepfake generators, HierCon incorporates a margin-based contrastive loss. This encourages the model to learn domain-invariant features by clustering "real" and "fake" samples in a geometric space, preventing the model from overfitting to dataset-specific artifacts.

Experimental Evaluation

The model was trained on the ASVspoof 2019 Logical Access (LA) dataset and evaluated on three rigorous benchmarks:

  • ASVspoof 2021 LA: Focuses on logical access attacks with various codecs and compression.
  • ASVspoof 2021 DF (Deepfake): A massive dataset featuring over 100 different deepfake generation pipelines.
  • In-the-Wild (ITW): A dataset of real-world deepfake examples found online, representing the most difficult "wild" conditions.

Results and Discussion

HierCon established new state-of-the-art benchmarks across the board, measured by Equal Error Rate (EER). On the ASVspoof 2021 DF dataset, it achieved a 1.93% EER, a 36.6% relative improvement over the standard XLS-R baseline. On the In-the-Wild dataset, it achieved a 6.87% EER, representing a 22.5% improvement.

Visualizations of the attention mechanisms provide further insight into the model's decision-making process.

Visualization of attention weights across the three stages: (a) Temporal, (b) Intra-Group, and (c) Inter-Group.

Key observations include:

  • The Interaction Zone: Mid-level layers often carry the most discriminative information, suggesting that deepfake artifacts are most apparent in the transition from acoustic to semantic features.
  • Temporal Stability: Temporal attention consistently focuses on the central regions of audio signals, where synthesis artifacts are most persistent.
  • Regularization for Generalization: Ablation studies showed that while hierarchical attention provides the structural capacity to find artifacts, contrastive learning provides the necessary regularization to ignore dataset-specific noise.

Conclusion

HierCon demonstrates that hierarchical modeling of SSL layers, combined with contrastive regularization, significantly improves audio deepfake detection. The framework's ability to capture temporal and hierarchical dependencies allows it to generalize effectively to unseen generation techniques. Future work will focus on improving computational efficiency for real-time deployment and exploring cross-modal detection strategies.

Reference: Liang, Z. N., Han, S. C., Wang, Q., & Leckie, C. (2026). HierCon: Hierarchical Contrastive Attention for Audio Deepfake Detection. Proceedings of the ACM Web Conference 2026 (WWW ’26).

Summary updated

Abstract

Audio deepfakes generated by modern TTS and voice conversion systems are increasingly difficult to distinguish from real speech, raising serious risks for security and online trust. While state-of-the-art self-supervised models provide rich multi-layer representations, existing detectors treat layers independently and overlook temporal and hierarchical dependencies critical for identifying synthetic artefacts. We propose HierCon, a hierarchical layer attention framework combined with margin-based contrastive learning that models dependencies across temporal frames, neighbouring layers, and layer groups, while encouraging domain-invariant embeddings. Evaluated on ASVspoof 2021 DF and In-the-Wild datasets, our method achieves state-of-the-art performance (1.93% and 6.87% EER), improving over independent layer weighting by 36.6% and 22.5% respectively. The results and attention visualisations confirm that hierarchical modelling enhances generalisation to cross-domain generation techniques and recording conditions.
BibTeX
@inproceedings{liang2026hiercon,
  author = {Zhili Nicholas Liang and Soyeon Caren Han and Qizhou Wang and Christopher Leckie},
  title = {{HierCon: Hierarchical Contrastive Attention for Audio Deepfake Detection}},
  booktitle = {The Web Conference},
  year = {2026},
  doi = {10.1145/3774904.3792873},
}
Figure