Weakly-supervised Audio Temporal Forgery Localization via Progressive Audio-language Co-learning Network

Junyan Wu, Wenbo Xu, Wei Lu, Xiangyang Luo, Rui Yang, Shize GuoIJCAI 2025 · DOI

TL;DR

This paper introduces LOCO, a novel weakly-supervised method for pinpointing manipulated segments within audio. It tackles the challenge of limited annotations by using an audio-language co-learning approach to align semantic and temporal features, followed by a progressive refinement strategy to enhance forgery detection.

What Problem Does This Paper Address?

The rise of AI-generated content (AIGC) has led to increasingly sophisticated audio manipulation techniques, particularly "partial forgery manipulation" (PFM). These attacks subtly alter small portions of audio, making them difficult to detect and localize, especially when only utterance-level labels (indicating whether the whole audio is manipulated or not) are available. Traditional methods for temporal forgery localization (TFL) require fine-grained, frame-level annotations, which are costly and impractical to obtain in real-world scenarios. This paper addresses the critical need for effective TFL methods that can operate under weak supervision, using only utterance-level labels.

Methods

LOCO employs a two-pronged approach:

  • Audio-Language Co-learning (A2LC) Module: This module is designed to capture semantic inconsistencies introduced by PFM. It comprises:
  • Temporal Forgery Attention (TFA) Adapter: Extracts temporal forgery cues from audio features (obtained via an ASR model like XLS-R).
  • Prompt-Enhanced Forgery Feature (PFF) Adapter: Dynamically incorporates "forgery-aware prompts" (derived from utterance-level labels and learnable prompts) into audio features. This injects global semantic context.
  • Co-learning Strategy: Aims to align and leverage complementary information from both TFA and PFF features using a multi-instance learning (MIL) loss and KL divergence.
  • Progressive Refinement Strategy (PRS): Since MIL focuses on segment-level predictions, PRS is introduced to refine frame-level forgery features. It works in two stages:
  • Stage 1: The base model is trained using MIL and KL divergence losses.
  • Stage 2: Pseudo frame-level labels are generated from the proposals of Stage 1. These labels are then used with a semantic-based contrastive learning (SCL) loss to further amplify the semantic distinction between real and fake audio segments.

Datasets

The paper evaluates LOCO on three datasets:

  • LAV-DF: A dataset for temporal forgery localization.
  • HAD: Another dataset for audio forgery detection and localization.
  • AV-Deepfake-1M: A large-scale, LLM-driven audio-visual deepfake dataset. For this dataset, a subset was constructed due to the lack of testing labels in the audio modality.

Key Insights and Ideas

  • Bridging the Annotation Gap: The core idea is that by aligning audio semantics with language-based prompts, we can infer forgery cues even without frame-level labels.
  • Prompting for Forgery: Using learnable prompts, inspired by prompt-based learning in NLP, effectively injects contextual and semantic information into audio features, guiding the model towards identifying manipulations.
  • Self-Supervision for Refinement: Generating pseudo frame-level labels from initial proposals and using contrastive learning is a powerful way to refine forgery detection without relying on ground truth at the frame level.
  • Co-learning Synergy: The combination of temporal attention and prompt-enhanced features in the A2LC module creates a synergistic effect, leading to more robust forgery representations.

Key Results

  • Detection Performance: LOCO significantly outperforms state-of-the-art weakly-supervised methods in frame-level detection, reducing EER by substantial margins (e.g., 3.91% on HAD, 17.26% on LAV-DF).
  • Localization Performance: LOCO achieves superior temporal forgery localization performance, outperforming previous best weakly-supervised methods by up to 26.43% in mAP on the HAD and LAV-DF datasets. It also shows competitive results against some fully-supervised methods at lower IoU thresholds.
  • Ablation Studies: The paper demonstrates the effectiveness of each component: TFA and PFF adapters are crucial for feature extraction and enhancement, respectively. The co-learning strategy and the progressive refinement strategy (especially the SCL loss) significantly boost performance.

Limitations and Future Work

  • Localization Thresholds: The paper notes that localization precision decreases with higher IoU thresholds. The best mAP was achieved at a threshold of 0.5, suggesting room for improvement in very precise localization.
  • Long-Duration Samples: The method shows strong performance but acknowledges that challenges remain for detecting forgery in very long audio samples and achieving precise localization at high IoU thresholds (e.g., > 0.9).
  • Future Directions: The authors suggest that improving localization under more precise IoU thresholds and for long-duration samples are important areas for future research.

Summary updated

BibTeX
@inproceedings{wu2025weakly,
  author = {Junyan Wu and Wenbo Xu and Wei Lu and Xiangyang Luo and Rui Yang and Shize Guo},
  title = {{Weakly-supervised Audio Temporal Forgery Localization via Progressive Audio-language Co-learning Network}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/65},
}
Figure