Av-Sepformer: Cross-Attention Sepformer for Audio-Visual Target Speaker Extraction

Jiuxin Lin, Xinyu Cai, Heinrich Dinkel, Jun Chen, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Zhiyong Wu, Yujun Wang, Helen Meng

Introduction

Target Speaker Extraction (TSE) is a critical task in speech processing, aimed at isolating a specific individual's voice from a "cocktail party" environment where multiple speakers and background noises overlap. While audio-only methods have made significant strides, they often struggle when speakers have similar vocal characteristics. Visual cues, such as lip movements, provide a robust, noise-invariant reference to identify the target speaker. However, integrating these two modalities effectively remains a challenge due to the fundamental differences in their temporal resolutions.

This paper introduces AV-SepFormer, a novel dual-scale transformer architecture that leverages cross-attention and specialized positional encoding to bridge the gap between audio and visual data, setting a new state-of-the-art in the field.

The Challenge: Temporal Granularity Mismatch

A primary hurdle in audio-visual fusion is the mismatch in temporal granularity. Audio features are typically processed at high frequencies (100–200 frames per second) to capture the nuances of speech, whereas video is recorded at much lower rates (typically 25 frames per second).

Traditional approaches often address this by up-sampling the video features to match the audio rate. However, this simple interpolation ignores the inherent structural differences between the signals and can lead to sub-optimal alignment. AV-SepFormer takes a different approach: it restructures the audio processing to align with the natural cadence of the visual data.

The AV-SepFormer Architecture

AV-SepFormer builds upon the successful SepFormer (Symmetric Encoder-Decoder Transformer) framework, adapting it for multi-modal interaction through three key technical innovations:

1. Synchronized Chunking

Instead of processing the audio as a continuous high-speed stream, the model employs a synchronized chunking strategy. The audio features are divided into a specific number of chunks that exactly match the number of available video frames. This ensures that each segment of audio is paired with a corresponding visual frame, creating a structural foundation for precise cross-modal alignment.

2. The Tri-Module Separator

The core of the model is a separator composed of three interleaved modules that handle different aspects of the signal:

  • IntraTransformer: This module focuses on the fine-grained details within each individual audio chunk, capturing short-term acoustic patterns.
  • CrossModalTransformer: This module utilizes a cross-attention mechanism to fuse visual features with the audio chunks. By using the visual signal as a query, the model dynamically learns which parts of the visual information (lip movements) are most relevant to the current audio segment.
  • InterTransformer: This module models the long-term temporal context across the different chunks, ensuring consistency and tracking the speaker over the duration of the recording.

3. 2D Positional Encoding (2DPos)

Standard transformers use 1D positional encoding to maintain sequence order. However, because AV-SepFormer treats data in a 3D structure (Feature Dimension × Chunk Length × Number of Chunks), 1D encoding is insufficient. The authors implemented a 2D Positional Encoding that tracks positions both within a chunk and across chunks. This allows the attention mechanism to navigate the complex 2D grid of audio-visual data more effectively, leading to sharper extraction results.

Experimental Results

The model was rigorously evaluated on the VoxCeleb2 dataset for 2-speaker mixtures, as well as LRS3 and TCD-TIMIT for cross-domain robustness.

Performance Benchmarks

AV-SepFormer achieved a new state-of-the-art on the VoxCeleb2 benchmark:

  • SI-SDR: 12.130 dB (an improvement of 0.9 dB over the previous best model, MuSE, and 1.8 dB over AV-ConvTasNet).
  • PESQ: 2.313, indicating significantly higher perceptual quality and naturalness in the extracted speech compared to previous architectures.

Ablation Studies

The importance of the model's unique features was confirmed through ablation testing. Removing the Cross-Attention (CA) mechanism resulted in a 0.4 dB drop in SI-SDR, while removing the 2D Positional Encoding led to a 0.1 dB decrease. These results validate that the structural alignment and the sophisticated attention mechanism are the primary drivers of the model's success.

Conclusion

AV-SepFormer demonstrates that the key to effective audio-visual fusion lies in respecting the natural structure of the data rather than forcing one modality to mimic the sampling rate of another. By synchronizing audio chunks with video frames and employing a dual-scale transformer with 2D positional encoding, the model achieves superior extraction performance and speech quality.

Future research may focus on reducing the computational complexity inherent in transformer-based models and exploring the model's performance in more complex, multi-talker environments beyond two-speaker mixtures.

Summary updated

BibTeX
@inproceedings{lin2023av,
  author = {Jiuxin Lin and Xinyu Cai and Heinrich Dinkel and Jun Chen and Zhiyong Yan and Yongqing Wang and Junbo Zhang and Zhiyong Wu and Yujun Wang and Helen Meng},
  title = {{Av-Sepformer: Cross-Attention Sepformer for Audio-Visual Target Speaker Extraction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10094306},
}
Figure