TL;DR
This paper introduces a novel single-channel distance-based source separation (DSS) model optimized for mobile GPUs, significantly improving performance in challenging outdoor environments. By incorporating a linear relation-aware self-attention mechanism and leveraging TensorFlow Lite GPU delegates, the model achieves substantial gains in energy efficiency and inference speed, making advanced audio processing feasible on mobile devices.
What Problem Does This Paper Address?
The paper tackles the challenge of separating individual sound sources from a single audio channel, a task known as single-channel source separation. Specifically, it focuses on Distance-Based Source Separation (DSS), which leverages the spatial distance of sound sources from the microphone to aid in separation. While existing DSS methods have shown promise, they have largely been confined to controlled indoor environments. Outdoor settings present a much tougher challenge due to factors like high ambient noise, wind, and unpredictable acoustic conditions, which degrade the performance of traditional models. Furthermore, the increasing use of mobile devices for content creation (e.g., vlogs) necessitates efficient and fast audio processing on these platforms, which are often resource-constrained.
Methods
The core of the proposed solution is a new DSS model architecture designed for mobile GPUs. Key components include:
- Two-Stage Conformer Blocks: These blocks are designed to capture both temporal and frequency dependencies in the audio signal, a standard practice in modern speech processing.
- Linear Relation-Aware Self-Attention (RSA): This is a novel adaptation of self-attention. Traditional RSA can be computationally expensive (quadratic complexity with sequence length). The authors propose a linear version, inspired by EfficientAttention and Rotary Positional Embedding (RoPE). This linear RSA aims to maintain context awareness and improve performance while drastically reducing computational cost, making it suitable for mobile deployment.
- TensorFlow Lite GPU Delegate: To maximize performance on mobile devices, the model is optimized to run on mobile GPUs using TensorFlow Lite. This involves careful selection of layers supported by the GPU delegate and ensuring consistent tensor shapes.
The model's training involves a weighted sum of magnitude, spectrogram, and time losses, common in speech enhancement tasks.
Datasets
The paper utilizes a combination of existing datasets for training and evaluation:
- Training Data: CSTR-VCTK, AISHELL, and AI-Hub datasets were used for source speakers and background noises. This dataset comprises approximately 5,000 hours of speech and 1,000 hours of noisy audio from various environments.
- Testing Data: The wsj0 dataset was used for source speakers. Additionally, an internal noise dataset of 30 hours was used.
- Simulated Data: For simulating outdoor and indoor environments, the Pyroomacoustics toolbox was employed to generate audio with realistic impulse responses, varying room dimensions, microphone positions, RT60 values, wall materials, and source distances.
Key Insights and Ideas
- Outdoor DSS is a Significant Challenge: The paper highlights that existing DSS models struggle in outdoor environments due to increased noise and unpredictable acoustics, underscoring the need for specialized approaches.
- Linear RSA for Mobile Efficiency: The adoption of a linear RSA mechanism is a crucial innovation. It demonstrates that computational efficiency can be achieved without a significant sacrifice in performance, a critical factor for on-device AI.
- Mobile GPU Optimization is Key: The successful integration with TensorFlow Lite GPU delegates shows the practical viability of deploying complex audio processing models on mobile hardware, offering substantial speedups and energy savings.
- Combined Outdoor/Indoor Training is Superior: Training the model on a dataset that includes both outdoor and indoor environments leads to better overall performance compared to models trained solely on indoor data, suggesting that exposure to diverse acoustic conditions is beneficial.
Key Results
The proposed model, when deployed on a mobile GPU, achieved remarkable improvements:
- Energy Efficiency: A 20x increase in energy efficiency compared to mobile CPUs.
- Inference Speed: A 10x faster inference speed compared to mobile CPUs.
- Real-Time Factor (RTF): On a mobile GPU, the model achieved an RTF of 0.06, a significant improvement over mobile CPU performance (1.32 RTF).
- Performance Gains: The paper reports improved separation quality (measured by SI-SDRi) in both indoor and outdoor simulated environments, particularly when trained on a mixed dataset. For instance, in outdoor environments, the model trained on a 60:40 indoor-outdoor split showed better performance than one trained solely on indoor data.
Limitations and Future Work
- Permutation Ambiguity: The paper acknowledges that for very near sounds (UR-0), distinguishing between estimated near and far sounds can be challenging, even for humans. This "permutation ambiguity" is a known issue in source separation.
- Future Work: The authors suggest exploring hierarchical embeddings that can represent distance and speaker information more effectively. They also note that if bidirectional LSTM layers were fully supported by TFLite GPU delegates, models like MTF-DPRNN could become more competitive for DSS.
Summary updated