Abstract
This paper introduces ClearerVoice-Studio, an open-source, AI-powered speech processing toolkit designed to bridge cutting-edge research and practical application. Unlike broad platforms like SpeechBrain and ESPnet, ClearerVoice-Studio focuses on interconnected speech tasks of speech enhancement, separation, super-resolution, and multimodal target speaker extraction. A key advantage is its state-of-the-art pretrained models, including FRCRN with 3 million uses and MossFormer with 2.5 million uses, optimized for real-world scenarios. It also offers model optimization tools, multi-format audio support, the SpeechScore evaluation toolkit, and user-friendly interfaces, catering to researchers, developers, and end-users. Its rapid adoption attracting 3000 GitHub stars and 239 forks highlights its academic and industrial impact. This paper details ClearerVoice-Studio's capabilities, architectures, training strategies, benchmarks, community impact, and future plan. Source code is available at https://github.com/modelscope/ClearerVoice-Studio.
Introduction
Real-world audio environments are inherently complex, often plagued by a combination of background noise, overlapping speakers, and reverberation. While existing toolkits like SpeechBrain or ESPnet provide robust frameworks for academic replication, they frequently lack the specialized, production-ready components required for complex multimodal deployments. ClearerVoice-Studio addresses this gap as an open-source, AI-powered toolkit designed for interconnected speech tasks, including enhancement, separation, and multimodal target speaker extraction.
Methodology
ClearerVoice-Studio enables the creation of composite pipelines, allowing users to chain tasks—such as denoising followed by super-resolution—to address multi-faceted audio degradation. The toolkit leverages two primary architectures:
- FRCRN (Frequency Recurrent Convolutional-Recurrent Network): This model utilizes complex-valued operations and a frequency recurrence mechanism to capture high-resolution spectral details.
- MossFormer2: An evolution of the Transformer architecture that replaces standard multi-head attention with a gated single-head mechanism and convolution-augmented joint self-attention. This design more efficiently models both long-range dependencies and local features.
The platform also introduces SpeechScore, a comprehensive evaluation suite for metrics such as PESQ, STOI, and DNSMOS, ensuring standardized benchmarking across tasks.
Key Results and Benchmarks
The models within ClearerVoice-Studio demonstrate state-of-the-art performance across several standard benchmarks.
Speech Enhancement
On the DNS-2020 (16kHz) benchmark, the MossFormerGAN_SE_16K model achieved a PESQ of 3.57 and a P808 MOS of 4.05, significantly outperforming established baselines like DCCRN+ and TridentSE.
Audio-Visual Target Speaker Extraction
A standout feature of the toolkit is its support for multimodal inputs. By utilizing visual cues such as lip movements, the AV_MossFormer2 model effectively solves the "cocktail party problem." On the VoxCeleb2 mixture dataset, it achieved superior SI-SNRi scores for both 2-speaker and 3-speaker scenarios, outperforming the previous state-of-the-art, AV-TFGridNet.
Discussion and Future Work
Experimental results highlight that interconnected audio problems require composite solutions; for instance, applying super-resolution is only effective after initial denoising. Furthermore, the success of the gated single-head attention in MossFormer2 suggests a more efficient path for temporal dependency modeling in speech than traditional multi-head Transformer blocks.
Future development for ClearerVoice-Studio will focus on integrating diffusion models, expanding multi-channel support, and optimizing architectures for real-time edge deployment to further bridge the gap between research and industry application.
Summary updated