TL;DR
This paper evaluates the voice quality of three emerging open-source codecs (Silk, CELT, and BroadVoice) against established standards using a subjective listening test. The findings reveal that while Silk offers competitive quality, its variable bitrate and bandwidth present challenges, and CELT emerges as a strong contender for high-quality audio at higher bitrates.
What Problem Does This Paper Address?
The paper addresses the growing need for efficient and high-quality voice codecs for internet telephony (VoIP). As the landscape of voice communication shifts towards open-source solutions, understanding their performance relative to established standards is crucial. The authors aim to fill this gap by conducting a rigorous subjective evaluation of three prominent open-source codecs: Silk, CELT, and BroadVoice. They compare these against well-known codecs like AMR, AMR-WB, G.718, G.718B, and G.722.1C, using both clean and noisy speech to provide a comprehensive picture of their real-world applicability.
Methods
The core of this research is a subjective listening test conducted in a dedicated listening test laboratory. Forty naïve listeners participated, evaluating over 30 different codec conditions, each presented with six voice samples. To ensure a robust evaluation, each listener assessed approximately 200 samples in a randomized order. The test employed a modified version of the traditional Absolute Category Rating (ACR) MOS (Mean Opinion Score) method, extending the scale to nine categories. This "Extended Range MOS" method aims to provide finer discrimination, especially for high-quality samples, while remaining more practical for naïve listeners than methods like MUSHRA. The test material included both clean and noisy mono speech signals, encompassing a variety of background noises like music, cafeteria, street, and office environments.
Datasets
The paper does not introduce new datasets. Instead, it utilizes a set of pre-recorded speech samples for the listening test. These samples consist of male and female voices presented in both clean conditions and with background noise at various levels (-15 dB and -20 dB). The specific sentences used were pairs lasting 8 seconds each.
Key Insights and Ideas
- Silk's Adaptive Nature is a Double-Edged Sword: Silk's ability to dynamically adjust its bitrate and signal bandwidth offers potential for efficiency but also introduces variability that can be problematic for network transmission.
- Bandwidth Claims vs. Reality: Silk's advertised bandwidth capabilities, particularly for Super Wideband (SWB), appear to require significantly higher bitrates in practice than initially suggested.
- Narrowband vs. Wideband Evolution: The paper highlights the ongoing trend towards wider bandwidths and higher sampling rates for improved user experience, moving beyond traditional narrowband limitations.
- CELT as a High-Quality Alternative: CELT demonstrates strong performance, particularly at higher bitrates, positioning it as a competitive option for applications demanding excellent voice quality.
- BroadVoice's Niche: While BroadVoice offers very low delay, its voice quality at comparable bitrates is not competitive with other tested codecs, suggesting it's best suited for specialized applications where ultra-low latency is paramount.
Key Results
- Silk's Competitive Quality: Silk generally performs well, often surpassing AMR-WB at comparable bitrates, especially in the 16-24 kbit/s range for Wideband (WB) signals. At 32 and 40 kbit/s, it competes effectively with G.718B and G.722.1C for Super Wideband (SWB).
- Silk's Bandwidth Requirements: To achieve WB quality, Silk requires approximately 9.5 kbit/s, and SWB needs around 24.5 kbit/s, which is higher than some initial claims.
- CELT Performance: For acceptable quality, CELT needs at least 32 kbit/s. At 40 kbit/s, its quality is very good, comparable to G.718B at 32 kbit/s. Truly transparent quality for CELT requires 48 kbit/s or more.
- BroadVoice Competitiveness: BroadVoice 16 kbit/s (NB) offers similar quality to AMR at 10.2 kbit/s, and BroadVoice 32 kbit/s (WB) is comparable to AMR-WB at 12.65 kbit/s. However, their bitrates are not competitive.
Limitations and Future Work
- Silk's Variable Bitrate: The paper acknowledges that Silk's highly variable bitrate can cause issues depending on the transmission network and channel. This is a significant consideration for its deployment.
- Silk's Bandwidth Promises: The authors found that Silk's claims regarding SWB bandwidth at 12 kbit/s are not met in practice, requiring substantially higher bitrates.
- BroadVoice's Niche: While BroadVoice excels in low delay, its lack of bitrate competitiveness limits its broader applicability. Future work could explore optimizations or specific use cases where its low-delay advantage outweighs its quality-per-bit trade-off.
- Further Evaluation: The paper implicitly suggests that continued evaluation of these and other emerging codecs is necessary as the field evolves.
Summary updated