Introduction
As generative AI continues to lower the barrier for creating and distributing synthetic audio, the need for robust watermarking—embedding imperceptible signatures for copyright and authenticity—has never been more urgent. However, the evaluation of these systems has historically been inconsistent, often relying on low-quality speech data or a narrow range of distortions. Researchers from Sony AI have addressed this gap by introducing RAW-Bench, a high-fidelity benchmark designed to test audio watermarking against 20 real-world distortions, with a specific focus on the emerging threat of neural audio codecs.
The RAW-Bench Framework
The Robust Audio Watermarking Benchmark (RAW-Bench) evaluates models across three primary dimensions: imperceptibility, robustness, and capacity. Unlike previous benchmarks, RAW-Bench utilizes high-fidelity 44.1 kHz audio across music, speech, and environmental sounds to ensure results are applicable to professional-grade audio production.
A core component of RAW-Bench is its comprehensive attack pipeline, which categorizes 20 different distortions into six groups. These range from traditional mixing and dynamics processing to modern neural compression techniques.
The benchmark introduces "Loose" and "Strict" settings for these attacks, defined by expert listening tests to distinguish between distortions that are entirely imperceptible and those that are audible but acceptable in common use cases.
Methodology and Dataset
To move beyond the limitations of 16kHz speech datasets, the authors curated a high-fidelity test set from nine open-source collections, including studio-quality speech (DAPS), diverse musical genres (MAESTRO, MoisesDB), and environmental soundscapes (Clotho).
The study evaluated four state-of-the-art models: AudioSeal (AS), SilentCipher (SC), Timbre (TI), and WavMark (WM). Additionally, the researchers performed adversarial retraining on AudioSeal and SilentCipher (denoted as AS and SC) using the RAW-Bench pipeline to determine if exposure to these specific distortions could improve long-term robustness.
Results: The Neural Codec Challenge
The findings reveal a fundamental "zero-sum game" between watermarking and neural codecs. Watermarks typically hide information in the perceptual redundancies of audio—the same redundancies that neural codecs like Encodec and Descript Audio Codec (DAC) are designed to identify and remove to achieve high compression ratios.
Key insights from the results include:
- The Neural Codec Wall: Neural compression proved to be the most difficult hurdle. Even after retraining, full-message accuracy for most models dropped to near zero when faced with the Descript Audio Codec (DAC). This suggests that watermarking models may be overfitting to specific compression artifacts rather than achieving generalized robustness.
- Imperceptibility vs. Robustness: SilentCipher emerged as the leader in audio quality (highest SI-SNR), while Timbre demonstrated the greatest resilience against traditional editing techniques like filtering and dynamics processing.
- Fragility to Low-Level Attacks: Some sophisticated models showed surprising vulnerability to simple operations. For example, polarity inversion or minor time-stretching (TJ) completely broke several state-of-the-art algorithms.
Conclusion and Future Directions
The RAW-Bench results indicate a stalemate: if a neural codec is the final step in a distribution pipeline, it is highly likely to strip away existing watermarks. The authors suggest that future research must move beyond simply making watermarks "tougher." Instead, we may need to develop "codec-aware" watermarking or integrate the watermarking process directly into the neural compression architecture itself to ensure that the hidden signature is preserved as essential information rather than discarded as noise.
Summary updated