A Comprehensive Real-World Assessment of Audio Watermarking Algorithms: Will They Survive Neural Codecs?

Yigitcan Özer, Woosung Choi, Joan Serrà, Mayank Kumar Singh, Wei-Hsiang Liao, Yuki Mitsufuji

Introduction

As generative AI continues to lower the barrier for creating and distributing synthetic audio, the need for robust watermarking—embedding imperceptible signatures for copyright and authenticity—has never been more urgent. However, the evaluation of these systems has historically been inconsistent, often relying on low-quality speech data or a narrow range of distortions. Researchers from Sony AI have addressed this gap by introducing RAW-Bench, a high-fidelity benchmark designed to test audio watermarking against 20 real-world distortions, with a specific focus on the emerging threat of neural audio codecs.

The RAW-Bench Framework

The Robust Audio Watermarking Benchmark (RAW-Bench) evaluates models across three primary dimensions: imperceptibility, robustness, and capacity. Unlike previous benchmarks, RAW-Bench utilizes high-fidelity 44.1 kHz audio across music, speech, and environmental sounds to ensure results are applicable to professional-grade audio production.

A core component of RAW-Bench is its comprehensive attack pipeline, which categorizes 20 different distortions into six groups. These range from traditional mixing and dynamics processing to modern neural compression techniques.

Table 1: The RAW-Bench attack pipeline, detailing 20 distortion types across six categories and their associated thresholds for evaluation.

The benchmark introduces "Loose" and "Strict" settings for these attacks, defined by expert listening tests to distinguish between distortions that are entirely imperceptible and those that are audible but acceptable in common use cases.

Methodology and Dataset

To move beyond the limitations of 16kHz speech datasets, the authors curated a high-fidelity test set from nine open-source collections, including studio-quality speech (DAPS), diverse musical genres (MAESTRO, MoisesDB), and environmental soundscapes (Clotho).

The study evaluated four state-of-the-art models: AudioSeal (AS), SilentCipher (SC), Timbre (TI), and WavMark (WM). Additionally, the researchers performed adversarial retraining on AudioSeal and SilentCipher (denoted as AS and SC) using the RAW-Bench pipeline to determine if exposure to these specific distortions could improve long-term robustness.

Results: The Neural Codec Challenge

The findings reveal a fundamental "zero-sum game" between watermarking and neural codecs. Watermarks typically hide information in the perceptual redundancies of audio—the same redundancies that neural codecs like Encodec and Descript Audio Codec (DAC) are designed to identify and remove to achieve high compression ratios.

Table 2: Comprehensive robustness results across all 20 attacks. Checkmarks indicate successful recovery, while numerical values represent bitwise accuracy. Note the significant performance drops under neural compression (EN, DA).

Key insights from the results include:

  • The Neural Codec Wall: Neural compression proved to be the most difficult hurdle. Even after retraining, full-message accuracy for most models dropped to near zero when faced with the Descript Audio Codec (DAC). This suggests that watermarking models may be overfitting to specific compression artifacts rather than achieving generalized robustness.
  • Imperceptibility vs. Robustness: SilentCipher emerged as the leader in audio quality (highest SI-SNR), while Timbre demonstrated the greatest resilience against traditional editing techniques like filtering and dynamics processing.
  • Fragility to Low-Level Attacks: Some sophisticated models showed surprising vulnerability to simple operations. For example, polarity inversion or minor time-stretching (TJ) completely broke several state-of-the-art algorithms.

Conclusion and Future Directions

The RAW-Bench results indicate a stalemate: if a neural codec is the final step in a distribution pipeline, it is highly likely to strip away existing watermarks. The authors suggest that future research must move beyond simply making watermarks "tougher." Instead, we may need to develop "codec-aware" watermarking or integrate the watermarking process directly into the neural compression architecture itself to ensure that the hidden signature is preserved as essential information rather than discarded as noise.

Summary updated

Abstract

We present the Robust Audio Watermarking Benchmark (RAW-Bench) to foster the evaluation of deep learning-based audio watermarking algorithms, establishing a standardized benchmark and allowing systematic comparisons. To simulate real-world usage, we introduce a comprehensive audio attack pipeline featuring various distortions such as compression, background noise, and reverberation and propose a diverse test dataset, including speech, environmental sounds, and music recordings. By assessing the performance of four existing watermarking algorithms on our framework, two main insights stand out: (i) neural compression techniques pose the most significant challenge, even when algorithms are trained with such compressions; and (ii) training with audio attacks generally improves robustness, although it is insufficient in some cases. Furthermore, we find that specific distortions, such as polarity inversion, time stretching, or reverb, seriously affect certain algorithms. Our contributions strengthen the robustness and perceptual assessment of audio watermarking algorithms across a wide range of applications while ensuring a fair and consistent evaluation approach.
BibTeX
@inproceedings{ozer2025comprehensive,
  author = {Yigitcan Özer and Woosung Choi and Joan Serrà and Mayank Kumar Singh and Wei-Hsiang Liao and Yuki Mitsufuji},
  title = {{A Comprehensive Real-World Assessment of Audio Watermarking Algorithms: Will They Survive Neural Codecs?}},
  booktitle = {Interspeech},
  year = {2025},
  doi = {10.21437/interspeech.2025-1530},
}

Loading related content...

Figure