Speech Pattern based Black-box Model Watermarking for Automatic Speech Recognition

Haozhe Chen, Weiming Zhang, Kunlin Liu, Kejiang Chen, Han Fang, Nenghai Yu

Abstract

Protecting the intellectual property of Automatic Speech Recognition (ASR) models is challenging due to the unique temporal nature of sequence-to-sequence architectures. This paper presents a novel black-box watermarking framework that utilizes continuous speech patterns as triggers and linguistic steganography for natural-sounding target labels. The approach effectively bypasses the Connectionist Temporal Classification (CTC) alignment problem and resists label-detection evasion attacks. Experimental results on the DeepSpeech model demonstrate high fidelity, perfect watermark extraction, and significant robustness against model pruning and fine-tuning.

Introduction

The high cost of training state-of-the-art Deep Neural Networks (DNNs) makes intellectual property (IP) protection a critical concern. Black-box watermarking allows owners to verify ownership by querying a model with specific "trigger" inputs and observing the outputs. However, traditional watermarking techniques designed for classification tasks fail when applied to ASR systems.

Two primary obstacles exist: the CTC alignment problem, where localized triggers disrupt the model's conditional independence assumptions, and label-detection evasion attacks, where obvious copyright strings in the output (e.g., "Copyright 2022") can be easily filtered by attackers. This paper introduces a framework designed to overcome these hurdles while maintaining model performance.

Proposed Framework

The authors propose a three-stage framework to address these challenges: trigger generation, watermark embedding, and extraction.

Figure 1: Overview of the proposed watermarking framework, including trigger generation using speech patterns and linguistic steganography, embedding via fine-tuning, and extraction through BER computation.

1. Continuous Trigger Injection

To solve the CTC alignment issue, the framework overlays a specific "speech pattern" (e.g., a repeated audio clip of the owner's voice) across the entire duration of the input audio. This forces the model to map the entire sequence to the target label consistently, bypassing the conflicts that arise from localized noise.

2. Linguistic Steganography

To prevent evasion, target labels are generated using a DNN-based linguistic steganography tool (VAE-stega). This hides the secret copyright bit-string within natural-sounding sentences. Because these "stego texts" appear benign, an attacker cannot filter them out without risking the interception of legitimate user queries.

3. Embedding and Extraction

The ASR model is fine-tuned on a combination of clean audio and trigger audio/stego text pairs. Ownership is verified by querying the model with trigger audio and decoding the hidden message from the output text. If the Word Error Rate (WER) and Character Error Rate (CER) are below a predefined threshold, the watermark is extracted.

Experimental Results

The framework was evaluated using the DeepSpeech (v0.5.1) ASR system on the LibriSpeech dataset.

Fidelity and Integrity

The watermarking process maintains high model performance. The Word Error Rate (WER) on clean audio increased only slightly from 3.376% to 3.723% after embedding. Furthermore, the system demonstrates high integrity; unwatermarked models yield a 100% WER when queried with triggers, preventing false positive claims.

Robustness

A key strength of this framework is its resilience to common model modifications intended to remove watermarks, such as parameter pruning and fine-tuning.

Figure 3: Robustness of the watermark against parameter pruning (left) and model fine-tuning (right). The Bit Error Rate (BER) remains at 0% across various pruning rates and training epochs.

As shown in Figure 3, the watermark remains perfectly extractable (0% BER) even when the model is subjected to extreme pruning (up to 90% of weights zeroed out) or extensive fine-tuning (up to 50 epochs). This indicates that the watermark is deeply embedded within the model's learned representations and is highly resistant to removal attacks.

Conclusion

This work introduces the first robust black-box watermarking solution tailored for ASR systems. By leveraging continuous speech patterns and steganographic labels, the framework provides a secure and high-fidelity method for IP protection. Future research may focus on further reducing the performance overhead and extending the method to non-CTC architectures like Transformers or Conformers.

Summary updated

Abstract

As an effective method for intellectual property (IP) protection, model watermarking technology has been applied on a wide variety of deep neural networks (DNN), including speech classification models. However, how to design a black-box watermarking scheme for automatic speech recognition (ASR) models is still an unsolved problem, which is a significant demand for protecting remote ASR Application Programming Interface (API) deployed in cloud servers. Due to conditional independence assumption and label-detection-based evasion attack risk of ASR models, the black-box model watermarking scheme for speech classification models cannot apply to ASR models. In this paper, we propose the first black-box model watermarking framework for protecting the IP of ASR models. Specifically, we synthesize trigger audios by spreading the speech clips of model owners over the entire input audios and labeling the trigger audios with the stego texts, which hides the authorship information with linguistic steganography. Experiments on the state-of-the-art open-source ASR system DeepSpeech demonstrate the feasibility of the proposed watermarking scheme, which is robust against five kinds of attacks and has little impact on accuracy.
BibTeX
@inproceedings{chen2021speech,
  author = {Haozhe Chen and Weiming Zhang and Kunlin Liu and Kejiang Chen and Han Fang and Nenghai Yu},
  title = {{Speech Pattern based Black-box Model Watermarking for Automatic Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp43922.2022.9747044},
}
Figure