Robust Control Encoders for DDSP Speech Purification

Audio classifiers such as keyword spotters and speaker identifiers can be broken by tiny, nearly inaudible changes to the waveform. The two defenses that work both have a cost on clean audio. Adversarial training retrains the classifier on attacked inputs and gives up clean accuracy. Diffusion purification, which adds noise to the input and then regenerates the audio with a large generative model, is slow and can invent different words. This note proposes a third option built on differentiable digital signal processing (DDSP). DDSP is a family of models in which a small neural network does not output audio samples directly. It predicts a few physical controls, such as pitch, loudness, harmonic amplitudes and a noise-filter shape, and fixed, differentiable synthesizer equations turn those controls into sound. We place a DDSP resynthesizer in front of a frozen classifier. We then make only its small control encoder robust, so that small waveform perturbations cannot move the predicted controls much.

The claim is that this moves the robustness–accuracy trade-off out of the classifier and into a low-dimensional, physically meaningful control space. The frozen classifier keeps its clean-data knowledge. The purifier runs in one deterministic pass with a few hundred thousand parameters. An attacker must now move pitch and harmonic structure rather than add arbitrary sample-level noise. If the idea is right, clean accuracy should fall much less than under classifier-side adversarial training at matched robust accuracy, and much faster than diffusion purification at similar robustness.

Core Claim and What Is New

The literature at this intersection uses differentiable signal processing in three ways, and none of them is as a robust purifier. Jati et al. (Adversarial attack and defense strategies for deep speaker recognition systems) and the Hybrid Adversarial Training paper (Adversarial Defense for Deep Speaker Recognition Using Hybrid Adversarial Training) use a fixed, differentiable log-Mel frontend. Its only role is to let the attacker send gradients back to the waveform. All defense then happens by adversarially training the classifier.

The cortical-feature defense (Cortical Features for Defense Against Adversarial Audio Attacks) and the biomimetic frontend (Biomimetic Frontend for Differentiable Audio Processing) insert auditory-model filters as analysis layers. The cortical paper finds that these layers slow the decline in accuracy under white-box attack but are bypassed near 25 dB SNR. The biomimetic paper never runs an adversarial attack. AudioPure (Defending against Adversarial Audio via Diffusion Model) is the closest match in spirit, since it also purifies at test time in front of an untouched classifier. It relies on DiffWave, which needs about one second of GPU time per second of audio, and its diffusion step count directly trades removing the perturbation against hallucinating phonemes.

Our proposal differs from all of these in one specific way. The purifier is an analysis-by-synthesis bottleneck: the encoder estimates controls, and the synthesizer rebuilds audio from them. Only that encoder is trained with a TRADES-style stability term, and the stability is defined on the controls rather than on classifier outputs. TRADES (Theoretically Principled Trade-off between Robustness and Accuracy) splits robust error into natural error (mistakes on clean inputs) and boundary error (clean inputs that a small perturbation can push across the decision boundary). Its penalty makes predictions stable in a ball around each input. We reuse that penalty but apply it to the encoder $E$ that maps a waveform $x$ to controls. The synthesizer $S$ and classifier stay fixed:

$$\min_{E}\; \mathbb{E}_x\Big[\, \mathcal{L}_{\text{spec}}\big(S(E(x)), x\big) + \lambda \max_{x' \in \mathbb{B}(x,\epsilon)} d\big(E(x'), E(x)\big) \Big]$$

Here $\mathcal{L}_{\text{spec}}$ is the multi-scale spectral reconstruction loss from the original DDSP work (DDSP: Differentiable Digital Signal Processing). $\mathbb{B}(x,\epsilon)$ is the $L_\infty$ or $L_2$ ball of radius $\epsilon$ around the waveform. $d$ measures distance between controls in perceptual units: cents for pitch, dB for loudness and log-magnitude for the filter bands. $\lambda$ sets how much reconstruction fidelity we give up for control stability.

In this setup the natural error of the whole system comes only from resynthesis losing information the classifier needs. We can measure that loss directly by running the frozen classifier on clean resynthesized audio. That gives a clean decomposition that classifier-side adversarial training cannot provide. The encoder is small, and the inner maximization runs only through the encoder, not through the classifier. The ten-fold slowdown that TRADES reports for adversarial training should therefore shrink considerably.

Research Program

The purifier uses a harmonic-plus-noise synthesizer with both sine and cosine harmonic terms, following Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP. That paper matched a 9M-parameter vocoder with 0.4M parameters and ran 4.9x faster on CPU. We add a per-frame spectral envelope and a time-varying filtered-noise branch. Speaker identity lives largely in formants and voice quality, which these components must carry. Pitch comes from a differentiable estimator that is part of the encoder, so no step in the pipeline blocks gradients.

The work proceeds in three stages. The first stage trains the encoder for reconstruction alone, sweeps control resolution (the number of harmonics and noise bands), and measures clean accuracy of the frozen classifier on resynthesized audio. This isolates the cost of the bottleneck itself. The second stage adds the stability term, sweeps $\lambda$ and $\epsilon$, and traces the full curve of clean accuracy against robust accuracy. The third stage attacks the complete pipeline with gradients through the encoder, synthesizer and classifier. It also includes an attacker aimed at the controls: that attacker searches for the waveform perturbation that most shifts the controls toward another class's control statistics. The third stage also reports the SNR and PESQ (a standard speech-quality score) of successful attacks, to test whether the bottleneck forces attacks to become louder and more structured, as the cortical-feature paper observed.

Baselines

The naive reference is the undefended classifier. On LibriSpeech speaker identification, Jati et al. report 94% clean accuracy and 0% under white-box PGD-100 (projected gradient descent, a standard iterative attack). A second naive reference is a static low-pass filter, which AudioPure reports falls to near 0% under adaptive white-box attack. We rerun both on our splits so that every comparison uses one evaluation protocol.

The strongest classifier-side competitor on speaker identification is Hybrid Adversarial Training (Adversarial Defense for Deep Speaker Recognition Using Hybrid Adversarial Training). It reports 97.68% clean accuracy and 78.84% accuracy under PGD-40 on LibriSpeech train-clean-100. We also run plain PGD-10 adversarial training, for which Jati et al. report 43% under PGD-100, and TRADES on the same classifier. Each gets the same training data and the same GPU-hour budget as our full pipeline, including encoder training. Each gets an equal hyperparameter sweep of about twenty configurations.

The strongest test-time competitor is AudioPure on SC09 spoken digits, which reports about 84% robust accuracy under strong white-box adaptive attacks. We use its released setup where possible. We compare at matched clean accuracy and report its latency next to ours on the same GPU and on CPU.

Two ablations target the key ingredient. The first removes the stability term, $\lambda = 0$, which tests whether a DDSP bottleneck alone does anything under adaptive attack. The second replaces the synthesizer with a generic convolutional autoencoder of equal parameter count and equal bottleneck dimension, trained with the same objective. This follows the spirit of Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals, which removes perturbations by projecting onto a small learned subspace.

The proposal beats the field under three conditions. It must beat Hybrid Adversarial Training's 78.84% robust accuracy with no more than a one-point loss from 97.68% clean, or match its robustness at higher clean accuracy. It must reach AudioPure-level robustness on SC09 with at least ten times lower latency. It must beat the generic-autoencoder ablation at every $\lambda$.

Why not just…?

Why not just use fewer diffusion steps or a smaller diffusion purifier? Fewer steps address latency, but AudioPure's own analysis shows that the step count is exactly where the trade-off lives: too few steps leave the perturbation in place. A smaller diffusion model still samples stochastically and can still hallucinate content. We will run AudioPure at a step count chosen to match our latency. If it still matches our robust and clean accuracy there, the speed argument for DDSP disappears, and we will say so.

Why not just use any learned autoencoder bottleneck? This is the most serious alternative. Subspace Defense shows that a generic low-rank projection can strip perturbations; on IMDB it reports 78.6% accuracy against 17.8% for adversarial training. If the physical structure adds nothing, the equal-size convolutional autoencoder ablation will match our curve. The idea then reduces to robust autoencoding, and we would report it as that. What we expect to separate them is clean accuracy. A generic bottleneck has to learn phase and harmonic coherence from data, while the DDSP bottleneck gets them from its equations, so it should lose less speaker and phoneme detail at the same dimension.

Why not just run TRADES on the classifier? TRADES spreads the cost across the whole classifier. On CIFAR10 it traded clean accuracy down to 88.64% to reach 56.61% robust accuracy, and it costs roughly ten times more training time. Jati et al. also found that $L_\infty$ adversarial training did not transfer to C&W $L_2$ attacks, a strong attack from Carlini and Wagner. Our comparison at matched compute tests both points directly: clean accuracy and cross-norm transfer.

Why not just low-pass filter or add noise during training? Low-pass filtering collapses to near 0% against adaptive attackers, as AudioPure reports. Jati et al. found that Gaussian-noise augmentation gives no defense and that C&W perturbations sit inside the speech band with PESQ above 4.0, so we keep these only as reference points.

Proposed Experiments and Decisive Checks

We evaluate on SC09 and on LibriSpeech train-clean-100 speaker identification. The attacks are PGD-100 under $L_\infty$ and $L_2$, C&W $L_2$, the control-space attack described above, and a query-based black-box attack. We report the clean-versus-robust accuracy curve over $\lambda$, clean accuracy on resynthesized audio alone, CPU and GPU latency, and the SNR and PESQ of successful attacks.

Because DDSP pitch estimation can hide gradients rather than resist attacks, we run the standard sanity checks for this failure. Robust accuracy must fall monotonically to zero as $\epsilon$ grows. Unbounded attacks must succeed. Black-box attacks must not beat white-box ones. Attacks crafted on a smooth surrogate encoder must be no stronger than attacks on the real one.

The decisive result has two parts. The first is a curve showing the frozen classifier behind a robust DDSP purifier dominating classifier-side adversarial training on clean accuracy at equal robustness. The second is a clear gap between the DDSP purifier and the equal-size generic autoencoder. If the gap is absent, the physical-structure hypothesis is false.