Reading Depth as a Robustness Knob in Frozen Speech Encoders

Large self-supervised speech models such as wav2vec 2.0, HuBERT and Whisper are usually used frozen: a small classifier reads their hidden states and is trained for keyword spotting, intent recognition or speaker identity. Which layer that classifier reads from is already known to matter a great deal for clean accuracy. It is also known that these encoders collapse under tiny, inaudible adversarial perturbations, which are inputs deliberately nudged to cause errors. Nobody has asked whether the choice of reading layer also sets the model's robustness. This proposal treats depth as a dial on the robustness-accuracy trade-off. The trade-off is the familiar pattern where defending a model against attacks lowers its accuracy on ordinary, unattacked inputs.

The proposal has two parts. The first is a measurement: under attacks that know exactly which layer is being read, how do robust accuracy and clean accuracy vary with reading depth? The second is a method built on that measurement. It chooses the reading depth from unlabeled audio using a cheap per-layer "adversarial drift" score. It then applies adversarial fine-tuning only to the encoder layers below that depth. If the measurement shows a real depth-wise frontier, this gives a cheaper and less accuracy-costly defense than adversarially fine-tuning the whole encoder. If the frontier is flat under adaptive attack, the study still settles an open disagreement in the literature.

Core Claim

We claim three things about a frozen speech encoder with a small adversarially fine-tuned adapter. First, the best robustness-accuracy trade-off is reached by reading from an intermediate layer $l^\*$ and adapting only layers $1..l^\*$, not by adapting and reading the full stack. Second, $l^\*$ can be predicted without labeled adversarial data. Third, the gain survives attacks that are aware of which layer the classifier reads.

The claim is falsifiable at each step. An adversarially trained learnable weighted sum over all layers might match truncation. Adaptive attacks might erase any depth advantage. The label-free score might fail to rank layers. The experiments below test each of these directly.

Where This Sits

Evidence that intermediate layers help. Two papers motivate the idea, both in vision.

  • FACM attaches auxiliary classifiers and autoencoders to intermediate CNN layers. It argues that perturbations damage features cumulatively, so middle layers keep usable class information. It raises CIFAR-10 PGD accuracy from near 0% to about 48.7% at roughly a 2.5% clean cost.
  • Intermediate Layer Classifiers for OOD generalization shows that linear probes on earlier layers beat final-layer probes under distribution shift (Waterbirds worst-group 87.1% vs 79.4%). It does not evaluate gradient-based attacks. It leaves layer choice dependent on out-of-distribution validation data.

We differ from both in three ways. We work on sequential speech encoders. We evaluate against attacks that target the chosen layer, rather than FACM's largely grey-box setting. We replace oracle layer selection with a label-free score.

Evidence that depth is task-specific in speech. These results supply the prior that depth matters.

  • Whisper-PMFA finds that Whisper blocks 17–24 carry speaker information and that including blocks 25–32 hurts.
  • Speech foundation models in healthcare reports a 13.6% balanced-accuracy gain from the best intermediate layer over the final layer. It also finds that a learned weighted sum comes within about 1.2–1.5% of the best layer.
  • DAISY shows noisy audio naturally defers to deeper exits. It tests only random MUSAN noise, not adversarial inputs.

Constraints. Two results rule out easy wins.

Borrowed method. How to Estimate Model Transferability of Pre-Trained Speech Models? ranks layers for clean utility with LogME at a Spearman correlation of 0.87–0.94. We reuse it as the clean-utility half of our selection score.

Research Program

Phase one: map the frontier. For each encoder layer $l$ we train a small classifier on that layer's pooled hidden states and record two numbers:

  • clean accuracy;
  • robust accuracy under an attack that maximizes that classifier's loss by backpropagating through layers $1..l$.

This yields a depth-wise curve for each task and model. It directly answers the tension in the file: do earlier layers expose more attackable low-level acoustic detail, or do they avoid final-layer shortcuts?

Phase two: truncated adversarial adaptation. Low-rank adapters (LoRA) are inserted only in layers $1..l$ and trained with PGD adversarial examples, while the reading classifier sits at $l$. The upper layers $l+1..L$ are dropped. This cuts forward, backward and attack cost roughly in proportion to $l/L$.

Phase three: label-free depth selection. For each layer we compute a drift score on unlabeled clean audio. Here $h_l(x)$ is the pooled layer-$l$ representation of utterance $x$, $\delta$ is a perturbation bounded in max-amplitude by $\epsilon$, and $x'$ is a different random utterance:

$$D_l = \frac{\mathbb{E}_x \max_{\|\delta\|_\infty \le \epsilon} \|h_l(x+\delta) - h_l(x)\|^2}{\mathbb{E}_{x,x'} \|h_l(x) - h_l(x')\|^2}$$

The numerator is how far an attacker can move a layer's representation. The denominator is how far the representation naturally moves between different utterances. Their ratio is the fraction of the natural spread an attacker can cover. It needs no labels, and it is computed with an attack aimed at layer $l$ itself, not at the output.

The selection rule picks the layer that maximizes the LogME score minus $\lambda D_l$. We tune $\lambda$ once on one task and freeze it for all others.

Baselines

Undefended final-layer classifier. The naive reference is a classifier on the final layer with no defense. This matches the setup of Characterizing the adversarial vulnerability of speech self-supervised learning, whose reported intent-classification drop from 100% to 55.67% calibrates our attack strength. We run it on the same splits.

Undefended oracle-layer classifier. The same undefended probe placed at the best intermediate layer, chosen on test data. This isolates whether depth alone, without adversarial training, buys robustness.

Full-depth adversarial LoRA. The strongest existing competitor is PGD adversarial training of LoRA adapters in all layers, with the SUPERB-style learned weighted sum over all layers as the head. It is the most natural "just defend the whole thing" option. Its weighted sum can in principle learn to down-weight fragile layers by itself.

No reported result was found on Lacuna for adversarially trained wav2vec 2.0, HuBERT or Whisper on SUPERB tasks. The only reference point is the qualitative finding in Adversarial Robustness Analysis in Automatic Pathological Speech Detection Approaches that standard adversarial training failed on PC-GITA. We will therefore produce this baseline ourselves and give it an equal budget:

  • identical LoRA rank;
  • identical PGD steps and $\epsilon$;
  • the same hyperparameter search size;
  • total adversarial-training GPU-hours at least equal to the truncated method. Because the truncated model is cheaper per step, the baseline receives more steps rather than fewer.

Ported FACM. FACM-style auxiliary classifiers are attached to several fixed intermediate layers of the frozen encoder and combined by a small decision network, evaluated under the same adaptive attack. Its reported vision figures (about 48.7% PGD robust accuracy on CIFAR-10 at about a 2.5% clean drop) are context only and are not comparable to speech.

Key ablation. Train the same truncated adversarial LoRA but read from the final layer, keeping the adversarial adaptation budget fixed. Any gain of the full method over this ablation is attributable to reading depth rather than to adversarial training.

What counts as beating the baseline. The method wins if it:

  • dominates full-depth adversarial LoRA on the clean-versus-robust accuracy plane on at least three of four tasks, at matched or lower compute;
  • lands within 2 points of the oracle-layer truncated result, using the label-free choice of depth.

Why not just…?

Why not just adversarially train a learned weighted sum over all layers? This is the most serious threat. The weighted sum could discover robust depths by itself, which would make explicit selection redundant. The evidence is mixed. Speech foundation models in healthcare finds weighted sums nearly match the best layer on clean data. However, a weighted sum still backpropagates attack gradients through every layer, and it pays the full-depth training cost. The full-depth adversarial LoRA baseline tests this directly. If it matches truncation on the robust-clean frontier, we report that result. The contribution then shrinks to the depth-wise measurement and the label-free score used as a cheap cost-reduction tool.

Why not just filter the input audio? Input purification harms benign audio. SpeechGuard reports clean samples misflagged as poisoned dropping by up to about 10%, and to near 60% accuracy in some cases. Muting Whisper shows a 0.64-second, $\epsilon=0.02$ hiss that resembles ordinary microphone noise yet silences Whisper on more than 97% of samples. We nonetheless include a bandpass and denoising front-end as an additional baseline, attacked end-to-end, so the comparison is empirical rather than assumed.

Why not just take FACM's static intermediate classifiers? FACM's layers are hand-placed. Its robustness was reported mainly in grey-box settings, where attack-aware evaluation can erase the gains. The ported-FACM baseline under our adaptive attack settles whether its property transfers to speech encoders. Our label-free selection addresses the placement problem it leaves open.

Proposed Experiments

Models and tasks. Encoders are wav2vec 2.0 Base, HuBERT Base and the Whisper encoder. Tasks are:

  • Speech Commands keyword spotting;
  • Fluent Speech Commands intent classification;
  • VoxCeleb1 speaker identification;
  • PC-GITA Parkinson's detection, as the small-data stress test where adversarial training reportedly fails.

Attacks and metrics. Attacks are $\ell_\infty$ PGD and BIM at several budgets, reported as SNR from 40 to 70 dB. Every attack is adaptive, differentiating through whatever pipeline is deployed. We report clean accuracy (EER for speaker tasks), robust accuracy, and training and inference compute. As a transfer check, universal prepend perturbations in the style of Muting Whisper are applied to Whisper classification heads.

Decisive check 1: gradient masking. Robust accuracy at the chosen layer must fall monotonically as attack steps and budget increase, and must reach near zero at large $\epsilon$. An apparent intermediate-layer advantage that disappears under a 100-step attack with random restarts would falsify the core claim.

Decisive check 2: label-free selection. Across all layer, task and model combinations, the drift-adjusted score should rank layers by robust accuracy with a Spearman correlation clearly above plain LogME. LogME's 0.87–0.94 figure for clean utility sets the level a useful ranking must approach.

Decisive check 3: small data. On PC-GITA, truncated adversarial LoRA must keep clean accuracy near the reported 82.6% while lifting robust accuracy above chance at 60 dB. That is the exact setting where full adversarial training is reported to fail.

If all three checks hold, reading depth is a practical and cheap robustness control. If check 1 fails, the study still resolves the depth-robustness question with attack-aware evidence.