What Does a Winning Ticket Stop Listening To? A Mechanistic Audit of the Noise Robustness of Pruned Audio Models

Sparse audio models found by iterative magnitude pruning hold up better in noise than the dense models they come from, and nobody knows why. Iterative magnitude pruning (IMP) trains a network, deletes the smallest-magnitude weights, rewinds the survivors to an early checkpoint, and repeats. In speech recognition, the resulting "winning tickets" kept their word error rate (WER) near 15.8% under heavy background noise, while the dense model's WER climbed from about 16% to over 38% (Audio Lottery). Pruned acoustic scene classifiers likewise generalized better to recording devices they had never heard (Acoustic Lottery). Both papers attribute this to "implicit regularization," which describes the effect without explaining it.

This proposal uses interpretability tools from speech models to explain the effect. Linear probes are small classifiers trained to read a property out of a layer's activations. Causal ablations remove a component and measure what breaks. We track where noise information lives inside the network across pruning rounds and test one question: does the ticket stop encoding noise, or does it still encode noise and simply stop using it when making predictions? Each answer leads to a different, testable way of transferring the robustness back to a dense model without the cost of IMP.

Core Claim

We hypothesize that IMP's noise bonus comes from decoupling, not deletion. Noise remains linearly decodable inside winning tickets about as well as inside dense models. What changes is that the subspace of activations carrying noise becomes nearly orthogonal to the directions the output layer reads from. In the dense model, by contrast, task outputs are sensitive to perturbations along that noise subspace.

This prediction follows from two prior results. First, dense speech encoders carry rich nuisance information: environmental noise is probe-decodable at 90.0% from intermediate Whisper layers (Beyond Transcription). Second, IMP specifically reduces off-diagonal coupling in the Fisher information, a measure of how strongly changes in one parameter interact with others in their effect on the loss. Matched-size unpruned networks do not show this reduction (IMP Reduces Weight-Space Coupling).

The claim is falsifiable three ways. The ticket may lose noise information altogether, which we call the deletion account. Any network at equal sparsity may be equally robust, which would make the effect a matter of capacity rather than anything specific to IMP. Or the noise bonus may simply fail to replicate. Each outcome is informative.

Relation to Prior Work

Three papers motivate the question. Audio Lottery applied IMP with rewinding to CNN-LSTM, RNN-Transducer and Conformer speech recognizers and reported the noise bonus, but did no internal analysis. Acoustic Lottery found device robustness in pruned scene classifiers. Because pruning there was combined with distillation and 8-bit quantization, its result cannot be attributed to pruning alone. We remove those confounds.

Two papers supply the methods. Beyond Transcription provides the probing and component-patching toolkit and the evidence that a single localized component (Whisper layer 18, head 13) can control a failure mode. It never asks how pruning reshapes such components, which is the step we take. IMP Reduces Weight-Space Coupling supplies the LOGDET coupling measure, defined as the log-determinant ratio between the full Fisher information and its diagonal. It was tested only on vision networks, and we test whether it tracks noise-task decoupling in audio.

That paper also constrains the design. It reports that LOGDET reverses and rises again below about 10% density. Our hypothesis therefore predicts that the noise bonus should weaken in the over-pruned regime. This prediction sits in tension with Audio Lottery's tickets at 4.4% density, and we test it directly rather than assume it away.

Composable Sparse Subnetworks is the closest work that uses IMP for interpretability. It isolates class-specific modules with a maximum-entropy loss on images and tabular data. It does not study nuisance robustness or audio, and it modifies the training objective, whereas we audit standard IMP as practitioners actually run it.

The post-hoc audio explanation literature is adjacent but answers a different question. This includes concept decomposition of CLAP embeddings, where CLAP is a contrastive audio-text model (Transformation of audio embeddings), NMF-based interpreters (L2I), and time-domain masks (LMAC-TD). Those methods explain individual predictions of a fixed model. We explain a property that differs between two models.

Research Program

Settings. We work in two regimes chosen so that full IMP is affordable.

The first is speech recognition with the CNN-LSTM model on TED-LIUM, mixing in DESED background noise at controlled signal-to-noise ratios. This replicates Audio Lottery's protocol.

The second is acoustic scene classification on the DCASE 2020 Task 1a development set. Here the recording device acts as the nuisance variable. We run pure IMP with rewinding and no distillation or quantization.

A Conformer on LibriSpeech is a stretch goal, used only to check whether the main finding transfers.

Training runs. For each regime we run IMP at 20% pruning per round, saving the rewound-and-retrained network at every density down to roughly 3%.

Two measurements per layer. At every saved network we measure two separate quantities.

Decodability is the accuracy of linear probes predicting noise type and signal-to-noise bin (or device identity) from each layer's activations.

Usage is measured by amnesic projection. We find the noise subspace with the probes, project it out of the activations at inference time, and record the change in task loss on noisy inputs. As a complementary measure, we compute the principal angles between the noise subspace and the subspace spanned by the gradients of the task output.

Geometry and timing. On the scene classifier, where computing the Fisher information is tractable, we also track LOGDET. This tests whether weight-space decoupling and noise-task decoupling move together. Because we have measurements at every round, we can ask whether robustness emerges in the same rounds as the decoupling. The timeline does not prove causation, but it makes the decoupling account easy to refute.

Causal transfer tests. These are the decisive tests, run in both directions.

Ticket to dense. In the dense model, we project out the noise-to-output coupling found from the ticket's geometry, with no retraining.

Dense to ticket. In the ticket, we restore the dense values of the pruned weights that connect the noise subspace to the output layers.

If the decoupling account is right, the first intervention should recover much of the dense model's robustness gap, and the second should remove much of the ticket's advantage.

Baselines

Dense model. The naive reference is the dense model trained once on the same data with the same recipe. Audio Lottery reports that on TED-LIUM under high DESED noise, the dense WER rises from about 16% to over 38% while the ticket stays at about 15.8%. Our first deliverable is reproducing this gap. If it does not reproduce, we report that and continue the audit only for whatever gap does exist.

Dense model with noise augmentation. This is the strongest practical competitor: the dense model trained with DESED-noise augmentation, given the same total GPU-hours that IMP consumes and the same hyperparameter search budget. No reported result was found on Lacuna for noise-augmented dense training on TED-LIUM with DESED. We will establish this number ourselves. The ticket "beats" this baseline in the engineering sense only if it matches or improves noisy WER at its lower parameter count. Scientifically, this baseline matters because it lets us check whether augmentation achieves robustness through the same decoupling signature or a different one.

One-shot magnitude pruning. We prune to each matched density in a single step, then fine-tune. Audio Lottery reports 3.96% WER for standard pruning against 2.51% for the IMP ticket on clean LibriSpeech with a Conformer. Its noise robustness was not reported, and we measure it.

Random pruning with matched layerwise sparsity. This control asks whether the specific mask matters or only the amount of sparsity.

Reinitialized ticket. We take the IMP mask but reinitialize the surviving weights. Zhou et al. show that the signs restored by rewinding carry much of a ticket's inductive bias (Deconstructing Lottery Tickets), so this control separates the effect of the mask from the effect of the initialization.

Small dense network at matched parameter count. Perez et al. found that such networks do not show low coupling, and we test whether they also lack the noise bonus.

Ablation of the key ingredient. The ablation that isolates our mechanism is the ticket-to-dense transfer test. Our account "wins" if projecting out the coupling in the dense model recovers at least half of the WER (or accuracy) gap between dense and ticket at matched clean performance. A random subspace of equal dimension, projected out the same way, should recover essentially none of it.

Why not just…?

Why not just train the dense model with noise augmentation and skip the mechanism? For deployment, that may well be the better choice, and we will say so if the augmented baseline matches the ticket. The question here is different, though. IMP is the only treatment we know of that produces robustness without exposure to the noise during training, since Audio Lottery's tickets were not trained on DESED. That points to a structural property of the network rather than a learned invariance. By running the same probes on the augmented model, we can see whether the two routes reach robustness through the same decoupling or through different ones.

Why not just use one-shot pruning, which is far cheaper? If one-shot or random pruning at matched density is equally robust, the effect comes from sparsity in general, and the expensive iterative procedure is unnecessary for robustness. That would be a real and useful negative result, and our matched-density controls are designed to detect it. Prior evidence leans the other way. One-shot pruning was clearly worse on clean WER in Audio Lottery, and only the iterative process lowered coupling in Perez et al. Neither paper, however, measured robustness to noise, so the question remains open.

Why not just train a small dense network? A capacity-limited network might simply lack the room to model noise. In that case the deletion account would hold, and decodability would drop. Our probes test this directly, and the matched-size dense control tests whether robustness follows parameter count or the pruning process.

Why not just probe the dense model and ablate its noise directions? This is exactly our ticket-to-dense transfer test, so we treat it as the experiment rather than an alternative. The open question is whether the noise directions in the dense model can be identified well enough on their own. If ablating probe-found directions in the dense model works as well as using geometry derived from the ticket, then IMP serves only as a discovery tool here, and dense-only probing is the cheaper route. We will report it as such.

Decisive Checks

Three results would settle the question.

First, we compare noise decodability between ticket and dense model at every layer. Near-equal decodability supports decoupling. A sharp drop in the ticket supports deletion.

Second, we look at the round-by-round curves. Robustness, the angle between the noise subspace and the task-output gradients, and (on the scene classifier) LOGDET should improve together across pruning rounds. All three should also reverse together below about 10% density, as Perez et al. found for LOGDET on vision models. If robustness persists at extreme sparsity while coupling rises again, the Fisher-geometry link is refuted for audio.

Third, the bidirectional transfer tests must clear the half-gap threshold relative to a random-subspace control. A positive result would give both an explanation of the noise bonus and a cheap, training-free robustness edit for dense audio models. A negative result would rule out the most natural mechanistic story and narrow the remaining possibilities.