Energy-Blind Pruning: Choosing 2:4 Sparsity Masks for Neural Vocoders by Spectrogram Structure

Modern GPUs can run a weight matrix twice as fast if exactly two of every four consecutive weights are zero. This pattern is called 2:4 sparsity, and the hardware that skips the zeros is the "Sparse Tensor Core." Deciding which two weights to drop in each group of four is usually done by asking which choice best preserves each layer's numerical output, measured as squared error. This proposal argues that for audio generators, and especially neural vocoders (networks that turn a spectrogram into a waveform), squared error is the wrong judge. It is dominated by loud, low-frequency energy. Masks chosen this way will tend to sacrifice the quiet upper harmonics and fine formant (vocal-resonance) trajectories that listeners hear as crispness and breath.

We propose to choose 2:4 masks, and to run a short recovery phase, with a structural similarity objective computed on output spectrograms. This objective is the audio adaptation of the image metric SSIM, which compares local brightness, local contrast and local pattern separately, with the latter two normalized by local energy. The claim is narrow and testable. At matched hardware speedup, structure-guided masks should keep more perceived quality than masks chosen by layer-wise squared error. The gain should show up specifically in low-energy high-frequency bands and in harmonically dense material such as singing.

Where the two directions actually meet

Structural similarity losses in audio have been used only to train dense models. DelightfulTTS adds structural terms to its mel-spectrogram loss so that its 48 kHz pipeline keeps acoustic detail. PL-TTS pairs an $L_1$ loss with SSIM to stop diffusion TTS from over-smoothing. SPA-SVC uses SSIM on harmonic texture to suppress hoarseness under wide pitch shifts. These papers motivate our premise that point-wise losses lose spectral texture. None of them prunes anything. On the evaluation side, ViSQOL v3 builds a quality metric from spectrogram patch similarity. It also constrains us in two ways. It needed explicit silence thresholds because log-spectrogram similarity overreacts to inaudible background energy. And it warns that generative vocoders produce plausible detail that does not line up sample-for-sample with the reference.

Hardware-aligned pruning chooses masks by other criteria. Some methods use magnitude. Others use layer reconstruction error: ARMOR alternates least-squares mask updates with small block-diagonal wrapper matrices. Others use the task loss: Spartan uses a soft-to-hard optimal-transport relaxation, and Accelerating Transformer Pre-training with 2:4 Sparsity uses masked gradient decay to stop masks from flickering. All of this work is on vision and language models. Towards Fully Sparse Training is the closest in spirit, since it exploits the similarity of neighbouring image pixels to fix 2:4 activation masks. Its "similarity" is a property of isotropic image feature maps, though, not a loss, and it never touches spectrograms.

The two speech-pruning papers in the file use coarse removal of heads and channels rather than 2:4 patterns, and neither uses a structural objective. Personalized Lightweight TTS prunes FastSpeech 2 heads and FFN units. Task-Agnostic Structured Pruning of Speech Representation Models prunes WavLM. Both report that acoustic and speaker information is what breaks. The first had to leave the hidden dimension dense. The second saw speaker-ID and emotion-recognition accuracy drop as lower layers were thinned. That pattern is the kind of fidelity loss we expect a structure-blind criterion to cause.

Core Claim

Layer-wise squared error is energy-weighted, so it implicitly ranks high-energy spectral regions as more important. Masks chosen under that criterion should keep the network's handling of the fundamental and low harmonics intact and let quiet upper harmonics and onsets degrade. A structural similarity objective on the output spectrogram compares contrast and pattern after per-patch normalization. It should therefore distribute the pruning damage more evenly in perceptual terms.

The new element is using this objective as the mask-selection signal under a fixed hardware pattern, in a low-budget regime. "Low-budget" means a calibration set plus a short recovery run, not full GAN retraining. That regime is how 2:4 pruning is typically deployed. It is also the regime in which the vocoder's own adversarial loss is not available to repair the damage.

Research Program

We target convolutional GAN vocoders of the HiFi-GAN family. These are the kind of high-fidelity vocoder DelightfulTTS relies on. Their convolutions are executed as matrix multiplications, so the 2:4 pattern applies along the input-channel-times-kernel dimension.

Let $x$ be an input mel-spectrogram, $f_W(x)$ the vocoder output, and $\mathcal{S}(\cdot)$ a log-magnitude short-time spectrogram. Following ViSQOL v3, frames whose energy falls below a fixed silence threshold are excluded. The structural loss against the dense teacher $W_0$ is

$$\mathcal{L}_{\text{struct}}(W) = \mathbb{E}_{x}\Big[1 - \tfrac{1}{|P|}\sum_{p \in P} \mathrm{SSIM}_p\big(\mathcal{S}(f_W(x)), \mathcal{S}(f_{W_0}(x))\big)\Big],$$

where $P$ is the set of non-silent time-frequency patches and $\mathrm{SSIM}_p$ is the usual product of brightness, contrast and pattern terms on patch $p$. Patches are rectangular rather than square, because time and frequency axes are not interchangeable.

Comparing against the teacher rather than the recording sidesteps ViSQOL's warning about generated detail that does not align with the reference. The objective asks the pruned network to reproduce its own dense self.

Masks are chosen in $K$ stages. Sparsity tightens from dense, through 4:8-style intermediate patterns where the hardware library supports them, to 2:4. At each stage, each weight $w_i$ gets a saliency $s_i = w_i^2\,\mathbb{E}[g_i^2]$, where $g_i = \partial \mathcal{L}_{\text{struct}}/\partial w_i$ is evaluated at the current partially pruned weights. This is the familiar Fisher-style estimate of how much the loss rises if $w_i$ is removed. Within each group of four, the two highest-saliency weights are kept. The same objective then drives a short recovery fine-tune, with masked gradient decay borrowed from Hu et al. to suppress mask flickering.

The design is deliberately modular. If the hypothesis holds, the same objective can replace squared error inside ARMOR's mask update.

We also run a diagnostic analysis to test the mechanism directly, not just the end metric. For each method we compute per-frequency-band error between pruned and dense outputs. We expect squared-error masks to show error concentrated above roughly 4 kHz and in low-energy frames, with structure-guided masks flatter.

Baselines

The naive reference point is magnitude 2:4 pruning: keep the two largest-magnitude weights per group, with no recovery. It gives the hardware speedup for free and shows how much quality naive pruning destroys.

The strongest existing competitor is layer-wise reconstruction pruning in the style of ARMOR, including its block-diagonal wrappers. It is the most accurate post-training 2:4 method in the file. On language models it reports a perplexity-gap reduction of nearly 50% over the next-best baseline on Llama-2-13B. It reports a layer speedup of 1.57x against 1.86x for naive 2:4 on Qwen 2.5-14B, with 3–5% wrapper overhead. There is no reported result found on Lacuna for ARMOR, or any 2:4 method, on a neural vocoder, so we will establish that number ourselves. We will run it on the same calibration utterances, with the same number of mask-update iterations and the same recovery-step budget as our method. Beating it means equal or better listener MOS together with lower high-band error at equal measured speedup. Because ARMOR's wrappers cost latency, we will report it both with and without them.

The ablation that isolates our key ingredient keeps the entire staged Fisher procedure but swaps the objective to an $L_1$ loss on the same log spectrogram against the same teacher. This alternative is also energy-compressed. A second ablation uses multi-resolution STFT loss, the standard vocoder reconstruction term. A third removes only the silence threshold.

As an upper reference rather than a competitor, we report full recovery training with the vocoder's original adversarial and mel losses at ten times the compute. This shows what the cheap regime leaves on the table.

Why not just…?

Why not just fine-tune the pruned vocoder with its original GAN and mel losses? The adversarial loss already fights over-smoothing, so with enough training it may close most of the gap regardless of which mask was chosen. This is a real threat. We do not argue against it; we bound it. The upper-reference run shows how much quality full retraining recovers. Our claim is confined to the calibration-plus-short-recovery budget. If full GAN retraining from magnitude masks matches our method at a modest extra cost, the contribution shrinks to "a cheaper path to the same place," and we will report it that way.

Why not just use $L_1$ on the log spectrogram, which is already energy-compressed? This is the most dangerous alternative to the mechanism claim. Taking the log already reduces the dominance of loud bands, so much of SSIM's benefit could come from the log alone. The matched-procedure $L_1$-on-log ablation settles it directly. If it equals the structural objective on MOS and band-wise error, we would conclude that the patch-level contrast and pattern normalization adds nothing for mask selection. We would then drop the SSIM claim and state that log-domain teacher matching is what matters. PL-TTS's finding that $L_1$ alone over-smooths dense diffusion outputs suggests a difference, but it was a different training regime and is not proof here.

Why not just prune coarsely, removing whole channels, and get dense speedups everywhere? The two speech papers show that coarse removal breaks exactly the acoustic and speaker information we care about. Personalized TTS could not prune the hidden dimension at all. WavLM pruning cost speaker-ID and emotion accuracy. Coarse pruning has the advantage that it needs no sparse kernels, so we include channel pruning at matched measured latency, not matched parameter count, as an extra comparison.

Why not just quantize? Quantization is orthogonal on Sparse Tensor Cores, and BMCook reports that combining 2:4 sparsity with quantization and distillation reached nearly 12x efficiency with over 97% retention on T5-3B. We will report the 2:4 results with and without INT8 rather than treat quantization as a rival.

Proposed Experiments

We train dense vocoders on LibriTTS-R for multi-speaker speech. Its restored audio reaches a ground-truth MOS of 4.09 on test-other, so it is clean enough that high-band losses are audible. We test speaker and accent preservation on VCTK, and use the Blizzard 2021 48 kHz Spanish data for full-band fidelity. As a harmonic stress test we use GTSinger, whose breathy, falsetto and vibrato annotations let us check whether quiet, structured content survives pruning.

There is a circularity risk. ViSQOL is itself a spectrogram-patch similarity metric, so a method optimizing structural similarity could win on ViSQOL without sounding better. ViSQOL scores will therefore be reported but will not be decisive. Primary metrics are listener naturalness MOS and side-by-side preference against the dense model. Secondary metrics are ASR word error rate, speaker-embedding similarity, singing-technique recognition accuracy on GTSinger, and the per-band error analysis above.

Efficiency is measured as wall-clock real-time factor on an Ampere-class GPU using Sparse Tensor Core kernels. We report both per-layer and end-to-end figures, because Hu et al. found that a 2x matrix-multiply speedup became only 1.21x end-to-end for GPT-2 once unpruned operations were counted. Vocoders' activation functions and upsampling may cap gains similarly.

The decisive check has three parts. Structure-guided masks must beat ARMOR-style masks on paired MOS at equal measured speedup. They must also beat the $L_1$-on-log ablation. And they must show the predicted flattening of high-band error, with the largest margin on GTSinger. If the MOS gain appears without the band-wise signature, the mechanism story is wrong even if the method works. If the $L_1$-on-log ablation ties, the SSIM-specific claim is refuted.