Skipping Whole Bands and Frames: Input-Adaptive Sparsity for Dual-Path Spectrogram Models

Many of the best audio separation models look at a spectrogram (a grid of frequency by time) along one axis at a time. One layer lets each time frame attend across its own frequency bins, which captures pitch and harmonics. The next layer lets each frequency band attend across time, which captures rhythm and reverberation. These "dual-path" models, such as DasFormer and TF-GridNet, are accurate and expensive. The usual way to cut cost is to zero out individual weights, but on GPUs this rarely saves time: tracking scattered zeros costs more than the arithmetic it skips. We propose a different unit of sparsity. Per input and per layer, the model learns to skip whole frequency bands in the temporal path and whole time frames in the spectral path. Each band or frame is an independent sequence in the batch, so dropping one means handing a dense kernel a smaller dense batch. There are no indices or bitmaps to decode, and the speedup should track the fraction dropped.

The factorized structure is what makes this safe. A band skipped by one temporal layer is still updated by the spectral layer around it, so it keeps receiving harmonic context from its neighbours. The model never fully loses the band. It only spends less effort tracking that band over time in that layer. Silent frames, quiet high bands, and single-speaker stretches in sparsely overlapping speech are natural candidates to skip. What counts as skippable changes from mixture to mixture, which a fixed architecture cannot exploit.

Core Claim

In alternating spectral-temporal networks, the cheapest hardware-friendly sparsity is not inside the weight matrices. It is along the batch-of-sequences dimension that each path already creates. We claim three things. First, a learned gate that keeps the top fraction of bands (temporal path) and frames (spectral path) under a compute budget gives wall-clock speedups close to the FLOP reduction on GPUs and CPUs. Second, at matched compute, it loses less separation quality than either a fixed frequency downsampling schedule or 2:4 weight sparsity. Third, the gain depends on the input: it should be largest on mixtures with silence or partial overlap, and smallest under heavy reverberation, where energy smears across frames.

For a temporal-path layer, let $F$ be the number of bands, $T$ the number of frames, and $d$ the embedding width. The cost is roughly proportional to $F\,(T^2 d + T d^2)$. The spectral-path layer costs roughly $T\,(F^2 d + F d^2)$. Keeping a fraction $\rho_F$ of bands and $\rho_T$ of frames therefore scales these costs by $\rho_F$ and $\rho_T$. This follows directly because the per-sequence cost is unchanged and only the number of sequences shrinks. The budget constraint is

$$\rho_F\,C_{\text{temp}} + \rho_T\,C_{\text{spec}} \le B,$$

where $C_{\text{temp}}$ and $C_{\text{spec}}$ are the dense costs of the two paths in a layer and $B$ is the target. A per-layer budget lets shallow layers, which DasFormer reports learn local texture, be treated differently from deep layers that track speaker identity.

Relation to Prior Work

The direction that motivates this work is efficiency in factorized audio models. Band-SCNet reaches real-time music separation by downsampling frequency with fixed, non-uniform strides of 1, 4 and 16 for low, mid and high bands. That is the static, hand-designed version of our idea. We make the allocation per input and per path, and we skip bands rather than pooling them. Mel-RoFormer and the band-split RoPE Transformer also shrink the frequency axis by grouping bins into fixed bands. These provide the backbones we gate, not competitors to the gating itself.

The closest work on pruning separation models is SepPrune. It learns channel-pruning masks with Gumbel-Softmax on convolutional and recurrent separators. It reports a 50.2% memory reduction but only a 1.13× inference speedup, and it never touches dual-path attention models. We borrow its lesson that the encoder and decoder should be protected, and its use of differentiable masks. Our masks act on sequences rather than channels, and they are recomputed for every input.

The direction that constrains the design is unstructured sparsity overhead. Okanovic et al. show that cuSPARSE is uncompetitive at neural-network sparsity levels. Their SMaT library beats dense cuBLAS only above about 78% sparsity, and only after a reordering step that suits static matrices. Attention patterns, and our gates, change with every input, so that reordering is ruled out. Grimaldi et al. supply the method we generalize. They zero out whole spatial positions across all channels so that the matrix multiply sees empty columns, giving 1.25 to 1.9× CPU speedups on ResNet18. Their unit is an isotropic pixel. Ours is an entire row or column of the time-frequency grid, chosen to match whichever axis the current layer attends along. Weight-level structured methods, SR-STE 2:4 sparsity and tile-wise sparsity, are complementary. They shrink the projection matrices, but they do nothing to the $T^2$ or $F^2$ attention term, and they cap at roughly 2× on supported hardware. We treat them as competitors at matched compute and also as additions on top of our method.

Research Program

We add a small gate before each spectral-path and temporal-path block. For each band (or frame), the gate pools that sequence's embeddings into a scalar score. The top $\lceil \rho F \rceil$ bands (or $\lceil \rho T \rceil$ frames) pass through the block, and the rest take the residual path unchanged. Training uses a straight-through top-k estimator. Gate scores multiply the block output so that the gate receives gradients. The per-layer keep ratios are learned under the budget penalty above.

At inference, the kept sequences are gathered along the batch dimension. The dense block runs on the smaller batch, and outputs are scattered back. This adds one gather and one scatter per block, and we will profile both explicitly.

The backbones are DasFormer, with its frame-wise spectral attention and band-wise temporal attention, and TF-GridNet. In TF-GridNet, band skipping applies to the sub-band temporal recurrent layer. Frame skipping applies to the intra-frame full-band recurrent layer and, as dropping queries, to the cross-frame attention. For music we use Mel-RoFormer. All gated models start from dense pretrained checkpoints and are fine-tuned. This follows SepPrune's finding that inheriting weights from a trained model converges much faster than training small models from scratch.

The analysis has three parts. We will ask what the gates choose (silence, high bands, single-talker stretches, reverberant tails). We will measure how the learned keep ratios vary with depth. And we will check whether causal, streaming variants keep the benefit when the gate only sees the past.

Baselines

The naive reference point is the dense backbone itself, evaluated with its published numbers and our own measured latency. For TF-GridNet the file reports 23.5 dB SI-SDRi on WSJ0-2mix (TF-GridNet). For DasFormer Plus it reports 17.3 dB SI-SDRi on WHAMR! (DasFormer). For Mel-RoFormer it reports 13.29 dB SDR on MUSDB18-HQ (Mel-RoFormer). We will retrain or reproduce these on our hardware, so that every comparison uses the same checkpoint lineage.

The strongest competitor on the efficiency question is fixed frequency downsampling of the Band-SCNet type (Band-SCNet), retrofitted into the same backbone. We will use non-uniform band strides, plus a fixed frame stride for the spectral path, tuned so that FLOPs match our gated model. Its fine-tuning schedule, data and hyperparameter search budget will equal ours. On its own architecture the file reports 7.79 dB SDR with RTF 0.478 on an i7 CPU. That number comes from a much smaller model and is only context, so the decisive comparison is the retrofitted, compute-matched version.

The second competitor is 2:4 weight sparsity trained with SR-STE on all projection and feed-forward weights of the separator. We will measure it with sparse tensor cores on an Ampere-class GPU. No reported result was found on Lacuna for 2:4 sparsity on any dual-path audio model, so we will produce that number ourselves. The third is SepPrune-style channel pruning (SepPrune) applied to the same backbone at matched FLOPs. Its reported 1.13× speedup on its own backbones is the bar for "pruning that barely moves wall-clock."

Two ablations isolate the key ingredient. The first is isotropic token skipping in the spirit of Grimaldi et al.. Here the gate drops individual time-frequency positions instead of whole rows or columns. This tests whether axis alignment matters for both quality and speed. The second keeps our gate but freezes it to a dataset-average mask. This tests whether input adaptivity matters.

We beat a baseline if, at equal measured latency (not just equal FLOPs), we lose at least 0.5 dB less SI-SDRi or SDR than it does. Alternatively, at equal quality we must be at least 1.3× faster in wall-clock time. These margins are our own choice, set as practically meaningful for separation.

Why not just…?

Why not just downsample high frequencies with a fixed schedule, as Band-SCNet does? This is the real threat. For music, high bands may be skippable almost always, and then a learned gate adds nothing. The frozen-mask ablation and the compute-matched static baseline settle it. If static allocation matches the dynamic gate on MUSDB18-HQ and on Libri2Mix, our contribution narrows to "axis-aligned, per-path static skipping beats weight pruning," and we will report it that way. We expect adaptivity to matter mainly on speech with silence and partial overlap. The sparse-overlap LibriMix variants (LibriMix) are where a static schedule cannot know which frames are empty.

Why not just use 2:4 weight sparsity, which already has hardware support? It acts only on the linear projections and is capped near 2× math throughput. It leaves the quadratic attention term over $T$ or $F$ untouched. That term dominates temporal attention on long utterances. It is also unproven on these models: no Lacuna result reports 2:4 sparsity on a dual-path audio network. The two methods compose, so we will also report gated plus 2:4. If 2:4 alone wins at matched latency, that is a clear negative result for us.

Why not just use a smaller model? Hand-shrunk models pay heavily in quality. Band-SCNet sits at 7.79 dB SDR, while Mel-RoFormer's 9.1M-parameter small variant exceeds 11 dB and the full model reaches 13.29 dB on MUSDB18-HQ. We will nevertheless include a dense model with fewer layers or a narrower width at matched latency. If it matches our gated model, adaptive skipping is not worth its complexity.

Why not just use an unstructured sparse kernel like SMaT? Its speedups depend on row reordering, which suits static matrices. Its crossover with dense cuBLAS sits near 78% sparsity (Okanovic et al.). Our sparsity changes per input and sits at 30 to 70%, so we do not expect it to compete. We will verify this with one measured comparison rather than assume it.

Proposed Experiments

Training and evaluation use WSJ0-2mix and WHAMR! with DasFormer and TF-GridNet, Libri2Mix and its sparse-overlap variants, and MUSDB18-HQ with Mel-RoFormer. We report SI-SDRi or SDR, FLOPs, A100 latency at batch sizes 1 and 16, CPU real-time factor, and gather/scatter overhead as a share of runtime. Each method is swept over keep budgets from 0.3 to 0.9, giving a quality-versus-latency curve per method.

The first decisive check is whether our measured speedup reaches at least 80% of the FLOP reduction. Weight-sparse baselines are expected to fall well short of this. If ours does not, the "no overhead" premise fails.

The second check is the overlap sweep on sparse LibriMix. If adaptivity matters, the advantage of the dynamic gate over the static mask should grow as overlap falls.

The third check is WHAMR! against its anechoic counterpart. Reverberation smears energy across frames, so frame skipping should help less there. We predict the learned frame keep-ratio rises under reverberation while the band keep-ratio stays low. If both collapse to near 1, the method does not survive realistic acoustics.

The last check is a causal streaming variant of the gated TF-GridNet, measured in real-time factor on CPU against the Band-SCNet reference of 0.478. It tests whether the idea carries over to the edge setting that motivates it.