Frequency-Dependent Temporal Anti-Aliasing for Spectrogram Networks

Convolutional networks for audio treat a spectrogram, a picture of sound with time on one axis and frequency on the other, as if it were an ordinary image. Two things follow from this. The first is that when these networks shrink the time axis with strided convolutions or pooling, they discard samples without first removing fast temporal fluctuations. Those fluctuations fold back into slower ones, an effect called aliasing, so delaying the audio by a few milliseconds can change the prediction. The second is that any fix taken from vision would apply one blur uniformly everywhere, including along frequency. Frequency is not like image space, though. A pattern that moves up the frequency axis becomes a different sound, and fast transients such as fricatives, clicks and onsets are concentrated in some bands, while slow modulations dominate others.

This note proposes a downsampling operator that treats the two axes differently. Along time, it applies a low-pass filter before subsampling so that outputs stay stable under small delays. The filter's cutoff is not constant: it is a staircase function of frequency, meaning a few contiguous frequency bands each get their own temporal cutoff, and the band edges are learned. Along frequency, the operator neither blurs nor mixes across band edges, so absolute frequency position is preserved. The staircase form comes from piecewise-constant additive models. The claim to test is that the right amount of temporal smoothing depends on frequency band, and that a few learned steps capture this dependence better and more cheaply than either one global filter or a filter predicted from content.

Core Claim and Where It Sits

Three pieces of prior work motivate the idea. Making Convolutional Networks Shift-Invariant Again splits each stride-2 operation into a dense evaluation, a fixed blur and a subsampling step (BlurPool). This improved ImageNet accuracy and shift consistency. Its blur is identical on both axes and at every location. Anti-Aliasing Regularization in Stacking Layers brings the same logic to speech recognition, applying a single 7-tap FIR filter along time before frames are stacked. It reports up to 5.1% relative word error reduction on compressed audio, but the filter is the same across all 640 feature dimensions. Why do deep convolutional networks generalize so poorly to small image transformations? shows that one-pixel shifts flip top-1 predictions up to 30% of the time, and that BlurPool reduces this without removing it. That result limits how much any anti-aliasing fix can be expected to deliver.

Work on non-stationary spectral processing supplies the other half of the argument. SubSpectral Normalization (SSN) gives each of $S$ equal frequency sub-bands its own normalization statistics. It finds a "Goldilocks" range of $S=2$ to $4$, beyond which finer slicing hurts. Full-frequency dynamic convolution (FFDConv) generates convolution kernels that depend on frequency position. Neither of these touches downsampling or measures stability under time shifts.

The closest existing method is Delving Deeper into Anti-aliasing in ConvNets. It predicts a low-pass filter at every spatial location from the local content. If it were applied to spectrograms, its filters would already vary across frequency rows. Our proposal differs in two ways. The cutoff depends on frequency position rather than on content, so it is a fixed and interpretable property of each band. It is also constrained to a few constant steps, which costs almost nothing at inference, whereas Zou et al. report an added latency of 4.3 to 6.4 ms. Whether content-adaptivity or position-dependence matters more is an empirical question, and the experiments below are designed to answer it.

The staircase parameterization comes from FAST, which fits step-shaped shape functions with group-sparse selection of cut points, and from the fixed 64-bin shape functions of GAMformer. Those papers are about tabular data. Here the "feature" is frequency index and the "shape function" maps it to a temporal cutoff.

The Operator

Take a feature map with frequency index $f$ and time index $t$. For stride-2 temporal downsampling, we compute

$$y[f,t] = \sum_{k} h_{c(f)}[k]\, x[f,\, 2t-k], \qquad c(f) = \sum_{j=1}^{K} c_j\, \mathbf{1}[b_{j-1} \le f < b_j],$$

where $h_c$ is a symmetric low-pass kernel with cutoff $c$, $c_j$ is the cutoff for band $j$, $b_0<\dots<b_K$ are the band edges, and $K$ is the number of steps. The kernel $h_c$ is interpolated between a fixed bank of windowed-sinc kernels, which keeps it a valid low-pass filter with non-negative gain.

The cutoffs are learned directly. The band edges are learned through a temperature-annealed soft assignment that hardens into a true staircase by the end of training. We also add a group-sparsity penalty on differences between adjacent cutoffs, in the spirit of FAST, so the network chooses its effective $K$ instead of us fixing it.

Frequency downsampling, where the architecture needs it, is done by pooling only within each learned band. This avoids blending physically distinct bands, and absolute frequency position is deliberately left encoded.

Padding along time is also a concern. On Translation Invariance in CNNs shows that zero padding lets networks read off absolute position, so we test circular and zero padding along time as a controlled factor. Along frequency, absolute position is wanted and needs no such control.

Research Program

The first stage checks the premise before anything is built. We take trained baseline CNNs on the three audio benchmarks below and measure, at every downsampling layer, the energy above the post-stride Nyquist frequency in each feature map's temporal spectrum, separately for each frequency row. If this aliasing energy is roughly flat across frequency, the case for a frequency-dependent cutoff collapses and we stop. If instead it shows band structure, the location of that structure predicts where the learned edges should land.

The second stage builds the operator and drops it into Res15, CP-ResNet and the DESED convolutional recurrent network, replacing every temporal stride. The third stage is analysis. We compare the learned edges and effective $K$ with SSN's Goldilocks range. We also check whether the learned cutoffs follow the band-wise aliasing profile measured in stage one.

Baselines

The naive reference point is each unmodified architecture, trained with its standard recipe and its standard augmentation, including SpecAugment where the recipe uses it. SSN reports two such anchors: Res15 with batch normalization at 95.8% on Google Speech Commands, and CP-ResNet at 82.3% on TAU Urban Acoustic Scenes 2019.

The strongest frequency-aware baselines are SSN and FFDConv. SSN reaches 97.5% on Speech Commands with $S=4$ and 83.6% on TAU with $S=2$ (SubSpectral Normalization). FFDConv reaches PSDS1 0.436 and PSDS2 0.685 on DCASE 2022 Task 4 (Full-frequency dynamic convolution). PSDS1 measures how well event timing is detected, and PSDS2 measures audio tagging. These are the numbers to beat on accuracy. We will reproduce them under our training pipeline and report our reproduced values alongside the published ones.

The anti-aliasing baselines come next. The first is 2D BlurPool with a Binomial-5 kernel. The second applies BlurPool along time only, and the third is the uniform 7-tap Remez temporal filter from Anti-Aliasing Regularization in Stacking Layers. The fourth is Zou et al.'s content-aware filter placed at the same layers. We found no reported result on Lacuna for any of these four on Speech Commands, TAU or DESED, so all of them will be run by us.

The key ablations isolate the staircase. They are $K=1$, which gives a single learned cutoff; an unconstrained per-row cutoff, which is smooth and has no steps; a fixed equal-width staircase matching SSN's partition; and SSN combined with time-only BlurPool.

Fairness rules apply to every comparison. All variants share data, schedules and augmentation. Each gets the same hyperparameter search budget: equal trials, with filter-specific settings tuned per method. Results are reported as mean over five seeds. Parameter count and measured latency are reported for every method.

Beating the baselines means two things. The first is a time-shift consistency gain over time-only BlurPool that exceeds the seed spread, with no loss in accuracy or PSDS. The second is matching or exceeding SSN and FFDConv on accuracy when combined with them. An improvement in stability alone, with accuracy pinned at the time-only BlurPool level, would count as a partial result.

Why not just…?

Why not just blur along time with one uniform filter? This is the most serious threat. The $K=1$ ablation and the time-only BlurPool baseline test it directly. If they match the staircase on both consistency and accuracy, the frequency dependence is unnecessary, and we would report the finding as "asymmetric but uniform anti-aliasing suffices." The stage-one measurement of aliasing energy per band tells us in advance whether a difference should be expected.

Why not just augment with random time shifts and SpecAugment? Why do deep convolutional networks generalize so poorly to small image transformations? finds that augmentation widens the set of inputs on which the network is stable but does not remove one-pixel fragility. The speech-recognition study also reports aliasing in a model that was already trained with heavy augmentation. Augmentation remains in every arm here, so any gain we measure comes on top of it.

Why not just use content-aware anti-aliasing, which already varies by location? This is a real competitor, and it is run as a baseline. Our bet is that in audio, where to smooth is mostly a matter of which band a feature is in, not of what the content is. If Zou et al.'s method wins clearly, content matters more than position, and we would then test a hybrid in which the staircase sets a floor that content adjusts.

Why not just remove temporal striding? Networks with no subsampling are perfectly shift-invariant on small data, but the cost becomes impractical at realistic resolution. We will report the stride-1 variant on Speech Commands as an upper bound on consistency, together with its compute cost.

Decisive Checks

Time-shift consistency is measured by shifting the waveform by fractions of the hop length. This creates genuine sub-frame shifts that no amount of frame-level augmentation covers. Two quantities are computed. On Speech Commands and TAU, it is the probability that the top-1 label is unchanged. On DESED, it is the jitter in predicted onset times.

A symmetric check guards against the operator erasing useful frequency information. Shifting spectrogram rows, or pitch-shifting the audio, should change predictions. We measure this sensitivity and require it not to fall relative to SSN.

The central test is whether the staircase beats $K=1$ and time-only BlurPool on consistency at equal accuracy. The secondary test is whether its learned edges settle at two to four bands, consistent with SSN's Goldilocks range, and whether they align with the stage-one aliasing profile. A null result on the central test is informative in its own right: it would show that frequency non-stationarity matters for normalization and convolution, but not for anti-aliasing.