Aligning Only the Bands That Still Differ: Subband Adversaries for Device-Robust Audio

When an audio model trained on one microphone is tested on another, accuracy collapses. On the DCASE 2020 mobile benchmark, a model trained only on the reference device scores 24.9% on unseen phones. The usual fix is domain-adversarial training. A second network, the discriminator, tries to tell which device a recording came from by looking at the model's internal features. The model is trained to fool it. This approach squeezes all the evidence about device identity into one pooled feature vector. Physically, though, a microphone mismatch is not spread evenly. It is a roll-off below some frequency, a resonance in one region, or a noise floor that dominates the top octaves. A single pooled discriminator cannot see where the mismatch lives. So it pushes the model to change everywhere, including frequency bands that were never corrupted. This plausibly explains why multi-device adversarial adaptation is reported to hurt accuracy on the original device.

We propose a different approach. First, remove the part of the device mismatch that a simple per-frequency normalization already removes. Then attach a small discriminator to each frequency band of an intermediate feature map. Each band's adversarial pressure is weighted by how well that band's discriminator can still identify the device. In short, normalize away the linear microphone coloring, and spend adversarial alignment only on the bands where a device fingerprint survives normalization.

Core Claim

The claim has two parts, and the second depends on the first.

The empirical part is a hypothesis we test before building anything. After frequency-wise normalization, the device information remaining in a spectrogram network's features is concentrated in a minority of frequency bands, not spread across all of them. This matters because a microphone's frequency response is mostly a fixed per-frequency gain. On a log-mel spectrogram that gain becomes an additive offset per frequency row, and per-frequency instance normalization largely cancels such offsets. What survives should be the nonlinear, level-dependent, or additive-noise effects, and those effects are band-localized.

The methodological part follows. If the leftover shift is band-localized, then per-band adversaries with weights set by each band's remaining discriminability should match or beat global adversarial adaptation on unseen devices, and should lose less accuracy on the source device.

Relation to Prior Work

Four papers bracket this idea.

Unsupervised Multi-Target Domain Adaptation for Acoustic Scene Classification is the closest adversarial work, and it motivates the proposal. It treats each target device as its own domain, prioritizes "hard" devices using distances computed from parallel recordings, and aligns a single global latent vector. Its authors report source-domain degradation. We keep the adversarial game but move it from the global vector to the frequency axis, and we require no parallel recordings.

FAN-Net also pairs a style-removal module with a global domain classifier trained through gradient reversal. Gradient reversal is a layer that passes features forward unchanged and flips the sign of the discriminator's gradient on the way back. FAN-Net works on MRI, separating image style from structure through a circular low-frequency mask on the 2D Fourier amplitude. It supplies the general recipe of "normalize style, then adversarially clean up the residue." Its isotropic mask and global classifier, however, ignore that a spectrogram's vertical axis is physical frequency.

Two normalization papers supply the first stage of our method and constrain the design. Residual Normalization, from Domain Generalization on Efficient Acoustic Scene Classification Using Residual Normalization and the QTI DCASE 2021 submission, blends frequency-wise instance normalization with a 0.1-weighted identity path. SubSpectral Normalization normalizes equal frequency subbands separately and beats a four-times-wider network. Neither paper ever looks at unlabeled target audio.

BWRFN learns per-frequency normalization weights with Bayesian uncertainty, using only labeled source data. It is evidence that frequency bins differ in how sensitive they are to domain shift, which is the premise our band weights rely on. It is also non-adversarial.

ALIGN shows, for drifting neural recordings, that attaching adversaries to intermediate representations rather than the final bottleneck helps. This supports placing our discriminators before frequency pooling.

Research Program

The backbone is BC-ResNet-ASC with ResNorm, the efficient acoustic scene model from Kim et al. We choose it because it keeps an explicit frequency axis through most of its depth. At a chosen intermediate layer, the feature map is split along frequency into $B$ contiguous bands, giving band features $h_b$. Each band gets a small convolutional discriminator $D_b$ that predicts device identity and receives gradients through gradient reversal.

The training objective is

$$\mathcal{L} = \mathcal{L}_{\text{task}} - \lambda \sum_{b=1}^{B} w_b\, \mathcal{L}_{\text{dom}}(D_b(h_b), d),$$

where $\mathcal{L}_{\text{task}}$ is scene-classification cross-entropy on labeled source clips, $\mathcal{L}_{\text{dom}}$ is device cross-entropy on source and unlabeled target clips with device label $d$, and $\lambda$ is the overall adversarial strength.

Each band weight $w_b$ is held fixed when computing gradients, so it acts as a constant multiplier. It is set from a running average of how much $D_b$'s held-out accuracy exceeds chance. The weights are normalized to sum to one and floored at zero. The effect is a feedback controller. A band that still carries a device fingerprint receives more alignment pressure. Once it is aligned, its discriminator falls toward chance and the pressure on it fades. A band that was never corrupted receives almost none. Unlike the multi-target method, the weights are estimated online from unlabeled data and need no parallel recordings.

Before any training, we run a probing study that can kill the project. On a frozen ResNorm backbone trained only on device A, we fit linear device classifiers on each band of each layer. We then measure how concentrated band-wise device accuracy is, using the entropy of above-chance accuracy across bands. We also check whether the concentration lines up with the known simulated impulse responses of devices S1–S6.

Baselines

The naive reference is the backbone trained on device A only, with no adaptation. The multi-target paper reports 24.9% for this setting on unseen devices S4–S6. We rerun it so all methods share one data pipeline.

The strongest existing adversarial method is MTDA from Yang et al.. It reports 41.7–42.3% accuracy on hard devices S1–S3 for MTDA-C2 against 35.1–37.1% for DANN, and 41.5% on unseen S4–S6 for MTDA-R. These are the figures to beat. We reimplement MTDA on our backbone with the same labeled device-A data, the same unlabeled target devices, and its parallel-recording distances. Note that this gives MTDA side information our method does not use. We also report the published numbers. Beating MTDA means higher S4–S6 accuracy with a smaller drop on device A, averaged over at least three seeds.

ResNorm alone, trained in the same unsupervised setting, is the strongest non-adversarial competitor. Its published 76.3% comes from the supervised DCASE protocol, which includes labeled target-device data, so that figure is not comparable and we rerun it. Global DANN and CDAN on the ResNorm backbone isolate whether the gain comes from being band-wise. CDAN conditions the discriminator on the classifier's predictions so that alignment respects class structure.

Two ablations target the key ingredient. The first uses per-band discriminators with uniform weights. The second uses adaptive weights computed on randomly permuted band assignments. If permuted bands do as well, the frequency localization is not what matters.

Every method gets the same tuning budget of 20 configurations over $\lambda$, learning rate, and $B$, selected on a validation split of the seen target devices, with identical training epochs and parameter counts within 5%.

Why not just…?

Why not just use frequency-wise normalization and skip the adversary? This is the most serious threat. If microphone shift is almost entirely a linear filter, then ResNorm or SubSpectral Normalization already removes it. The probing study settles the question directly. If device accuracy from per-band probes on ResNorm features is near chance in all bands, adversarial training has nothing left to remove, and we would report that negative result rather than the method. BWRFN shows that deterministic frequency normalization still leaves a gap on unseen far-field audio, which suggests a residue exists, but it does not show where.

Why not just put one global discriminator on top of ResNorm? This may capture most of the benefit. The comparison against global DANN and CDAN on the identical backbone is the decisive test. Our prediction is specific. The global version should match ours on S4–S6 only if the band-wise probes show device information spread evenly. When that information is concentrated, the global version should cost more accuracy on device A.

Why not just augment with random equalizer curves or device translation? Random per-band gain augmentation is cheap. Like normalization, though, it mainly covers linear coloring, which is the part we already strip. The GAN device translation in the QTI submission needs a model trained per known device. We include random-equalizer augmentation as a baseline, and also stack it with our method, to check whether the gains add.

Why not just widen the network? In SubSpectral Normalization, a four-times-wider CP-ResNet reached 83.2%, below the lightweight subband model's 83.6%. Even so, we match parameter counts across all methods so that capacity cannot explain any gain.

Proposed Experiments

The main benchmark is TAU Urban Acoustic Scenes 2020 Mobile. Labels come from device A only. Devices B, C, and S1–S3 are unlabeled targets, and S4–S6 are held out entirely. We report accuracy per device, on device A, and on S4–S6. We also report per-band discriminator accuracy after training, to verify that the weights went where the probes predicted.

Two results would decide the question. The first is the probe concentration result. The second is the sign of the gap between band-wise and global adversaries, read in light of that concentration. Across devices, we expect the band-wise advantage to grow as the probe entropy falls. If no such relation appears, the localization story is wrong, even if average accuracy improves.

A secondary test checks whether the idea transfers beyond acoustic scenes. We train on VoxCeleb and evaluate speaker verification on unseen far-field HI-MIA. The bar there is BWRFN's 8.15% equal error rate, against 12.65% for the R-vector baseline. In this setting reverberation is known to decay faster at high frequencies, as studied in the room reverberation and pitch estimation study, so the band weights should favor the upper bands.

Finally, we sweep the number of bands $B$ from 2 to 16 and the layer where the discriminators attach. Very early layers have narrow frequency receptive fields. Deep layers mix bands together, which should make band-wise and global discriminators behave alike, and this gives an internal consistency check.