Audio that a general-purpose model must handle spans whispers, jet engines, solo piano and orchestras, so the first stage of almost every audio network rescales loudness. Recent front-ends do this adaptively, adjusting their compression frame by frame. That works well for robustness, but it has a cost that nobody has measured: every adaptive normalizer divides out the very level and contrast information that some tasks need. Speaker identity wants loudness removed. Musical dynamics and environmental events such as gunshots or machinery want it kept. We propose a front-end, modeled on how sensory neurons adapt, that normalizes each frequency band quickly while exporting the quantities it divided out as a separate, slowly varying signal. The downstream model then sees both the normalized detail and the background statistics.
The biological model comes from recordings of a fly motion neuron by Fairhall et al.. That neuron rescales its input-output curve to the stimulus variance within 100–200 ms. Its firing rate adapts much more slowly, on a timescale that grows with the period of the environmental change, and it carries a separate report of the background variance. Information stays near 1.4 bits/spike throughout. Our bet is that the tension between domains documented by HEAR and HARES comes partly from each front-end committing to one normalization policy. A two-timescale design with the adaptation state exported should shrink that tension.
Core Claim
Existing adaptive front-ends fall into two groups. Ada-FE adjusts the sharpness (Q-factor) of Gabor filters through a roughly 1.5K-parameter feedback controller. LEAF-APCEN uses a small recurrent network (a bidirectional GRU plus an MLP) to predict the compression exponent and divisor power of per-channel energy normalization. Per-channel energy normalization is a band-wise automatic gain control; we call it PCEN below. Both treat adaptation as something to apply and then discard. Both use a single controller operating over one time horizon. Neither measures what information its normalization destroys.
We make two claims, and each can be tested on its own. First, a front-end that exports its adaptation state (running band means and variances) beats an equally adaptive front-end that does not, specifically on tasks and conditions where absolute level or background statistics carry label information. It should not lose on tasks where they do not. Second, splitting the adaptation into a fast gain stage and a small bank of slow context trackers beats a single-timescale controller when the acoustic statistics change at varying rates. A single controller given the same context length is the comparison. If the first claim holds but the second fails, the contribution reduces to "export your normalizer state." That is still a useful and cheap result, and we will report it as such.
Research Program
Let $x_{t,f}$ be the log energy of band $f$ at frame $t$ from a learnable Gabor filterbank. We keep running exponential averages of the band mean at a fast timescale ($\mu^{\text{fast}}_{t,f}$, time constant near 150 ms) and at $K$ slow timescales ($\mu^{(k)}_{t,f}$, time constants of roughly 0.5, 2 and 8 s). We also keep a fast running standard deviation $s_{t,f}$. The normalized output is
$$y_{t,f} = \frac{x_{t,f} - \lambda_f\,\mu^{\text{fast}}_{t,f} - (1-\lambda_f)\,\bar\mu^{\text{slow}}_{t,f}}{s_{t,f} + \epsilon},$$
where $\bar\mu^{\text{slow}}_{t,f}$ is a learned convex mix of the slow means, $\lambda_f \in [0,1]$ is a learned per-band weight, and $\epsilon$ is a small constant. Subtracting the mean is the "parallel shift" that Skrzypek described for cone feedback. Dividing by the running spread is the variance rescaling Fairhall et al. measured.
The per-band weight $\lambda_f$ is a loose functional analogue of the integrator/differentiator distinction that Lundstrom et al. derived in conductance space. Near 1, a band ignores sustained level and responds to fluctuations. Near 0, it tracks level against a slow baseline. We borrow the concept and do not simulate conductances. Their boundary was derived for point neurons under stationary input and has no direct audio parameterization.
The side channel is the exported state $c_{t,f} = [\mu^{\text{fast}}_{t,f}, \mu^{(1..K)}_{t,f}, \log s_{t,f}]$. It enters the backbone through a separate low-rate input stem and is fused at an intermediate layer. The normalized detail and the context therefore remain separable, and we can remove either at test time for analysis. The front-end adds only a few hundred parameters beyond the filterbank. It is causal, so it can run in streaming deployment, which answers the "static deployment" limitation noted for engineered front-ends.
We will build it in two settings. In the first, we replicate the downstream setups of Ada-FE and LEAF-APCEN: small EfficientNet-B0 and MobileNetV2 backbones trained per task. In the second, we pretrain a moderate encoder with a contrastive objective. HARES found contrastive objectives transfer better across domains than supervised cross-entropy, and found batch normalization harmful to transfer. We therefore follow its normalizer-free choice. We then evaluate the frozen encoder with the HEAR linear/shallow-probe protocol.
Baselines
The naive reference is a fixed log-Mel spectrogram, together with a static learnable front-end, LEAF with learnable PCEN. Ada-FE and LEAF-APCEN both compare against LEAF, so it anchors every comparison.
The strongest existing competitor for robustness is LEAF-APCEN. It is the most recent adaptive-compression front-end, and it was evaluated on exactly the stress we care about: babble noise, background music and ±8 dB loudness jumps every 250 ms. Under those "Complex Acoustic Conditions" on ESC-50 it reports 55.75% accuracy, against 40.00% for fixed LEAF and 38.50% for learnable PCEN. On clean VoxCeleb1 it reports 49.84%, against 41.34% for LEAF (Meng et al.). Ada-FE is the strongest competitor on clean cross-domain classification. It reports relative gains over LEAF of 17.1–19.7% on GTZAN, 6.4–13.5% on IEMOCAP, and 34% on VoxCeleb1 with MobileNetV2 (Zhang et al.).
We will re-run both from their described architectures rather than copy their numbers. Everything will share data splits, backbone, training epochs and a matched hyperparameter search budget of the same number of trials per method. Where our re-run falls short of a reported number, we say so. For the frozen-encoder setting, the reference is the HARES average of 74.6 for Slowfast NFNet-F0 against 71.7 for a Slowfast ResNet50 (Wang et al.). Our compute will be smaller, so we treat this as context and not a target. The comparison that counts is our front-end against log-Mel and against LEAF-APCEN under the same pretraining. The file reports no per-model HEAR task numbers, so no reported result is found on Lacuna for any baseline's per-task HEAR score. HEAR comparisons will be internal.
Three ablations isolate the ingredients. The first uses the same front-end with the side channel removed, which tests the first claim. The second collapses the slow bank to a single timescale while keeping total context length equal, which tests the second claim. The third fixes $\lambda_f$ to 0 or 1 in all bands. Beating LEAF-APCEN means matching it within noise on its complex-conditions ESC-50 protocol while exceeding it on the level-sensitive tests below, with gains that exceed seed-to-seed variation over five seeds.
Why not just…?
Why not just append a loudness or utterance-statistics feature to an existing adaptive front-end? This is the most serious threat. Appending global statistics, such as per-clip band means and variances or a loudness track, to LEAF-APCEN output could recover most of the benefit. We include it as a baseline in two forms: clip-level statistics, and frame-level raw log energy concatenated to the input. If either matches our front-end, the lesson is that exporting level information matters and the biological structure does not. The two-timescale question would then be settled negatively. Our reason to expect a difference is that clip-level statistics cannot follow statistics that shift within a clip. Raw energy leaves the backbone to relearn the fast/slow separation itself. The variance-switching test below is designed to expose exactly this.
Why not just give LEAF-APCEN's recurrent controller more context? A GRU can in principle learn multiple timescales on its own. We will run LEAF-APCEN with its controller widened and its input window lengthened to match our longest time constant, and also with its hidden state exported to the backbone. If this matches us across modulation periods, the fixed timescale bank adds nothing beyond interpretability and streaming cost. We would then report that as the finding.
Why not just rely on gain augmentation and scale, as with large pretrained encoders? Random loudness augmentation teaches invariance. That is the wrong target for tasks where level is informative, and it does not address the representational trade-offs HEAR reports: wav2vec 2.0's pitch blind spot, and speech-centric models' weakness on non-linguistic events (La Quatra et al.). We will train log-Mel with aggressive gain augmentation at matched compute. Our prediction is that it matches us on standard accuracy but loses on the level-decoding probes.
Why not just use standard normalization layers inside the network? HARES reports that batch normalization degrades cross-domain transfer. Normalizer-free networks improved the average from 71.7 to 74.6. So we already build on the normalizer-free choice and do not compete against it. We also test instance normalization at the input, which, like PCEN, discards the statistics it removes.
Proposed Experiments
The first decisive check replicates LEAF-APCEN's four datasets (ESC-50, FMA-Small, CREMA-D, VoxCeleb1) in clean and complex conditions, plus GTZAN and IEMOCAP from Ada-FE. The requirement is modest: no loss relative to LEAF-APCEN. A front-end that exports state must not pay for it on invariance-favoring tasks such as speaker identification.
The second check is information-preservation probing. On the frozen representations of each front-end, we train linear probes to decode true presentation level, per-band SNR and the background-noise type. Normalized-only front-ends should fail at this by construction. We then apply the same probes to HEAR tasks where dynamics plausibly matter: NSynth pitch, Mridangam and FSD50K events. We also build a controlled variant of ESC-50 in which one class pair differs only in presentation level. This variant separates "exporting level helps" from incidental effects.
The third check, which targets the second claim, is a Fairhall-style switching test. Clean ESC-50 and Speech Commands clips are embedded in backgrounds whose variance switches with periods from 1 to 16 s. Accuracy is measured as a function of time since the switch and of the switching period. A single-timescale controller should show a recovery-time profile tied to its fixed horizon. The two-timescale bank should recover within about the fast timescale at all periods.
The fourth check probes the learned structure. We ask whether the per-band weights $\lambda_f$ separate systematically when the front-end is trained on speech, music and environmental data. We also ask whether the effective gain falls with stimulus contrast. That second question mirrors the negative contrast–gain correlation Mischler et al. found in human auditory cortex. That study was a descriptive encoding model of speech in noise, and here the effect becomes a test of an engineered system across domains.
Finally, in the pretraining setting, we report HEAR and HARES probe scores. Beyond per-task scores, we report the spread of relative performance across the speech, music and environment groups. A smaller spread at equal or better mean would be direct evidence that exporting the adaptation state reduces the trade-off between domains.
References
- Multiple Timescales of Adaptation in a Neural Code
- HEAR: Holistic Evaluation of Audio Representations
- Towards Learning Universal Audio Representations
- Should Audio Front-Ends be Adaptive? Comparing Learnable and Adaptive Front-Ends
- Adaptive Per-Channel Energy Normalization Front-end for Robust Audio Signal Processing
- Feedback Synapse to Cone and Light Adaptation
- Two Computational Regimes of a Single-Compartment Neuron Separated by a Planar Boundary in Conductance Space
- The Efficacy of Self-Supervised Speech Models for Audio Representations
- Benchmarking Representations for Speech, Music, and Acoustic Events
- Deep neural networks effectively model neural adaptation to changing background noise and suggest nonlinear noise filtering methods in auditory cortex