A speech quality score is one number for a whole clip. When a listener gives a phone call a 2.5 out of 5, that number does not say whether the problem was a 20-millisecond dropout six seconds in, steady background hiss, or muffled audio. The cause of the degradation lasts milliseconds; the judgment covers seconds. Neuroscience has a name for this gap, the temporal contiguity dilemma, and a standard answer to it. A brief event leaves a decaying local trace, called an eligibility trace, and a later global signal converts that trace into a lasting change. We propose to build the final stage of a speech quality model on this principle. A frame-by-frame encoder produces a separate impairment stream for each perceptual dimension (noisiness, coloration, discontinuity, loudness). A small set of biologically motivated dynamical units then combines those streams into the overall score. Each unit has a named time constant and a shared depleting resource, the same short-term depression mechanism that makes synapses respond less to repeated input.
The core claim is that this constrained integrator can match standard attention pooling on overall-score accuracy while buying two things attention pooling does not. The first is diagnosis without extra labels: because the score is an explicit sum of decayed, depleted frame contributions, each frame's share of the final score can be read off directly. We test whether this locates distortions even though the model was trained only on clip-level scores. The second is correct behavior when several impairments occur together. Here, human listeners are known to penalize the combination less than the sum of its parts. A shared depleting resource produces that sub-additivity from its mechanics. An additive readout cannot produce it.
What the two directions contribute
The diagnostic-quality literature both motivates the problem and sets its constraints. Diagnostic prediction of transmitted speech quality splits telephone quality into coloration, noisiness, discontinuity and loudness, and recombines them into an overall score. Its combination is a static function of per-dimension values and has no notion of when in the clip an impairment occurred. NISQA predicts the same four dimensions from the degraded signal alone, with no clean reference. It uses a per-frame convolutional network, self-attention across frames, and attention pooling that its authors motivate as a model of human recency effects. Its pooling weights are learned freely, so they are not guaranteed to decay with elapsed time, and nothing ties them to a specific dimension or to masking.
Multidimensional evaluation and predicting overall speech quality supplies the key negative result. A linear model from dimension scores to overall score fits single-impairment samples well (correlation 0.91, RMSE 0.39). On multiple-impairment samples its RMSE rises to 0.75, and it under-predicts quality by about 0.5 MOS. In other words, listeners find combined impairments less bad than the sum predicts. The authors patch this with polynomial interaction terms but offer no mechanism.
On the other side, SMARTMOS shows that on-device quality monitoring must be cheap. It solves the frame-rate versus judgment-rate mismatch with an accumulating encoder and a predictor run once per minute, which constrains us to a head with a tiny compute footprint. QualiSpeech annotates the onset and offset of artifacts. A LoRA-finetuned SALMONN-7B trained on those intervals localizes them with IoU of about 0.8. That gives us both the evaluation labels for localization and a fully supervised upper reference.
The plasticity literature supplies the mechanism. Neuromodulated STDP and three-factor learning rules and Sequential neuromodulation of Hebbian plasticity describe local traces set by brief events and later converted into lasting change by a delayed signal. Neural Correlates of Serial Dependence shows that short-term depression, a fast depleting resource, suppresses repeated input in a sensory layer. It also shows that short-term facilitation adds attraction toward past input in a higher layer. Continuous-Time Heteroassociative Memory at Biological Timescales shows that learning from delayed signals needs well-separated time constants. It reports that the plasticity window must be 10 to 50 times the stimulus duration.
We borrow the equations from these papers, not their biological claims. None of these models has been applied to real audio, and we do not assert that the auditory system implements our head.
Core mechanism
Let a framewise encoder emit nonnegative impairment intensities $d_k(t)$ for each dimension $k$ at frame time $t$. The dimensions draw on a shared resource $x(t)$ between 0 and 1, which recovers with time constant $\tau_r$ and is used up by impairment at rate $u$. The effective impairment of dimension $k$ is scaled by the remaining resource. That scaled impairment feeds a leaky memory $m(t)$ with time constant $\tau_m$, which carries the recency effect:
$$ \frac{dx}{dt} = \frac{1-x}{\tau_r} - u\,x\sum_k d_k(t), \qquad \frac{dm}{dt} = -\frac{m}{\tau_m} + x(t)\sum_k w_k\, d_k(t). $$
The overall score is $\hat{s} = 5 - g(m(T))$, where $w_k$ are learned dimension weights, $g$ is a monotone learned squashing function, and $T$ is the clip end. Per-dimension scores are read out from dimension-specific leaky memories that follow the same equations.
Two properties follow directly. First, when two impairments co-occur, the first depletes $x$, so the second contributes less. That is the sub-additivity Berger et al. measured, arising here from a single recoverable resource rather than a fitted cross term. Second, the contribution of frame $t$ to $m(T)$ is its effective impairment times the decay factor $e^{-(T-t)/\tau_m}$. That decaying factor is exactly the eligibility trace of a three-factor rule. Two consequences follow. The clip score decomposes exactly into per-frame, per-dimension attributions, with no post-hoc saliency method. And the gradient of the loss with respect to the readout weights at clip end equals the error times the stored trace. So the head can be updated online, on device, when a delayed rating arrives, without storing audio or backpropagating through the encoder.
The whole head has a handful of parameters: $\tau_r$, $\tau_m$, $u$, the weights $w_k$, and $g$. It runs in constant memory per frame.
Research Program
The encoder is NISQA's framewise convolutional network and self-attention, kept identical across all systems. Only the pooling head changes. We train on the NISQA training corpora with clip-level overall and dimension labels only, and never show the model QualiSpeech's intervals.
The analysis has three strands. The first checks that the constrained head matches free pooling on accuracy. The second measures how well its exact attributions localize distortions on QualiSpeech, against the attention weights and gradient saliency of unconstrained models. The third tests whether the shared resource predicts multi-impairment ratings better than either a linear readout or a readout with explicit interaction terms.
A final strand tests on-device adaptation. We freeze the encoder and update only the readout from delayed clip ratings with the trace rule. We compare this against offline retraining of the same head, on a DNS Challenge / P.835 split held out by noise-suppressor system.
Baselines
The naive reference is mean pooling of the framewise outputs, followed by the same squashing function. It has no recency and no interaction, so any benefit of temporal structure has to show up relative to it.
The strongest existing competitor is NISQA's own self-attention plus attention pooling, retrained by us on the identical data split, with the same encoder and the same hyperparameter search budget. It is the natural rival because it already models recency with learned weights. NISQA reports dimension RMSEs between 0.16 and 0.56. We will reproduce its numbers on the NISQA test sets (P501, FOR, NSC, LIVETALK) before comparing. On overall and dimension accuracy, we count as "matching" if RMSE stays within the confidence interval of the retrained NISQA. We count as "beating" it on localization if IoU against QualiSpeech intervals is higher than that obtained from NISQA's attention weights.
For localization, the fully supervised upper reference is finetuned SALMONN from QualiSpeech, which reports IoU of about 0.8. It uses interval labels and a 7B model, so we do not expect to beat it. The question is how much of that gap a clip-supervised head closes with orders of magnitude less compute.
For multi-impairment integration, the reference is Berger et al.'s linear readout (RMSE 0.39 on single impairments, 0.75 on multiple impairments). The stronger version of the same idea adds their pairwise interaction terms, fitted on the same training data. We beat it only if, on held-out multiple-impairment conditions, our RMSE is lower and our bias is closer to zero.
For noise suppression data, DNSMOS P.835 reports model-level PCC of 0.94 for SIG, 0.98 for BAK and 0.98 for overall, and clip-level PCC of 0.71 to 0.83. We report both levels. We do not claim gains there; we require no loss.
Three ablations isolate the key ingredients. The first gives each dimension its own resource instead of a shared one, which removes cross-dimension masking but keeps self-adaptation. The second removes depression entirely by setting $x \equiv 1$. The third replaces the head with an unconstrained GRU of matched width, which tests whether any gain comes from the specific structure rather than from recurrence as such.
Why not just…?
Why not just read NISQA's attention weights as the localization? This is the most direct threat, because the weights exist already and cost nothing. Their weakness is that the pooling weights come after self-attention has mixed information across frames. A high weight on frame $t$ therefore need not mean the distortion is at frame $t$, and the weights carry no dimension label. The comparison on QualiSpeech intervals settles it directly. If attention weights reach the same IoU as our attributions, the diagnosis claim fails, and the proposal reduces to the masking and online-adaptation results.
Why not just add Berger's interaction terms to a linear readout? On a small, fixed set of impairment types, pairwise products may fit masking as well as a shared resource does, and we say so plainly. The difference we test is temporal. Our model predicts that masking depends on how close in time the two impairments are, and on their order. A static product term cannot express this. If the interaction-term model matches ours on the Berger conditions and on the timing-controlled test below, we will drop the resource and keep the simpler readout.
Why not just train frame-level detectors on QualiSpeech's intervals? Where interval labels exist, direct supervision will localize better, as SALMONN's IoU of about 0.8 shows. But interval labels are rare and expensive, while clip scores exist at scale in the NISQA and DNS corpora. The relevant question is how far clip labels alone go. We also report a hybrid in which a small fraction of QualiSpeech intervals is added, to see whether the structured head needs fewer interval labels to approach the supervised reference.
Why not just use an unconstrained recurrent or transformer head? It may fit overall scores equally well. NISQA found BiLSTMs worse than attention, but a GRU pooling head on top of attention is untested. However, such a head offers no exact per-frame decomposition and no local update rule. The matched-width GRU ablation shows whether the structure costs accuracy. If it costs nothing on accuracy and wins on localization or masking, the structure is justified.
Decisive Checks
The first check is weakly supervised localization. On QualiSpeech's annotated artifacts, we compare the IoU of the top-attributed segments from our head against attention weights, gradient saliency and the GRU ablation. All are trained only on clip labels.
The second check is masking on held-out combinations. We train on single-impairment conditions only and test on Berger et al.'s 120-sample, 30-condition set, and on Möller databases A and B where their condition metadata marks combined impairments. We report RMSE and mean bias against the 0.75 and 0.5-MOS figures.
The third check is the one that most separates our mechanism from static alternatives: a small crowdsourced listening test. Using the same crowdsourced rating protocol as NISQA's data, we splice a fixed dropout into clean and noisy speech at controlled positions and with controlled gaps to a second impairment. The shared-resource head predicts that listeners forgive the second impairment more when it follows the first closely, with the effect fading on the scale of $\tau_r$. It also predicts a recency curve governed by $\tau_m$. Mean pooling and interaction terms predict neither the gap effect nor the position effect. Attention pooling can fit position only after the fact. If listeners show no gap dependence, the depression component is falsified.
The fourth check is on-device adaptation. We run delayed-rating updates of the readout with the trace rule on held-out noise suppressors and compare PCC against offline retraining of the same head, at equal numbers of labeled clips.
References
- Diagnostic prediction of transmitted speech quality: a new framework for signal-based and parametric models
- NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets
- Multidimensional evaluation and predicting overall speech quality
- SMARTMOS: Modeling Subjective Audio Quality Evaluation for Real-Time Applications
- QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions
- Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules
- Sequential neuromodulation of Hebbian plasticity offers mechanism for effective reward-based navigation
- Neural Correlates of Serial Dependence: Synaptic Short-term Plasticity Orchestrates Repulsion and Attraction
- Continuous-Time Heteroassociative Memory at Biological Timescales