When there are few labelled examples, emotion recognition from speech reliably detects how aroused a speaker is and reliably fails to detect whether the feeling is positive or negative. Arousal is the energy or intensity of the voice. Valence is the pleasant-versus-unpleasant dimension. The best cross-corpus method in our file reaches 74.3% weighted accuracy on arousal and only 58.6% on valence (Emo-DNA). Older neutral-anchored models could not separate happy from angry speech because both are loud and fast (Using neutral speech models for emotional speech analysis). We propose that valence information is not missing. It sits in a weak high-frequency "tail" of fast amplitude modulations in the voice, and standard smoothed features treat that region as negligible. Pulsar astronomy recently faced the same situation and resolved it. For years, gamma-ray spectra of pulsars were assumed to die off exponentially above a few GeV. Careful model comparison then showed a hard power-law tail beyond that point, produced by a second emission mechanism.
The proposal borrows the statistical method of that discovery, not its physics. We describe each utterance's modulation spectrum relative to the same speaker's neutral speech. We then ask, utterance by utterance and with a formal model comparison, whether that spectrum ends in an exponential cutoff or bends into a power-law tail. The few fitted parameters of that curve become a compact, gain-invariant description of affect that can be learned from very little labelled emotional data. The headline test is whether these parameters raise cross-corpus valence accuracy above 58.6%, and whether they beat simply feeding the raw high-frequency coefficients to a classifier.
Core Claim
Our claim has two parts. The first is an empirical hypothesis. Take second-order scattering coefficients, which are the outputs of a cascade of two wavelet filterings with a squared-magnitude step in between. Normalise them by the speaker's neutral speech and view them as a function of modulation frequency. For emotional speech, these spectra are better described by a smooth broken power law than by a cutoff. The break point and tail slope differ between same-arousal, opposite-valence emotions, such as happy versus angry.
The second part is a method claim. Summarising the tail with a handful of fitted parameters, instead of hundreds of raw coefficients, cuts variance enough to help when only a few labelled utterances exist. It also transfers better across corpora, because a slope on a log-log plot does not change when a recording channel multiplies the overall level by a constant.
What would be new: neutral-anchored emotion modelling has so far used likelihoods or linear difference vectors. Scattering features for speech have been tested only on word recognition. To our knowledge, no work has posed the existence of a high-frequency affective tail as a hypothesis to be decided by explicit model selection.
What Carries Over from the Pulsar Work, and What Does Not
The astrophysics papers motivate the method and supply its parametric family. Two papers fit pulsar spectra across instruments with different energy ranges. MAGIC detection of Geminga: an Inverse Compton tail? combined Fermi-LAT and MAGIC data and rejected a sub-exponential cutoff at 18σ in favour of a steep power law. A detailed study of the very high-energy Crab pulsar emission with the LST-1 preferred a smooth broken power law over a cutoff. It also found that the two pulse peaks have measurably different tail slopes, about 3.44 versus about 3.03.
Three things transfer. The first is the pair of competing functional forms. The second is the habit of letting a likelihood comparison, rather than a prior assumption, decide whether a tail exists. The third is the observation that two components with similar total energy can be told apart by their tail slopes. That last point maps directly onto happy versus angry, which have similar arousal.
The earlier MAGIC work on the Crab between 25 and 100 GeV (Aleksić et al., 2011) and up to 1.5 TeV (Ansoldi et al., 2015) shows a second relevant point. Tails are photon-starved, so they need long integrations to measure. The speech analogue is that a single short utterance has few high-frequency modulation events. The tail must therefore be fitted with pooled statistics across frequency bands and with uncertainty estimates. The physics itself, inverse Compton upscattering near the light cylinder, has no speech counterpart, and we claim none.
Let $S_2(\lambda, \omega)$ be the time-averaged second-order scattering energy of an utterance. Here $\lambda$ is the acoustic carrier band and $\omega$ the modulation frequency. Let $\bar S_2^{\mathrm{neu}}(\lambda,\omega)$ be the same quantity averaged over that speaker's neutral utterances. We define the neutral-normalised spectrum $R(\omega) = S_2(\lambda,\omega)/\bar S_2^{\mathrm{neu}}(\lambda,\omega)$ for each group of carrier bands. We fit it with either a cutoff model or a smooth broken power law:
$$R_{\mathrm{cut}}(\omega) = A\,\omega^{-\gamma}\exp\!\left(-(\omega/\omega_c)^{\beta}\right), \qquad R_{\mathrm{SBPL}}(\omega) = A\left(\frac{\omega}{\omega_b}\right)^{-\gamma_1}\left[1+\left(\frac{\omega}{\omega_b}\right)^{1/s}\right]^{-(\gamma_2-\gamma_1)s}$$
In these models, $A$ is an amplitude, $\gamma$, $\gamma_1$ and $\gamma_2$ are log-log slopes, $\omega_c$ is a cutoff frequency, $\beta$ sets how sharp the cutoff is, $\omega_b$ is the break frequency, and $s$ is the break smoothness. The affect descriptor is the vector of fitted $(\omega_b, \gamma_1, \gamma_2, s)$ per carrier-band group, together with their fit uncertainties. It also includes the per-utterance log-likelihood ratio between the two models, which works as a "tail strength" score. The amplitude $A$ absorbs any constant channel or level gain, so the slopes are unaffected by it.
Research Program
We first build a fixed, non-learned scattering front end. It follows the squared-magnitude cascade of Deep Scattering Power Spectrum Features for Robust Speech Recognition, which showed on AURORA4 that second-order coefficients recover attack transients that filterbanks smooth away. That paper studied word recognition in noise and never emotion. We reuse its feature computation and change what is done with the output.
Neutral anchoring follows Using neutral speech models for emotional speech analysis, which treated neutral speech as the reference state and scored emotion as deviation from it. We differ in what the deviation is. Busso and colleagues measured a likelihood under a neutral model. We measure the shape of the normalised modulation spectrum, specifically whether it has a tail. Neutral speech is the abundant resource here. It requires only unlabelled neutral recordings of each speaker, and every corpus in the cross-corpus benchmark includes a neutral class.
Fitting is done by weighted nonlinear least squares in log space. Bootstrap resampling of frames gives an uncertainty for each parameter, mirroring the error bars that make the pulsar slopes comparable. Utterances that are too short to constrain the tail are flagged rather than forced through the fit.
The classifier on top is deliberately small: logistic regression or a two-layer network on roughly 30 to 60 parameters. This keeps the comparison focused on the representation. The program has three stages. The first is a descriptive study asking where and for which emotions the tail model wins. The second is few-shot and cross-corpus recognition. The third tests whether the tail parameters add information beyond a strong pretrained speech model.
Baselines
The naive reference point is the arousal-only neutral-deviation approach of Busso et al.. It reports 65% four-way accuracy on its data against 25% chance, and it confuses happy with angry. We re-implement neutral-referenced log-Mel statistics with the same small classifier. This shows what neutral anchoring gives without any tail modelling. Beating it on happy-versus-angry pairs at matched shot counts is the minimum bar.
The strongest existing cross-corpus method is Emo-DNA. It uses contrastive decoupling to separate emotion prototypes from corpus prototypes, plus pseudo-label and kernel-distribution alignment. Across 30 transfer pairs among CASIA, EMO-DB, EMOVO, IEMOCAP, RAVDESS and SAVEE it reports an average of 58.6% weighted accuracy on valence and 74.3% on arousal. It is the right competitor because it attacks the same cross-corpus scarcity problem and publishes the valence number we target. We run on its 30 pairs with its binary valence and arousal protocol, and we quote its published figures. If we can reproduce the method, we also run it with our neutral-normalised input concatenated to its features, under the same tuning budget, to see whether the two combine. Beating it means average valence weighted accuracy clearly above 58.6% across the 30 pairs, with arousal not falling below 74.3%. "Clearly" means outside the spread of five seeds.
The strongest in-corpus learned baseline is wav2vec 2.0 with pseudo-label task-adaptive pretraining, from Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition. It reports 74.3% unweighted accuracy on IEMOCAP. We fine-tune it on the same few-shot subsets with the same hyperparameter search budget as our model. Beating it means higher valence accuracy at 5, 10 and 25 labelled utterances per class, where we expect its parameter count to hurt. We do not expect to beat it with full IEMOCAP training. At full data the meaningful result is whether tail parameters appended to its pooled embedding add accuracy.
The learnable superlet front end of Learnable Fractional Superlets reports 87.5% on IEMOCAP and 91.4% on EMO-DB under full supervision. It is a reference for supervised ceilings, not for scarcity, since it was never tested few-shot or cross-corpus. We run it in our few-shot settings only if its code is available. Otherwise we report it as a published ceiling and do not compare it directly.
The key ablation is "raw scattering". It uses the identical neutral-normalised second-order coefficients and the identical classifier, but without the curve fit, with dimensionality matched by PCA to the parameter count. No reported result was found on Lacuna for second-order scattering features on emotion recognition, so this baseline must be built here. If raw scattering matches the fitted parameters, the tail-model contribution is nil.
Why not just…?
Why not just fine-tune a large pretrained speech model? Prior work shows that plain fine-tuning struggles with the gap between read neutral speech and emotional speech. It needed a specialised pretraining stage to reach 74.3% on IEMOCAP (Chen & Rudnicky). Large audio-language models are also known to lean on words rather than tone (ParaS2S). Still, with enough data a fine-tuned model may absorb the tail information on its own. The complementarity test settles this. If appending tail parameters to the fine-tuned embedding gives no gain at any data size, the tail carries nothing the large model lacks, and the proposal's value shrinks to a low-data trick.
Why not just feed the raw second-order scattering coefficients to a classifier? This is the most serious threat, because the fit discards information. Our argument rests on two points. A fit with few parameters should have lower variance at 5 to 25 shots. Slopes should also survive channel changes better than raw energies, because $A$ absorbs gain. The dimension-matched raw-scattering ablation above tests both points directly. If raw scattering is as good in-corpus but worse cross-corpus, the claim narrows to transfer. If it is as good everywhere, we drop the parametric fit and report the neutral-normalised scattering representation as the finding.
Why not just apply per-speaker normalisation and standard domain alignment? Per-speaker z-scoring of log-Mel features is cheap and removes some speaker effects. However, Emo-DNA reports that adversarial alignment trails decoupled alignment by 13.3 points on arousal, and it falls short on valence even with decoupling. Speaker z-scored log-Mel features with an adversarial domain classifier will be run as an additional baseline. If they close the valence gap, the high-frequency tail is unnecessary.
Why not just use a linear neutral-plus-delta model? The additive scheme of Two-stage prosody prediction shifts pitch, duration and energy by a scaled emotion vector. Those are bulk, arousal-like quantities with no handle on high-frequency modulation shape. Linear transforms of neutral models such as Factored MLLR carry the same limitation. Neutral-referenced prosodic statistics are already contained in the naive baseline. If that baseline matches us on valence, this objection wins.
Proposed Experiments
The first experiment is the existence test, and it can kill the project early. On IEMOCAP and EMO-DB we fit both models to every emotional utterance. We report the distribution of log-likelihood ratios per emotion and per carrier-band group, with neutral utterances held out from the anchor as a null distribution. The hypothesis fails if the cutoff model is preferred about as often for emotional as for held-out neutral speech. It also fails if fitted tail slopes do not differ between happy and angry utterances matched on arousal. The arousal matching uses dimensional ratings, mirroring the slope difference between the two Crab pulse peaks.
The second experiment is few-shot recognition. On IEMOCAP and EMO-DB we use 5, 10, 25 and 100 labelled utterances per class, with speaker-independent splits and five seeds. We report unweighted accuracy for four-way categories and for binary valence, against the naive baseline, the raw-scattering ablation and wav2vec 2.0 with task-adaptive pretraining.
The third experiment is cross-corpus transfer on the 30-pair benchmark. Weighted accuracy and F1 on valence and arousal are compared to Emo-DNA's 58.6% and 74.3%. A second robustness test simulates telephony by band-limiting and adding codec noise to source corpora. It checks whether slopes stay stable when amplitudes shift. The real telephony set NSPL-CRISE, used for the superlet front end (Learnable Fractional Superlets), is private, so we do not plan on it.
The decisive outcome is the combination. A tail must exist in the first experiment, and the fitted parameters must beat raw scattering cross-corpus while lifting valence above 58.6% at matched compute. All three conditions together would show that the pulsar-style model comparison found real structure in emotional speech, rather than merely relabelling known features.
References
- Emo-DNA: Emotion Decoupling and Alignment Learning for Cross-Corpus Speech Emotion Recognition
- Using neutral speech models for emotional speech analysis
- MAGIC detection of Geminga: an Inverse Compton tail?
- A detailed study of the very high-energy Crab pulsar emission with the LST-1
- OBSERVATIONS OF THE CRAB PULSAR BETWEEN 25 AND 100 GeV WITH THE MAGIC I TELESCOPE
- Teraelectronvolt pulsed emission from the Crab Pulsar detected by MAGIC
- Deep Scattering Power Spectrum Features for Robust Speech Recognition
- Exploring Wav2vec 2.0 Fine Tuning for Improved Speech Emotion Recognition
- Learnable Fractional Superlets with a Spectro-Temporal Emotion Encoder for Speech Emotion Recognition
- ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction
- Two-stage prosody prediction for emotional text-to-speech synthesis
- Factored MLLR Adaptation Algorithm for HMM-based Expressive TTS