Focal Loss Without Distorting the Annotators: Decoupled Training for Naturalistic Speech Emotion

In real conversations most speech sounds neutral, and when listeners hear emotion they often disagree about which one. The best current emotion recognizers for podcast speech therefore train on the full spread of annotator votes, a soft target, instead of a single hard label. To handle the rarity of emotions like fear or contempt, many of these systems also use focal loss, a variant of cross-entropy that down-weights examples the model already gets right. Focal loss carries a side effect that matters once targets are soft. Its optimum is not the target distribution but a warped version of it that flattens confident probabilities and inflates middling ones, as shown formally in Functional Properties of the Focal-Entropy. Applied to soft annotator votes, focal loss rewrites what the annotators said. It does more than reweight hard examples.

We propose to separate the two things focal loss does. The first is emphasizing hard utterances. The second is reshaping each utterance's target, which we treat as unwanted. We do this inside a two-stage, or decoupled, pipeline: first learn the audio and text encoders with a plain soft-label loss, then freeze them and train only a small classification head. The head is trained with a focal-style weight that is detached from the gradient, so it rescales how much each utterance counts but leaves its optimum equal to the annotator distribution. The class imbalance is then handled by a separate, explicit per-class bias at decision time. Our claim is that this system keeps the Macro-F1 gains credited to focal loss on MSP-Podcast. It should also produce predicted distributions that match human vote spreads and are better calibrated, which today's focal-trained systems neither achieve nor measure.

What is already done, and what is not

The three ingredients have been paired before, so the plain combination is not new. Lessons Learnt: Revisit Key Training Strategies for Effective Speech Emotion Recognition in the Wild fine-tunes WavLM and RoBERTa separately, freezes them, and trains a concatenation head with focal loss on the Interspeech 2025 MSP-Podcast challenge. It reports focal loss at F1-Micro 0.592 against 0.487 for class-weighted cross-entropy and a Macro-F1 of 0.355 with Mish activations. From Pretraining to Performance freezes the lower WavLM-Large layers, fine-tunes the upper 15 with focal loss and multi-layer pooling, and reaches Macro-F1 0.3661 on test-3. 1st Place Solution to Odyssey Emotion Recognition Challenge Task1 trains end-to-end with class-weighted focal loss and a seven-model vote, reaching Macro-F1 35.69% on MSP-Podcast v1.11. All three use hard labels with focal loss. None measures calibration or agreement with annotator distributions.

A separate line of work shows that soft targets matter more than any imbalance trick. MEDUSA trains on vote distributions with a Kullback–Leibler (KL) divergence loss and reports an 8.23% drop when reverting to one-hot targets. SAILER also trains on vote distributions, keeps the roughly 19% of utterances without majority agreement, and reaches Macro-F1 0.411 with a single model. Neither paper uses focal loss, and neither freezes its encoders before training the head. The gap our proposal targets lies between these two lines. Soft labels and focal loss have not been combined in a way that respects what the soft labels mean.

The decoupling literature supplies both the method and a constraint. Decoupling Representation and Classifier for Long-Tailed Recognition found that the encoder learns best under ordinary, unreweighted training and that the tail-class deficit lives mostly in the classifier. This is why we keep focal weighting out of Stage 1. Improving Calibration for Long-Tailed Recognition showed that decoupled vision models become badly overconfident, and that label-aware smoothing plus re-estimated normalization statistics in Stage 2 cut expected calibration error (ECE) on CIFAR-100-LT from 27.5% to 4.83%. This warns us that decoupling alone will not give calibration. Both papers assume hard labels.

Core Claim

Let $q(x)$ be the annotator vote distribution for utterance $x$ over the eight classes and $p_\theta(x)$ the model's predicted distribution. The natural soft extension of focal loss is

$$L_{\text{soft-FL}} = -\sum_c q_c (1-p_c)^{\gamma} \log p_c ,$$

where $\gamma \ge 0$ is the focusing strength. Its per-utterance minimizer is a deformation of $q$ rather than $q$ itself, following Functional Properties of the Focal-Entropy. That paper also identifies an over-suppression regime in which very small probabilities shrink further. We instead use

$$L_{\text{det}} = \mathrm{sg}\!\left[\big(\tfrac{1}{2}\lVert q - p_\theta\rVert_1\big)^{\gamma}\right]\,\mathrm{KL}\big(q \,\Vert\, p_\theta\big),$$

where $\mathrm{sg}$ stops gradients and $\tfrac{1}{2}\lVert q-p\rVert_1$ is the total-variation distance between vote and prediction, which serves as the soft analogue of $1-p_t$. The weight is a constant for each utterance during each step, so every utterance's optimum is still $p=q$. The weight changes only how much the utterance counts. Imbalance is then corrected at decision time by adding $-\tau \log \pi_c$ to the logits, where $\pi_c$ is the training-set class prior and $\tau$ is tuned on development data. The head therefore gives two outputs. The unadjusted distribution estimates what listeners would say. The prior-adjusted argmax is the decision tuned for Macro-F1.

The testable prediction has two parts. On Macro-F1, the proposed system should at least match soft focal loss and focal-trained hard-label pipelines. On distribution agreement and calibration, it should clearly beat them.

Research Program

Stage 1 fine-tunes each encoder alone with plain soft-label KL and natural, unreweighted sampling. The speech encoders are WavLM-Large and Whisper-Large-V3, pooled with the multi-layer attentive statistics pooling from From Pretraining to Performance. The text encoder is RoBERTa-Large on Whisper transcripts. Using both speech encoders lets us revisit, under frozen features, the disagreement between SAILER and the SSL benchmark about which one is better. Stage 2 freezes all encoders and trains a lightweight fusion head with $L_{\text{det}}$. The head uses pointwise convolutions over concatenated pooled features, similar to SAILER. During Stage 2 we re-estimate normalization statistics, following Improving Calibration for Long-Tailed Recognition.

Two analyses accompany the main system. The first is a cheap, training-free measurement. For every development utterance we compute the exact minimizer of $L_{\text{soft-FL}}$ given $q$ and report how far it lies from $q$, across a range of $\gamma$ and separately for tail classes. If this deformation is negligible on real MSP-Podcast votes, the motivation weakens and we will say so. The second is a placement study that separates stage effects: focal-style weighting in Stage 1 only, in Stage 2 only, or in both, with a linear probe on frozen Stage-1 features measuring representation quality directly.

Baselines

The naive reference is a single end-to-end model trained with hard-label cross-entropy on the same encoders. This shows how much any imbalance or soft-label handling buys.

The strongest single-model competitor is SAILER-style end-to-end soft-KL training, which reports Macro-F1 0.411 (SAILER). MEDUSA's 14-model ensemble reports 0.4316 (MEDUSA). We cite the ensemble as a ceiling rather than a like-for-like competitor. We will re-implement SAILER's recipe without its audio mixing and annotation dropout, then add those two tricks back as a separate row, since they moved Macro-F1 from 0.401 to 0.406 in the original. The re-implementation uses our encoders, data splits, and tuning budget. The primary comparison is on our own runs; the reported numbers are only a sanity check that the re-implementation is faithful.

The obvious combination is end-to-end soft focal loss, $L_{\text{soft-FL}}$, with $\gamma$ swept across a wider range than the 2.0–2.5 tested by Chen et al.. The hard-label decoupled focal pipeline of Lessons Learnt is re-run with its reported Macro-F1 of 0.355 as the reference.

Two ablations isolate the key ingredient. The first replaces $L_{\text{det}}$ with plain KL in Stage 2 and keeps the same logit adjustment, which removes the detached weighting. The second uses $L_{\text{soft-FL}}$ in Stage 2, which keeps the focal emphasis but allows target distortion. All systems get the same seeds (five per configuration), the same GPU budget, and the same number of dev-set tuning trials for $\gamma$, $\tau$, and learning rate.

Beating a baseline means two things. Macro-F1 must be within seed noise or higher. Distribution agreement must be better beyond seed noise, measured as KL and Jensen–Shannon divergence to $q$ on held-out data, with the no-majority utterances reported separately. Calibration must also be better, measured as ECE and reliability diagrams for the unadjusted outputs.

Why not just…?

Why not just train with soft KL end-to-end and apply a post-hoc logit adjustment? This is the real threat. It uses one stage, no focal term, and one tuned bias, and it could plausibly capture most of the Macro-F1 gain. The first ablation, plain KL in Stage 2 plus the same adjustment, and an end-to-end version of the same are both in the experiments. If either matches the full system on both Macro-F1 and distribution agreement, we will conclude that the focal emphasis adds nothing once targets are soft and the prior is corrected explicitly. That would be a useful negative result for a community that treats focal loss as a default.

Why not just use soft focal loss directly? It is the natural combination, but by the focal-entropy analysis its optimum is not the vote distribution, and under heavy imbalance it can push rare-class probabilities down (Functional Properties of the Focal-Entropy). The training-free deformation measurement and the second ablation show directly whether this distortion occurs on MSP-Podcast and whether it costs agreement or calibration.

Why not just apply temperature scaling after a focal-trained model? A single temperature can fix overall confidence but cannot restore the shape of a distribution that focal loss has deformed class by class. We will run temperature scaling on every baseline so that any remaining gap in distribution agreement comes from something one scalar cannot fix.

Why not just ensemble? Ensembles help, but MEDUSA's ablations show they did not recover the losses from hard labels or weaker fusion. SAILER matched MEDUSA's result with three models. We compare single models and report ensembles separately.

Proposed Experiments

All experiments use the Interspeech 2025 MSP-Podcast challenge partitions with eight classes and full vote distributions, and are reported on the official test split where labels are available and on a held-out portion of the released data otherwise. As a secondary check, we train on MSP-Podcast v1.11 against Chen et al.'s 35.69%. IEMOCAP serves only as an out-of-domain transfer probe, because its acted, high-agreement labels would not exercise the soft-target question.

Three results are decisive. First, the deformation measurement must show non-trivial shifts in tail-class probability under the $\gamma$ values used in practice. Second, the placement study should reproduce, for speech, the finding of Decoupling Representation and Classifier for Long-Tailed Recognition that plainly trained Stage-1 features probe as well as or better than focal-trained ones. If focal-trained features probe better, the decoupled design is wrong for this domain, and we would move the weighting into Stage 1. Third, the full system must match the strongest re-run baseline on Macro-F1 while lowering divergence to annotator votes and ECE, with the largest gains expected on no-majority utterances and tail classes. If the logit-adjusted plain-KL ablation ties on all three measures, the headline becomes that focal loss is unnecessary for soft-label naturalistic emotion recognition, and we will report it that way.