Speech emotion models learn well from acted recordings, where actors exaggerate anger or fear so each emotion sounds distinct. They fail on real conversations, where those same emotions are muted, brief, and rare. In a podcast corpus, contempt or fear may make up a small fraction of speaking turns, buried under neutral speech. We propose to fix the rare classes by generating synthetic training points for them. The generated points are not built from scarce natural examples alone. Each one blends abundant acted examples with the few natural ones, inside the embedding space of a frozen speech foundation model (a large network pre-trained on unlabeled audio whose internal vectors summarize each utterance). Before blending, the acted examples are pulled toward the natural neutral region, by an amount estimated from data, to mimic how real expression is attenuated.
The same structure appears in medical imaging. A clean, expensive device supplies plenty of labeled data. A cheap target device supplies few images, and its class proportions are shifted. The bet is that one geometric operation can handle both class imbalance and domain shift at once in these settings: interpolation across a mixed-domain simplex (the triangle, tetrahedron, or higher analogue spanned by a few nearby points), anchored on target examples. Today these two problems are handled separately, by reweighting the loss and by adapting the network.
Core Claim
Simplicial over-sampling, applied to frozen foundation-model embeddings, will raise minority-class F1 on natural speech more than three alternatives: loss reweighting, single-domain over-sampling, and naive pooling of acted data. Three conditions apply. Each synthetic point must lie in a simplex that contains at least one natural (target) vertex. The acted vertices must be shrunk toward the natural neutral centroid before sampling. The gain should appear only for classes that the acted corpus actually covers. That last prediction is what makes the claim testable and not just another augmentation trick.
Where This Sits
Three lines of work motivate the proposal, and each stops short of it.
Simplicial SMOTE replaces the line-segment interpolation of SMOTE with sampling inside cliques of a minority k-nearest-neighbor graph, using Dirichlet-distributed weights. It reports +4.5% (k-NN) and +5% (gradient boosting) average F1 over SMOTE on 21 tabular datasets. It is single-domain and tabular. We reuse its sampling mechanism but change which points may form a simplex.
The winning Odyssey 2024 system, 1st Place Solution to Odyssey Emotion Recognition Challenge Task1, handled imbalance on spontaneous MSP-Podcast speech only through focal loss with inverse-frequency weights and a seven-model ensemble. Its authors note that class weighting tends to overfit minority classes. This supplies our strongest speech competitor.
Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation transfers from acted to natural speech in two stages using small adapter modules. It does not address the class skew of the natural target. We share its premise that acted data is a useful resource, but we use acted data as raw material for synthetic minority points rather than as a pre-training stage.
On the imaging side, Imbalance-Aware Domain Adaptation in Embryo Development Assessment (IADA) tackles joint hardware and label shift. Blastocyst rates move from 28.9% on the source device to 81.2% on a portable microscope. IADA relies on class-specific attention, balanced adversarial alignment, and per-class thresholds, with no synthetic data, and it needs source and target data in the same place. We test whether mixed-domain interpolation of frozen features can compete with it using far less machinery.
Two papers constrain the design. Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift shows that shifted models make confident errors that uncertainty estimates fail to flag. Synthetic points placed near decision boundaries could make this worse, so we measure referral quality and do not just assume it holds. SMOTE and Mirrors shows that interpolating between real records can leak them. Mixing private target vertices with public acted vertices may dilute that leakage, which we check as a secondary question.
Research Program
All sampling happens on utterance-level embeddings from a frozen encoder. For speech this is WavLM with attentive statistics pooling, as in the Odyssey 2024 baseline. For embryos it is an ImageNet-pretrained Xception, matching IADA's backbone. Each domain is standardized separately.
For a minority class $c$, let $\mu_N$ be the centroid of natural neutral (or target-majority) embeddings. Let $m_c^{nat}$ and $m_c^{act}$ be the natural and acted centroids of class $c$. Each acted embedding $a$ of class $c$ is attenuated to
$$a' = \mu_N + \alpha_c\,(a - \mu_N), \qquad \alpha_c = \frac{\lVert m_c^{nat} - \mu_N\rVert}{\lVert m_c^{act} - \mu_N\rVert},$$
so that, on average, acted points sit as far from neutral as real ones do. We then build a k-nearest-neighbor graph over the natural class-$c$ points together with the attenuated acted points. We keep only cliques of size $p+1$ that contain at least one natural vertex, and sample
$$x = \sum_{i=0}^{p} \lambda_i v_i, \qquad \lambda \sim \mathrm{Dirichlet}(\beta_{nat}, \beta_{act}, \ldots),$$
where the $v_i$ are the clique's vertices, and the concentration $\beta_{nat} \ge \beta_{act}$ biases samples toward natural vertices. A light classifier head is trained on real plus synthetic embeddings until classes are balanced.
The method has four hyperparameters: $k$, $p$, $\beta_{nat}/\beta_{act}$, and whether attenuation is on. They are tuned on the official development split under the same budget as every baseline. A second phase asks whether gains survive when the encoder is first adapted with adapters on natural data, following Lashkarashvili et al. In that phase embeddings are re-extracted after adaptation and the sampling is rerun. This directly probes the open question of whether small adapters collapse minority classes under skew.
Referral is evaluated with a multi-class extension of split referral. Samples are ranked and referred separately within each predicted class. This lets us check whether boundary-pushing synthetic data produces the confident-error failure documented by Band et al.
Baselines
The naive reference is the Odyssey 2024 baseline: fine-tuned WavLM with attentive pooling and standard cross-entropy. It reports F1-macro 0.327 and F1-micro 0.311 on MSP-Podcast v1.11 Task 1 (Odyssey 2024). We also run its frozen-encoder variant with an unweighted head, since that is the exact backbone our method uses.
The strongest existing method is the Sheffield-MINI recipe (1st Place Solution). It reports macro-F1 35.69% and accuracy 37.32% for the ensemble, and 34.2% macro-F1 for its single best model. Our method is acoustic-only and single-model. The fair comparison is therefore their single-model, audio-only, focal-plus-prior-weighted configuration, re-run by us on the same frozen WavLM features with an identical head, epochs, and tuning trials. Beating it means a macro-F1 gain whose 95% confidence interval excludes zero across five seeds, carried mainly by contempt, disgust, and fear. We will also add our synthetic points to their full loss. The method is useful only if it adds to focal loss, not just substitutes for it.
Three further baselines isolate the key ingredient. Within-domain Simplicial SMOTE uses natural points only, which tests whether acted vertices matter. Pooled acted data with inverse-frequency weights adds the same acted embeddings as real training points, with no interpolation or attenuation, which tests whether geometry matters. Our method with $\alpha_c = 1$ tests whether attenuation matters. Classic SMOTE and feature-space mixup over the mixed pool complete the comparison.
For embryos, the baseline is IADA on ED4 to ED3, ED2, and ED1. The paper reports relative gains: +25.19% accuracy over MD-Net on ED4 to ED3, AUC up to +12.56% on ED4 to ED1, and recall 1.0000 on ED4 to ED2 with Xception. No absolute accuracy or AUC for IADA was found on Lacuna, so we reimplement it with the same backbone and splits and compare against our reimplementation. The target sets are small (69 to 296 images), so all embryo results use repeated stratified splits.
Why not just…?
Why not just pool the acted data with class weights? This is the most serious threat. Acted minority examples are real data, and adding them may account for most of any gain. The pooled-acted baseline is run at identical data volume, and our method has to beat it. If attenuation-free interpolation matches pooling and attenuated interpolation does not beat it, the geometric claim fails and we will report that. The mechanism we expect is that raw acted points sit too far from neutral and teach an exaggerated boundary. That expectation is exactly what the acted-to-natural literature describes, but it is untested in embedding space.
Why not just use focal loss with prior weights? Reweighting only rescales gradients from the few natural minority points the model already has. It cannot add coverage of regions where no such points exist. Chen et al. themselves report that weighting overfits minority classes, and their single model peaks at 34.2% macro-F1. The additive experiment, focal loss with and without our synthetic points, settles whether the two are complementary.
Why not just run Simplicial SMOTE on the natural data alone? With a handful of contempt or fear turns, the natural-only graph has few cliques, and those simplices span tiny regions. Interpolation cannot extend beyond the observed points. The within-domain ablation measures this directly. We also plot gain against the number of natural minority examples, expecting the cross-domain advantage to shrink as natural data grows.
Why not just recalibrate outputs post hoc? Split Platt Scaling and test-time prior adjustment (Mitigating Label Shift in Tabular In-Context Learning) fix label shift cheaply, but they do not change the representation. They cannot separate classes that overlap in feature space. We apply prior adjustment on top of every method. If it closes the gap to our method, the representation-side contribution is not needed.
Proposed Experiments
The primary testbed is MSP-Podcast v1.11 Task 1 with IEMOCAP as the acted source. The decisive check uses an awkward fact: IEMOCAP's labels cover only part of MSP-Podcast's eight classes. Our method can only help classes that have acted vertices. It should improve per-class F1 for overlapping classes and leave non-overlapping classes such as contempt unchanged, apart from indirect boundary effects. A uniform improvement across all classes would suggest the gain comes from generic regularization, not borrowed geometry.
The second check is attenuation. Natural turns are binned by annotated arousal from the dimensional labels. The $\alpha_c$-attenuated variant should gain most on low-arousal minority turns, the regime where the literature reports systematic failure.
The third check is the embryo cross-hardware benchmark. ED4 plays the role of the acted, prototypical source, and each low-cost device is the natural target. We ask whether frozen-feature mixed-domain sampling reaches IADA's performance without adversarial training or co-located source data at training time; only embeddings need to be shared.
The fourth check is safety. Risk-coverage curves under per-class split referral must not degrade relative to the baseline. A method that raises macro-F1 by producing confident errors on rare classes would be unacceptable for clinical or mental-health monitoring.
A final secondary analysis runs the nearest-vertex reconstruction attack from SMOTE and Mirrors. It compares how much target-record leakage remains when synthetic points use mixed-domain cliques rather than natural-only cliques.
References
- Simplicial SMOTE: Oversampling Solution to the Imbalanced Learning Problem
- SMOTE: Synthetic Minority Over-sampling Technique
- 1st Place Solution to Odyssey Emotion Recognition Challenge Task1: Tackling Class Imbalance Problem
- Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation
- Addressing Domain Shift via Imbalance-Aware Domain Adaptation in Embryo Development Assessment
- Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift
- SMOTE and Mirrors: Exposing Privacy Leakage from Synthetic Minority Oversampling
- Odyssey 2024 - Speech Emotion Recognition Challenge: Dataset, Baseline Framework, and Results
- Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment