Speech emotion recognizers that score well inside one dataset lose much of that accuracy on a different dataset. The usual fix is to align the two feature distributions with optimal transport, which moves probability mass from the labeled source corpus onto the unlabeled target corpus along the cheapest paths. That fix fails in a predictable place: the rare emotions. Classes like disgust or fear have few examples. They live in the 768- to 1024-dimensional feature spaces of modern speech models, where transport estimated from few samples is close to noise. Rare classes are also where the target corpus most often has different class proportions from the source. Oversampling the rare classes does not help either, because existing oversamplers create synthetic points that carry the source corpus's microphones, language, and acting style, not the target's.
We propose a single mechanism that addresses both problems. We fit a low-rank transport plan between source and target, meaning one that routes all mass through a small set of "hubs" (cluster centres). Hubs are allotted per emotion class by a fixed budget rather than in proportion to class frequency, so rare classes get as many hubs as common ones. The plan's hub-to-hub transfers then serve two purposes. They estimate how class proportions shift in the target. They also act as a low-variance displacement field: we push labeled rare-class source utterances through it to create synthetic training examples that sit on the target corpus's acoustic manifold. The claim is that dimension-robust transport and cross-domain oversampling are the same estimation problem once hub capacity is assigned by class rather than by mass, and that solving them jointly helps most exactly where current methods help least.
Core Claim
The claim has two parts. In cross-corpus speech emotion recognition with high-dimensional self-supervised features, the gap left by current transport-based adaptation is concentrated in minority emotion classes. A factored coupling with class-budgeted hubs, used both to align and to generate, closes that gap more than any of three alternatives: reweighting, source-domain oversampling, or dimensionality reduction followed by ordinary entropic transport.
The prediction is specific and falsifiable. Improvements over the strongest transport baseline should appear mainly in per-class recall of the rare emotions. They should grow with the imbalance ratio and with feature dimension. They should vanish when hubs are allotted in proportion to mass, which is the ablation that removes the key ingredient.
Where This Sits in Prior Work
Three groups of papers motivate the gap.
SOT is the closest speech work. It aligns source and target emotion features with margin-regularized transport plus a self-supervised clustering term, reaching 70.65% unweighted accuracy on IEMOCAP→EmoDB. It uses a full, unfactored coupling and assumes no class imbalance. We keep its setting but change the coupling's structure and add generation.
EmoBox benchmarks ten self-supervised speech models across 32 corpora. It shows that larger models do not close the cross-corpus gap, so we cannot simply rely on frozen Whisper or WavLM features.
Acted vs. Improvised and C²GAN show that transfer is worst toward spontaneous or induced speech such as eNTERFACE. In those corpora subtle emotions are both acoustically compressed and scarce.
The method comes from the low-rank transport literature. FactoredOT restricts the coupling to pass through k latent hubs. It reports an estimation rate independent of ambient dimension and 14.10% versus 17.47% error against full transport on single-cell batch correction, a setting with more than 20,000 dimensions. FRLC generalizes this with a non-diagonal inner coupling between source and target hubs, which allows mass to split. Both are unsupervised and allocate hubs where the mass is. Under imbalance that starves rare modes, and we change exactly that. Entropic regularization (Genevay et al., Mena et al.) also improves the rate in sample size. However, the relevant sample size for a rare class is its own tiny count, which is why per-class hub budgets matter more than the choice of regularizer.
Other work constrains the design. MARS shows that blind alignment under label shift mismatches classes. It handles this by clustering the target, matching clusters to source classes, and reweighting. It never generates samples, and it depends on a geometric-proximity assumption that acoustic shifts may violate. JUMBOT relaxes marginals so unmatched mass can stay put. That protects against negative transfer but also lets rare target classes go unaligned, and the authors report instability when tuning the regularization strength. We adopt JUMBOT's relaxed marginals on the inner hub coupling only, where there are few variables to tune.
OTOS supplies the generative idea. It synthesizes minority points as transport barycenters, which beat SMOTE and random oversampling in G-mean on LIBSVM and fundus-image tasks. It works within a single domain and uses a source-trained SVM to find hard boundary points, which does not transfer across corpora. Our synthetic points are barycenters on the target side of the plan instead.
Research Program
The model works on frozen or lightly fine-tuned utterance embeddings, for example mean-pooled WavLM-large or Whisper-large features. As a comparison point with the older literature, it also runs on the IS09/IS10 statistical feature sets.
Let $Q$ be the soft assignment of $n$ source utterances to source hubs and $R$ the soft assignment of $m$ target utterances to target hubs. Let $T$ be the inner coupling between the two hub sets, and let $g_Q$ and $g_R$ be the hub masses. The coupling is
$$P = Q\,\mathrm{diag}(1/g_Q)\,T\,\mathrm{diag}(1/g_R)\,R^\top,$$
following the FRLC factorization.
The departure is in how source hubs are defined. Each source class $c$ receives a fixed budget $k_c$ of hubs, by default equal across classes. Source utterance $i$ may be assigned only to hubs of its own label. Hub masses are not tied to class frequency. Target hubs are shared and unlabeled. The inner coupling $T$ has hard marginals on the source side and KL-relaxed marginals on the target side, so the column sums of $T$, aggregated by source class, give an estimate of the target's label proportions. We solve this with FRLC's mirror-descent updates inside minibatch training of the feature adapter and classifier. The classifier is trained on source labels, on target pseudo-labels weighted by transport confidence, and on synthetic examples.
Generation works as follows. A source utterance $x$ of a rare class $c$, assigned to source hub $h$ with centre $a_h$, is moved to
$$\tilde{x} = x + \sum_{j} \frac{T_{hj}}{\sum_{j'} T_{hj'}}\,(b_j - a_h),$$
where $b_j$ is the centre of target hub $j$. The synthetic point keeps its within-hub variation, such as speaker and phrasing, but its hub-level position is relocated to where the target corpus puts that kind of emotion. Because the displacement is estimated from hub centres pooled over many points, it has much lower variance than the per-point displacements a full coupling would give from the same few rare-class examples. We add OTOS-style interpolation between transported points within the same target hub. We generate up to a capped ratio rather than forcing exact balance, following the overfitting evidence in Fu et al..
The analysis component is a controlled study in which we vary feature dimension, imbalance ratio, and hub budget. On synthetic Gaussian mixtures with a known transport map and planted rare modes, we measure the per-class error of the estimated displacement. This checks the mechanism directly, before any speech results.
Baselines
The naive reference is a classifier on source embeddings with no adaptation, trained with class-weighted cross-entropy. Its reported counterpart is SOT's no-adaptation baseline, at 58.69% unweighted accuracy on IEMOCAP→EmoDB. We also run plain source-domain random oversampling. Both use the same backbone and head as every other system.
The strongest existing competitor on the speech side is SOT, at 70.65% unweighted accuracy on IEMOCAP→EmoDB and 60.39% on IEMOCAP→CREMA-D in SOT. On the six-pair EmoDB/CASIA/eNTERFACE protocol, the strongest reported number is 45.41% average unweighted average recall with IS10 features in C²GAN, against 44.79% for CAEADA. SOT matters most because it is the transport method our change targets. We reimplement it on our backbone, report both its published figure and our reproduction, and add class weighting to it so that it is not handicapped on imbalance.
JUMBOT and MARS are the strongest label-shift-aware transport methods. Neither has been run on speech, and no reported result found on Lacuna for either on any speech emotion benchmark. We run both on identical embeddings.
Fairness is enforced in three ways. First, every method gets the same backbone, the same training epochs, and the same hyperparameter search budget, measured in trials. Second, selection uses only source validation data and an unsupervised target criterion (reverse validation), never target labels. Third, results are reported as means over seeds.
Two ablations isolate the key ingredient. The first allots hubs in proportion to mass, which is plain FRLC. The second uses one hub per class, which collapses the method to class-centroid shifting. A further pair separates the parts of the method: alignment alone with no synthesis, and synthesis by source-domain SMOTE or OTOS feeding into the same aligner.
Beating SOT means two things. Overall unweighted accuracy or recall must be at least equal. Minority-class recall must improve by more than seed variance, under both natural and induced imbalance.
Why not just…?
Why not just add class-weighted loss or oversampling to SOT or JUMBOT? This is the most serious threat. Reweighting fixes the classifier's priors but not the alignment: the rare class's transport is still estimated from few points in high dimension. If the full coupling is accurate enough, reweighting will match us. The "SOT + class weighting" and "JUMBOT + class weighting" baselines test this directly. If they equal our method on minority recall, the generative part of the claim is wrong, and we will report it that way.
Why not just reduce dimension with PCA and run ordinary entropic transport? Projection also attacks the curse of dimensionality. However, PCA directions are dominated by majority-class and corpus variance. That is the same mass-proportional bias our hub budget removes, so we expect rare-emotion directions to be discarded. We test PCA plus Sinkhorn transport at several target dimensions. If it matches us, the class budget is unnecessary and the low-rank coupling adds nothing.
Why not just shift each class by the difference of its source centroid and its pseudo-labelled target centroid? This is our method with one hub per class, so it is an ablation rather than an external alternative. It fails if rare classes are multimodal in the target, for example across speakers or arousal levels, which Bellinger et al. suggest is common. If multiple hubs do not beat one, we will drop the claim about mass splitting.
Why not just use a stronger foundation model? EmoBox shows that cross-corpus collapse persists across Whisper, WavLM, and HuBERT. We nonetheless run every method on the strongest backbone available, so any gain is measured on top of scale.
Proposed Experiments
The first set of experiments uses the six-pair EmoDB/CASIA/eNTERFACE protocol, which allows comparison with C²GAN and CAEADA. The second set uses IEMOCAP→EmoDB, IEMOCAP→CREMA-D, and the MUSAN 0 dB noise condition, which allows comparison with SOT (50.42% unweighted accuracy reported in the noise case). Each is run in two regimes: the natural class distributions, and induced imbalance in which source rare classes are subsampled to 10:1 and 20:1 and the target receives a different, shifted label distribution. Each regime is run with IS10 features and with WavLM/Whisper embeddings, which also isolates the effect of dimension. Metrics are unweighted average recall, per-class recall reported separately for the two rarest classes, G-mean, and the error of the estimated target label proportions.
Three decisive checks follow. First, gains over class-weighted SOT should concentrate in rare-class recall and grow from IS10 features to 1024-dimensional embeddings. Second, the mass-proportional hub ablation should lose most of that gain. Third, synthetic points produced by the displacement formula should be classified as target-domain, not source-domain, by a corpus classifier trained to tell the two apart, while source-domain SMOTE points should not. If the first check fails, the method is not needed. If the second fails, the class budget is not the cause. If the third fails, the oversampling is not cross-domain at all.
References
- SOT: Self-supervised Learning-Assisted Optimal Transport for Unsupervised Adaptive Speech Emotion Recognition
- EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark
- Acted vs. Improvised: Domain Adaptation for Elicitation Approaches in Audio-Visual Emotion Recognition
- Boosting Cross-Corpus Speech Emotion Recognition using CycleGAN with Contrastive Learning
- Statistical Optimal Transport via Factored Couplings
- Low-Rank Optimal Transport through Factor Relaxation with Latent Coupling
- Sample Complexity of Sinkhorn Divergences
- Statistical bounds for entropic optimal transport: sample complexity and the central limit theorem
- Optimal transport for conditional domain matching and label shift
- Unbalanced minibatch Optimal Transport; applications to Domain Adaptation
- Oversampling for Imbalanced Data via Optimal Transport
- CASIA@SMM4H'22: A Uniform Health Information Mining System for Multilingual Social Media Texts
- Convolutional Auto-Encoder and Adversarial Domain Adaptation for Cross-Corpus Speech Emotion Recognition
- Sampling a Longer Life: Binary versus One-class classification Revisited