Programs that label the emotion of each turn in a conversation fail most often on the turn where a speaker's feeling changes. The usual explanation is that shifts are rare, since most consecutive turns keep the same emotion. That is only partly true. On IEMOCAP about 70% of consecutive turns keep the same emotion, but on EmoryNLP only 36% do (EmoTransKG). The binary imbalance of shift versus no-shift is mild, and it is easy to fix by reweighting the loss. The severe imbalance sits one level down. Once each transition is written as a pair (previous emotion, current emotion), a seven-class dataset such as MELD has 49 transition types, and many of them, such as neutral→disgust or fear→sadness, have only a handful of training examples. These rare transition types are where minority emotions collapse into "neutral," and where upweighting cannot help much: multiplying the loss on five examples still teaches the model about only five points.
We propose to generate synthetic training examples for rare transition types directly in the model's internal feature space. Each synthetic example is a jointly interpolated pair of previous-turn and current-turn representations, sampled from inside small groups of real pairs that share the same transition type. The sampling follows Simplicial SMOTE, which draws points from the interior of a triangle or tetrahedron formed by neighbouring examples rather than along a line between two of them. The claim is specific. Oversampling keyed to the transition type, and carried out on turn pairs, should raise recall on rare transitions and on the minority emotions they end in. Oversampling keyed to the single-turn emotion label, which prior dialogue work found ineffective, should not. Loss reweighting by transition frequency should not either.
Core Claim
The two literatures meet at a gap that the research file states plainly. Shift-aware emotion recognizers never modify the training distribution, and oversamplers never respect conversational structure. On the emotion side, CFN-ESA adds an auxiliary binary loss that predicts whether the emotion changed between turns. Shapes of Emotions uses a paired-input (Siamese) text network whose shift probability resets the memory of a recurrent network. Emotion-Shift Aware CRF uses a pre-trained shift detector to reshape the transition scores of a conditional random field, a sequence decoder that scores whole label sequences. ERNetCL orders training conversations from few shifts to many. All four train on the natural distribution of transitions, and all four treat "shift" as a single binary event. ERNetCL's authors report that minority emotions still collapse into neutral, and they name synthetic augmentation as future work. This group of papers motivates the proposal and supplies the backbones we build on.
On the oversampling side, SMOTE creates new minority points on line segments between neighbours. Simplicial SMOTE samples inside higher-dimensional groups of mutual neighbours and reports average F1 gains of 4.5–5% over SMOTE on 21 tabular datasets. SMOTExT interpolates sentence embeddings and decodes the results back into text. All three treat examples as independent points. Simplicial SMOTE supplies our sampling method. SMOTExT's reports of semantic drift when interpolating between distant points constrain our design: we never decode back to text, and we only interpolate among close neighbours of the same transition type.
A third body of work constrains the proposal directly. Vaidya et al. and Bogatyreva et al. found that oversampling dialogue data did not improve recall on disgust or anger, and that it produced overfitted spikes around duplicated examples. We take this negative result seriously. Our hypothesis is that it reflects what was oversampled, namely isolated utterances grouped by their own emotion label, rather than a flaw in oversampling as such. One ablation below tests this hypothesis directly. If utterance-keyed and transition-keyed oversampling perform the same, the hypothesis is wrong.
Research Program
We begin with a trained conversational emotion model. The main backbone is a text-only context encoder of the kind used in CFN-ESA's text branch. For turn $t$ it gives a context-aware representation $u_t$. We form a pair vector $z_t = [u_{t-1}; u_t; u_t - u_{t-1}]$ labelled with its transition type $c_t = (y_{t-1}, y_t)$. Here $y$ is the gold emotion, and $t-1$ is the preceding utterance. In multi-party data we also test a variant in which $t-1$ is the same speaker's previous turn.
For every transition type with fewer than a target count of examples, we build a nearest-neighbour graph using only pairs of that type. We then find small cliques in that graph: groups of up to four pairs that are all mutual neighbours. We sample synthetic pairs as
$$\tilde z = \sum_{j=1}^{m} w_j\, z^{(j)}, \qquad w \sim \mathrm{Dirichlet}(\alpha \mathbf{1}_m),$$
where $z^{(1)},\dots,z^{(m)}$ are the pairs in the clique, $w$ are random mixing weights that sum to one, and $\alpha$ controls how far the samples spread into the clique's interior. Because the three parts of the pair vector are mixed with the same weights, the synthetic previous turn, current turn and difference stay consistent with one another. This is the property that utterance-level oversampling lacks.
A simple guard rejects any synthetic pair whose nearest real neighbour belongs to a different transition type. The guard addresses the "holes" problem that Simplicial SMOTE raises. Transition types with too few examples to form any clique fall back to line interpolation between their two nearest members.
We use the synthetic pairs in a second training stage. The encoder is frozen, and we retrain two components on real plus synthetic pairs. The first is the transition head, which predicts $c_t$, or equivalently shift and target emotion. The second is the per-turn emotion classifier. We also report a variant that fine-tunes the encoder end to end while the synthetic pairs are refreshed each epoch from the current representations. Retraining the transition head this way also produces an improved shift probability, which we feed into the ESA CRF decoder and into the reset gates of the Shapes-of-Emotions model. This addresses the weakness the ESA CRF paper itself identifies: its decoder is only as good as its shift detector, and the detector is trained on unbalanced data.
The analysis tracks how the benefit varies with the rarity of each transition type. The prediction is that gains appear in the rarest types and are close to zero in common ones. We also check calibration, since label-shift work warns that training on a rebalanced distribution distorts predicted probabilities at test time. Where calibration degrades, we apply a post-hoc adjustment of prior probabilities toward the true transition frequencies.
Baselines
The naive reference is the same backbone and heads trained on the natural data with plain cross-entropy. It shows how much of any gain comes from the second training stage alone.
The strongest published systems are CFN-ESA and ESA CRF. CFN-ESA reports 66.70% weighted F1 on MELD and 71.04% on IEMOCAP (CFN-ESA). ESA CRF reports 74.47% unweighted accuracy on 4-class IEMOCAP and a +4.16% unweighted-accuracy gain on shift turns over its DED baseline (Emotion-Shift Aware CRF). For context, ERNetCL reports 69.73% weighted F1 on IEMOCAP, and DVL-CER reports 70.11% on IEMOCAP and 39.92% on EmoryNLP (DVL-CER). We found no reported per-transition-type or macro-F1 results on Lacuna for any of these systems. We will therefore re-run CFN-ESA's text branch and ESA CRF from their published designs, with our splits and seeds, and compare against our reproductions. We will also report whether each reproduction matches its published headline number.
We run each oversampling method on top of these reproductions: the same encoder, the same second-stage budget, and the same hyperparameter search of twenty trials per method, selected on validation macro-F1. Beating a method means a statistically reliable gain over five seeds in two metrics: recall on rare transition types, and macro-F1 over emotions. Weighted F1 must not drop by more than half a point. Weighted F1 is dominated by the neutral class, so a method can improve it while failing minorities.
Three baselines isolate the key ingredient. The first is cost-sensitive reweighting by inverse transition-type frequency, together with a focal-loss variant, using the same heads and no synthetic data. The second is utterance-level Simplicial SMOTE keyed on the single-turn emotion label, which is the setting Vaidya et al. found wanting. The third is transition-keyed line SMOTE, which isolates the benefit of sampling inside cliques rather than along lines. We also run cross-type mixup in the style of From SMOTE to Mixup and the ERNetCL curriculum. Both have the same compute budget.
Why not just…?
Why not just reweight the loss by transition frequency? This is the most serious threat, and for the binary shift label we expect reweighting to be as good as oversampling, because the 70/30 split on IEMOCAP is not extreme. The proposal's claim is confined to rare transition types. There, reweighting concentrates gradient on a few fixed points, which is the spike pattern Bogatyreva et al. observed even when weighted losses were combined with augmentation. The reweighting baseline is run with a matched tuning budget. If it matches transition-keyed oversampling on rare-type recall, we conclude that the synthetic geometry adds nothing and report that.
Why not just duplicate conversations that contain rare shifts? Duplication adds no new points in feature space, and Vaidya et al. found that it did not improve disgust recall. We include random duplication of rare-type pairs as a cheap arm. It should behave like reweighting, and a clear gap between duplication and clique sampling would show that the new interior points are what matters.
Why not just use plain SMOTE, or mixup, on the turn representations? Plain SMOTE confines samples to lines between two neighbours, which is the coverage limitation shown in Simplicial SMOTE. That paper's evidence is tabular, however, and it may not carry over to dense text features. The transition-keyed line-SMOTE ablation settles this question. We will report honestly if the simplex adds nothing in this setting, in which case the contribution reduces to keying on transitions. Cross-type mixup blends labels across transition types, and it may blur exactly the boundaries between neighbouring emotions such as disgust and anger that ERNetCL identifies as hard. We test mixup rather than assume this.
Why not just fix the decoder, as ESA CRF does? The ESA CRF authors show that their decoder comes close to its upper bound only when it receives ground-truth shift labels. The remaining problem is therefore upstream, in the detector. We measure this directly in the experiments below.
Proposed Experiments
We use four datasets. IEMOCAP is used in both its 4-class and 6-class forms, text-only, and it has high emotional inertia. MELD has 7 classes across multiple speakers, extreme minority classes, and 49 transition types. EmoryNLP has shifts as the majority, so binary rebalancing should help least there, while rare transition types still exist. DailyDialog, dominated by neutral, stresses minority collapse. The per-type count threshold for oversampling is set on validation data.
The first decisive check stratifies results by the training frequency of each transition type. Our claim predicts gains that grow monotonically as types become rarer, and that are present for transition-keyed sampling but absent for utterance-keyed sampling. A flat profile, or equal gains for the two, would refute the claim.
The second check is a gap-closure test on ESA CRF. On IEMOCAP and on MELD, where ESA CRF was never evaluated, we run the decoder three ways: with its original shift probability, with the shift probability from our retrained transition head, and with ground-truth shifts. We then report what fraction of the gap between the original and ground-truth runs the retrained head closes, on shift turns specifically. The ESA CRF paper gives no numeric upper bound from its ground-truth run in the material we have, so we will establish that bound ourselves.
The third check guards against artefacts of the feature space. For a sample of synthetic pairs, we retrieve the nearest real training pairs and have two annotators judge whether those neighbours plausibly represent the labelled transition. We also verify that no synthetic point lies nearer to another type's pairs than to its own; the rejection guard enforces this, and we report the rejection rate.
Finally, we report the expected calibration error of predicted probabilities before and after prior adjustment, so that any gain in minority recall is not bought with miscalibrated confidences.
References
- EmoTransKG: An Innovative Emotion Knowledge Graph to Reveal Emotion Transformation
- CFN-ESA: A Cross-Modal Fusion Network With Emotion-Shift Awareness for Dialogue Emotion Recognition
- Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts
- Emotion-Shift Aware CRF for Decoding Emotion Sequence in Conversation
- ERNetCL: A novel emotion recognition network in textual conversation based on curriculum learning strategy
- SMOTE: Synthetic Minority Over-sampling Technique
- Simplicial SMOTE: Oversampling Solution to the Imbalanced Learning Problem
- SMOTExT: SMOTE meets Large Language Models
- CLTeam1 at SemEval-2024 Task 10: Large Language Model based ensemble for Emotion Detection in Hinglish
- QiMP at SemEval-2025 Task 11: Optimizing Text-based Emotion Classification in English Beyond Traditional Methods
- Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment
- Dual-View Learning for Conversational Emotion Recognition Through Context and Emotion-Shift Modeling
- From SMOTE to Mixup for Deep Imbalanced Classification