Models that recognize emotions in two-person conversations fail most often at the moments that matter: when a speaker abruptly changes state, such as going from neutral to angry, because of something the partner just said. Current systems carry a built-in "emotional inertia," the assumption that a person stays in the state they were in. Contrastive learning trains a model to pull similar examples together and push different ones apart inside its internal representation. In this field it has only been used to group utterances by their final emotion label. We propose to group them by the transition instead. The model's representation of each turn is split into two parts. One records where the speaker was emotionally. The other records what move the partner's last turn caused. The second part is trained contrastively so that all "neutral→anger" turns cluster together, all "frustrated→sad" turns cluster together, and so on.
The bet is that emotion shifts are rare and scattered when you look at them by final label, but they share structure when you look at them as transitions. A representation built explicitly around partner-caused change should therefore improve recognition exactly on shift turns, where inertia-heavy models and state-level contrastive methods both struggle. It should do this without hurting stable turns. This needs no new annotation, because conversational emotion datasets already label every turn, so the transition type for each training turn is known.
Core Claim
Three bodies of work fall short of one another. Dyadic interaction models separate the two forces that drive conversation, self-persistence and partner influence, but only architecturally and with plain cross-entropy training. DialogXL uses separate speaker and listener attention masks. DialogueEIN uses intra- and inter-speaker masks. EmotionIC gates a recurrent unit by inertia and contagion. Nothing forces the "partner" pathway to encode partner-induced change rather than another copy of utterance semantics. EmotionIC itself reports that contagion is harder to model than inertia, and MELD and DialogXL both identify emotion-shift turns as the main failure mode. These papers motivate the proposal.
The contrastive conversational emotion methods, SPCL with class-prototype queues and EACL with label anchors kept angularly dispersed, regularize a single undifferentiated utterance vector against state labels. SPCL's curriculum actually postpones the ambiguous turns where shifts tend to fall. These papers supply the method we reuse, prototype queues that stay robust at small batch sizes, and they constrain the design. EACL's finding that pulling classes toward centroids collapses similar emotions such as happy and excited warns against simply clustering transitions toward means without dispersion.
The closest structural analogue is SPYL. It uses contrastive estimation to disentangle static and dynamic latent factors, but for one agent and without labels. Our "dynamic" factor is instead defined by a second agent's influence and keyed by supervised transition types.
Research Program
We build on the backbone used by SPCL and EACL: a prompt-tuned SimCSE-RoBERTa-Large encoder that reads the target utterance with preceding turns. On top of it sit two small attention heads. The self head attends only to the target speaker's own previous turns, like DialogXL's speaker mask, and produces $z^{s}_t$. The partner head attends only to the partner's turns since the speaker last spoke, like the listener mask, and produces $z^{p}_t$. The classifier predicts the emotion from the concatenation.
Training adds two prototype-contrastive terms in the style of SPCL. The first keys $z^{s}_t$ on the speaker's previous emotion $y_{t^-}$, where $t^-$ is that speaker's preceding turn. The second keys $z^{p}_t$ on the transition type $(y_{t^-}\!\to y_t)$:
$$ \mathcal{L} = \mathcal{L}_{CE}(y_t) + \lambda_s\,\mathcal{L}_{proto}(z^{s}_t;\,y_{t^-}) + \lambda_p\,\mathcal{L}_{proto}(z^{p}_t;\,y_{t^-}\!\to y_t) + \lambda_a\,\mathcal{L}_{Ag}. $$
Here $\mathcal{L}_{proto}$ contrasts a representation against per-key prototypes averaged from a memory queue, as in SPCL. $\mathcal{L}_{Ag}$ is EACL's angle penalty, applied to the transition prototypes so that adjacent transitions such as frustrated→angry and neutral→angry do not collapse into each other. The $\lambda$ values are loss weights.
Transition keys are sparse: IEMOCAP's six classes give 36 types, many of them rare. We therefore pool rare transitions by target emotion within a "shift" versus "stay" split, so the minimum key is "any→anger (shift)." This is exactly the regime where SPCL's queues beat in-batch contrast; SPCL reports only a 0.82% drop from batch size 32 to 4, against about 7% for standard supervised contrastive learning. Previous-turn labels are used only as training keys and never as test-time inputs.
The analysis side tests whether the two parts really separate. Linear probes should find that $z^{s}$ predicts the previous emotion but not the transition, and that $z^{p}$ does the reverse. In a partner-swap test, we replace the partner's intervening turn with one drawn from another dialogue. This should move $z^{p}$ substantially and $z^{s}$ little, and the prediction should shift in the direction implied by the substituted turn's own transition context.
Baselines
The naive reference is the same SimCSE-RoBERTa-Large prompt encoder trained with cross-entropy only and the same context window. It shows how much any regularizer adds.
The strongest encoder-scale competitor is EACL, which reports 70.41% weighted F1 on IEMOCAP, 67.12% on MELD and 40.24% on EmoryNLP (EACL). SPCL is a close second at 69.74% on IEMOCAP and is slightly higher on MELD at 67.25% (SPCL). We reimplement both on our exact backbone, data splits and context length, with the same hyperparameter search budget of equal trials and seeds. Our method differs from them only in the two-head split and the transition keys.
The strongest interaction model, EmotionIC at 69.61 weighted F1 on IEMOCAP (EmotionIC), will be run with its reported configuration on our backbone features where feasible.
LaERC-S reports 72.40% on IEMOCAP and 69.27% on MELD (LaERC-S). It uses an instruction-tuned 7B model and is not a compute-matched comparison. We report it as a ceiling rather than a target, and add one run that places our loss on a LoRA-tuned model of the same size if compute allows.
Two ablations isolate the key ingredient. The first is a state-keyed control: the same two heads, but both trained with prototypes keyed on the current label. This tests whether transition keying, rather than the extra heads, does the work. The second is an auxiliary transition classifier on $z^{p}$ trained with cross-entropy instead of prototypes, which tests whether the contrastive form matters.
Since the claim is about shifts, beating a baseline means a statistically significant gain in weighted F1 on shift turns across at least five seeds. We define shift turns as those where a speaker's label differs from their own previous turn. Overall weighted F1 must stay within noise of the baseline or improve. No reported shift-turn F1 was found on Lacuna for EACL, SPCL or EmotionIC, so we will compute these from our reimplementations.
Why not just…?
Why not just add an auxiliary head that classifies the transition type with cross-entropy? This is the most serious threat. It uses the same supervision and is simpler. Our argument is that rare transitions get very few gradient updates under cross-entropy at realistic batch sizes, while prototype queues keep a stable target for each key; SPCL's batch-size result is the evidence for that mechanism at the state level. The argument may fail. The auxiliary-head ablation is run at batch sizes 4, 16 and 32. If it matches the contrastive version on shift-turn F1, we will report that the gain comes from transition supervision itself and simplify the method accordingly.
Why not just put SPCL or EACL on top of an interaction model like EmotionIC? That combination still clusters by final state. Shift turns are then pulled toward the same centroid as the far more numerous stable turns of the target class, which is the over-smoothing MELD and DialogXL describe. The state-keyed two-head control is effectively this combination on a matched architecture. If it closes the gap on shift turns, the transition-keying claim is wrong.
Why not just upweight shift turns in the loss? Reweighting changes how much shift turns count but gives them no shared structure. A neutral→anger shift and a sad→anger shift are treated as unrelated anger examples. We include a shift-reweighted cross-entropy baseline tuned to the same budget. Our method should beat it specifically on low-frequency transition types, where shared structure across keys is the only source of help.
Why not just use an LLM that reasons about listener reactions? LaERC-S shows that listener reactions are informative and reports the highest numbers in the file. Its signal is generated text from a 7B model at inference time. Our question is whether a compact encoder can absorb partner influence into its geometry. The two approaches are complementary, and the LoRA run tests whether our loss adds anything on top of LLM-scale encoders.
Proposed Experiments
The main testbed is IEMOCAP, six-class with session-held-out evaluation, because it is genuinely dyadic (IEMOCAP). MELD and EmoryNLP test whether the idea survives multi-party, short dialogues, which DialogueEIN and EmotionIC note leave little history to work with. For those datasets, "partner" means all other speakers' turns since the target speaker last spoke. DailyDialog, which is dominated by the neutral class, is a stress test for whether sparse shift keys still help when almost every turn is "stay."
We report overall weighted F1, shift-turn and stay-turn weighted F1 separately, per-transition-type recall for the ten rarest types, and the probe and partner-swap measurements described above.
The decisive checks are three. First, on IEMOCAP, shift-turn F1 must beat the reimplemented EACL, the state-keyed control and the auxiliary cross-entropy head without losing stay-turn F1. Second, the probes must show the predicted asymmetry, which confirms that the two parts encode different things rather than duplicating semantics. Third, the shift-turn gain should be larger on IEMOCAP than on MELD. Failing this would suggest that the mechanism does not depend on sustained two-person interplay, and that the method is a generic regularizer rather than a model of dyadic influence.
References
- DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition
- DialogueEIN: Emotion Interaction Network for Dialogue Affective Analysis
- EmotionIC: Emotional Inertia and Contagion-driven Dependency Modelling for Emotion Recognition in Conversation
- MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
- Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation
- Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation
- Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation
- LaERC-S: Improving LLM-based Emotion Recognition in Conversation with Speaker Characteristics
- IEMOCAP: interactive emotional dyadic motion capture database