Expert-Move Bottlenecks for Emotion Recognition in Professional Dialogue

When a system tries to read a speaker's emotion from a conversation, the previous turns help a great deal. The trouble is what the model actually picks up from those turns. In professional conversations such as psychotherapy sessions, clinical interviews, or customer service with a scripted agent, one side of the dialogue is an expert following domain-specific routines. The expert's turns carry real emotional triggers. A therapist's interpretation can provoke a client, and an agent's third reprompt can turn "that's fine" into frustration. Those same turns also carry position and phrasing artifacts that a classifier can memorize. We propose to route all dialogue history from the expert side through a small, typed channel: the model first predicts which domain-specific move the expert made (for example Support versus Interpretation in therapy, or confirm versus reprompt in a booking system), and only an embedding of that move reaches the emotion classifier. The client's own prior turns still flow in as full text.

Put in two sentences: context helps emotion recognition in expert dialogue largely because of what the expert did, so we make "what the expert did" an explicit, domain-vocabulary variable. That should keep most of the benefit of context, make shortcut learning measurable, and generalize better to unseen experts than feeding raw expert text to a general-purpose encoder.

Core Claim

We make two claims. First, in expert-led dialogue, a context model whose expert-side history passes through a bottleneck of predicted expert moves will match a full-text context model on standard in-distribution splits, within about one point of F1. Second, the same bottleneck model will be clearly better when the test conversations come from experts the model never saw, or when expert turns are rephrased or swapped. The second claim is the one that matters.

A further claim is diagnostic. Because the expert channel is a small discrete vocabulary, we can measure exactly how much label information arrives through it alone. That gives a direct, per-dataset estimate of the "interviewer shortcut." Otherwise this shortcut is only discovered after the fact, as DAIC-WOZ: On the Validity of Using the Therapist's prompts did when it found that a graph model reading only the virtual interviewer's prompts reached 0.88 F1 for depression.

Where This Sits

Three strands of prior work motivate the proposal. Predicting Client Emotions and Therapist Interventions in Psychotherapy Dialogues jointly predicts therapist intervention types and client emotion over 872 Hebrew sessions, using an AlephBERT encoder with adapters and a fixed three-turn window. It reports that context helps, with 0.66 micro F1 for emotion. It treats the two tasks as parallel outputs, and it never asks whether the intervention should be the channel through which context enters the emotion prediction. EmoWOZ shows that context lifts weighted F1 from 73.5 (BERT) to 79.7 (ContextBERT) in task-oriented dialogue, and that minority classes such as Dissatisfied remain hard. The earlier frustration-detection work of Liscombe et al. found that system actions such as apologies and reprompts are strong predictors. We build on that observation but use the actions as a learned, trained bottleneck rather than as handcrafted input features.

One paper constrains the design. The DAIC-WOZ study shows that adding the interviewer's turns raises Longformer depression F1 from 0.64 to 0.80 and up to 0.90, mostly through question-sequence artifacts. A naive expert-move channel could carry exactly the same shortcut, because an interviewer's choice of follow-up question is itself a reaction to the participant. We do not claim the bottleneck removes this shortcut. We claim it makes the shortcut visible and bounded, and we test that honestly.

The domain-knowledge and domain-embedding literatures supply the method rather than the motivation. KMQA and MKGAT show that domain structure improves over text-only encoders in medical reading comprehension and biomedical entity recognition. MKGAT, for example, reaches 91.61 F1 on NCBI-disease versus 89.36 for Bioformer alone, and its ablation shows that typed relations beat untyped concepts. Neither handles dialogue. We borrow a narrower idea from them: a domain's own typed vocabulary, here its inventory of expert moves, is a better carrier of meaning than generic token embeddings.

SKAIG injects general commonsense relations such as intent and effect into conversational graphs. Is Discourse Role Important for Emotion Recognition in Conversation? learns unsupervised discourse roles. Both operate on chit-chat with generic categories. Our channel instead uses the domain's own professional act inventory, supervised where labels exist. The positivity bias of general LLMs in counseling, documented in Are Large Language Models Possible to Conduct Cognitive Behavioral Therapy?, is one reason we expect generic encoders to misread expert moves such as confrontation or interpretation.

Research Program

The model has two parts. An utterance encoder is first adapted to the domain by continued pretraining on unlabeled in-domain transcripts, meaning further training of the language model on raw domain text without labels. For the Hebrew corpus this text is the 676 unannotated "872_Silver" sessions; for EmoWOZ it is the MultiWOZ and DialMAGE text. For each expert turn, a move classifier predicts a distribution $z_t$ over the domain's move inventory $\mathcal{A}$: the annotated intervention categories in the Hebrew corpus, and the system dialogue acts in the task-oriented data. The emotion classifier for client turn $t$ then conditions on three inputs: the full text of the current turn $u_t$, the client's own previous turns $c_{<t}$, and only the expected move embeddings of preceding expert turns,

$$ \hat{y}_t = f\big(u_t,\; c_{<t},\; \textstyle\sum_{a \in \mathcal{A}} z_{s}(a)\, E_a \;\text{for } s<t\big), $$

where $E_a$ is a learned embedding for move $a$, initialized from the domain-adapted encoder's representation of that move's label description, and $f$ is a small recurrent network over turns. We train the move classifier and the emotion classifier jointly, with move supervision on the gold sessions and pseudo-labels on the silver sessions. We add a sequence-level variant that restricts the channel to coarser move groups, so we can trace how channel granularity trades accuracy against shortcut exposure.

The analysis side is as important as the model. We define the move-only probe as a classifier that sees only the sequence $z_{<t}$ and nothing the client said. Its score is a per-dataset upper bound on how much emotion information can come from the expert's behavior alone. On DAIC-WOZ, where Ellie's prompts come from a script, we cluster prompts into move types. This lets us check whether the move channel reproduces the 0.88 interviewer-only shortcut or falls well below it.

Baselines

The naive reference points are a context-free utterance classifier and a model that sees only the client's own turns. The context-free classifier is BERT in EmoWOZ, which reports 73.5 weighted F1, and the context-independent adapter model in the Hebrew corpus. The client-turns-only model is the standard fix recommended after the DAIC-WOZ finding; that paper reports 0.64 F1 for participant-only Longformer. Our model must clearly beat the client-only reference, or the expert channel adds nothing.

The strongest existing competitor is a full-text context model on the same domain-adapted encoder. On EmoWOZ this is ContextBERT (79.7 weighted F1, from EmoWOZ). On the Hebrew corpus it is the three-turn context model with adapters (0.66 micro F1 for emotion, from the psychotherapy paper). We rerun both ourselves with the identical domain-adapted encoder, identical history length, identical tuning budget, and the same therapist-disjoint splits, so the only difference is whether expert turns arrive as text or as moves.

We also run the multi-task version of that competitor, which predicts moves and emotion jointly from full-text context. This isolates the bottleneck from the mere presence of move supervision.

As an LLM competitor, we fine-tune an open 7B model with low-rank adapters on the same multi-turn prompts, following DialogueLLM and Affect Recognition in Conversations Using Large Language Models. That second paper reports near-state-of-the-art EmoWOZ results with half the training data but gives no number that can be quoted here. For this LLM baseline on the Hebrew corpus, no reported result was found on Lacuna. We match its training examples and splits, and we report its compute separately.

The key ablations are three. We replace predicted moves with gold moves, to measure how much move-classifier error costs. We replace the domain-adapted encoder with the general one, to isolate the domain-embedding contribution. We randomize the move embeddings, to confirm that the channel carries meaning and is not acting as a positional signal.

Beating the strongest competitor means two things together. The bottleneck model must stay within one F1 point of the full-text model in distribution. It must also exceed that model by a statistically reliable margin, across seeds, on held-out therapists and under expert-turn perturbation.

Why not just…?

Why not just continue pretraining the encoder on in-domain transcripts and feed full-text context? This is the most serious threat. Domain-adaptive pretraining could well capture most of what the domain vocabulary buys. We therefore give it to every baseline, so it is not our contribution. If a domain-adapted full-text model matches the bottleneck on held-out therapists and under perturbation, the bottleneck idea fails, and we will report that. The claim that survives is that the typed channel adds robustness and auditability beyond domain adaptation, not that it replaces it.

Why not just train moves and emotions jointly, as the psychotherapy paper already does? Joint training shares an encoder but still lets raw expert phrasing reach the emotion head. Phrasing is exactly where therapist-specific style and interview-script position live. The multi-task baseline above tests this directly. If it generalizes to unseen therapists as well as the bottleneck does, the constraint is unnecessary.

Why not just fine-tune a large language model on the transcripts? That study reports that fine-tuned LLMs approach supervised results on EmoWOZ but that in-context learning helps little on clinical interviews. Nothing in the file shows that LLMs avoid interviewer shortcuts; a longer context window gives more of them. We include the fine-tuned LLM baseline and apply the same perturbation and held-out-expert tests to it.

Why not just drop the expert's turns entirely? This is safe but discards real signal. The 0.64 to 0.80 gap on DAIC-WOZ, and the 73.5 to 79.7 gap on EmoWOZ, show that at least part of the context gain is available. The question is how much of that gain the move channel recovers.

Proposed Experiments and Decisive Checks

We evaluate on three datasets. The Hebrew psychotherapy corpus uses therapist-disjoint splits over its 59 therapists and reports micro F1 and per-class F1. EmoWOZ reports weighted F1 and recall on Dissatisfied, the class where EmoWOZ reports 31.4% rising to 60.4% only with human-machine data. DAIC-WOZ serves purely as a shortcut stress test at the interview level.

The first decisive check is generalization to unseen experts. The gap between random and therapist-disjoint splits should shrink under the bottleneck compared with full-text context.

The second check is counterfactual swapping. We replace each expert turn with a turn from another session that carries the same gold move. The full-text model's predictions should shift more than the bottleneck model's. We also swap in a different move, and the bottleneck model should shift in clinically plausible directions, for example toward negative client affect after a confrontation. Clinicians will blind-rate a sample of these shifts.

The third check is the move-only probe on DAIC-WOZ. If clustered interviewer moves alone approach the 0.88 interviewer-only figure, the channel inherits the shortcut. We would then restrict the channel to moves that precede each participant answer and report the cost in accuracy.

As an external validity check, we correlate session-aggregated predicted emotions with clients' post-session mood self-reports (POMS). The baseline correlation is 0.27 for positive and 0.21 for negative emotions, as reported in the psychotherapy paper. The bottleneck model should at least match these correlations on held-out therapists, where we expect full-text models to degrade.