A language model told to play a character tends to drift after a handful of turns. It slides back toward a generic polite assistant, or it starts copying the person it is talking to. In spoken dialogue the copying is emotional: an agent meant to stay calm becomes agitated because the user's voice is agitated. We propose to catch and correct this drift by running the problem backward. A second model reads only the agent's candidate reply and tries to reconstruct two things from it: the agent's intended emotional state, and the user's emotional state as heard in their voice. A good reply lets you recover both, in the right places. The agent's own feeling should come back as the agent's, and the user's feeling should show up as something the agent registered, not something it took on. A mirroring reply makes the user's emotion come back as the agent's.
The opportunity comes from three findings that have not been put together. Reverse models, which predict context from responses, have so far been used only offline, to find inputs that break dialogue safety (Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation; Eliciting Language Model Behaviors using Reverse Language Models). Emotion-aware spoken systems pass continuous speech features into a language model, but nothing checks whether the perceived emotion actually shapes the reply (E-chat). And multimodal reasoning work shows that models often perceive inputs correctly and then fail to use them (MATH-Vision; GeoSQA). We treat emotional mirroring as a version of that perceive-but-misuse failure, and we use reverse reconstruction both to measure it and to suppress it at decoding time.
Core Claim
Persona drift in emotion-aware spoken dialogue has a specific, measurable signature. As turns go on, the information in the agent's reply about the user's emotion shifts from "registered by the agent" to "adopted by the agent." We claim a reverse reconstruction model can expose this shift and can also be used to choose among sampled replies. Concretely, the reverse model reads a candidate reply and recovers two continuous emotion vectors, one for the agent and one for the user. Reranking with this model should lower mirroring without making the agent flat. Our success criterion is that the agent keeps its persona and still responds appropriately to the user's emotion.
The idea sits in a specific place relative to existing work:
- Split-Softmax and Persona Dynamic Decoding (PDD) act forward on text tokens (Measuring and Controlling Persona Drift in Language Model Dialogs; Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agents). Neither separates the agent's affect from the partner's.
- AMOUR keeps a hand-designed state that a separate program updates before each reply (Amour). It never checks whether the generated reply actually expresses that state.
- R-CoT uses backward derivation to verify synthetic geometry data offline (R-CoT). We apply the same "check by going backward" logic online, to affect.
We considered and rejected the most obvious reading of the research file. That reading is terminal-anchored decoding, which would commit to a turn-final emotional target and then backfill the reply. Its advantage in poetry and rap comes from hard end-of-line constraints (DeepRapper; Ghazvininejad et al.). Persona consistency is a constraint on every turn, not an end constraint. A forward "decide the state, then speak" scheme like AMOUR's would almost certainly do as well, so backward decoding of tokens adds nothing. Backward reconstruction as a verifier is where reversal earns its place.
Research Program
Forward agent. We build an E-chat-style agent: a self-supervised speech encoder feeds a 7B-scale chat model through a small connector, so the user's emotion arrives as continuous features rather than a label. The persona includes an affective baseline. That baseline is given as a continuous anchor vector taken from reference emotional speech, following the zero-shot style-transfer line (Seen and Unseen emotional style transfer; ZEST).
Reverse model. The reverse model is trained on response-to-context pairs. Given the agent's reply text and the preceding turn, it predicts two vectors in the same continuous space: $\hat a$, the agent's affect, and $\hat u$, the user's affect.
Training data. Training pairs come from E-chat200 and from MELD conversations. In both, the true speaker affect for each utterance is computed by a speech emotion encoder on the actual audio. To prevent the evaluator from sharing the reverse model's biases, we hold out a different encoder family for evaluation only.
Decoding score. At decoding time the agent samples $N$ candidate replies. Each candidate $r$ is scored as
$$s(r) = \log p_{\text{fwd}}(r) - \lambda\,\lVert \hat a(r) - a^{*} \rVert - \mu\,\cos\!\big(\hat a(r) - a^{*},\; u_t - a^{*}\big)$$
The terms are defined as follows:
- $p_{\text{fwd}}$ is the forward agent's probability of the reply.
- $a^{*}$ is the persona's affective anchor.
- $u_t$ is the user's affect at turn $t$, measured from the user's audio.
- $\lambda$ and $\mu$ are weights.
The second term penalizes leaving the persona's affect. The third penalizes leaving it specifically in the direction of the user, which is the mirroring signature. Responsiveness should survive because $\hat u$ is not penalized. We also report how well $\hat u$ recovers $u_t$ as a separate "did the agent register the user" score.
Diagnostic study. Before using the reverse model for control, we measure the perceive-but-misuse gap directly. A linear probe on the forward model's hidden states tests whether the user's emotion is decodable at the input. Reverse reconstruction tests whether it appears in the output, and whether it appears as the agent's own state. Following GeoSQA's design, we then replace the audio with a gold text description of the user's emotion. If mirroring persists with perfect perception, the failure lies in how the model uses what it perceived, not in what it hears.
Baselines
Do nothing. The naive reference is the forward agent with the persona in the system prompt, sampled once. It establishes the drift curve over 8–16 turns. On text instruction-following, the persona-drift benchmark reports roughly a 10% drop for GPT-3.5-turbo-16k over 8–16 rounds (Measuring and Controlling Persona Drift). Showing a comparable drop in our spoken setting is a precondition for the rest.
Split-Softmax. This method boosts attention to system-prompt tokens with a power-law scaling at inference time (Measuring and Controlling Persona Drift). It is the strongest training-free fix for attention decay. We tune its exponent on the same development split we use to tune $\lambda$ and $\mu$, with an equal number of trials. No exact win numbers are reported on Lacuna for Split-Softmax; the paper claims only a Pareto improvement over prompt repetition. Beating it means lower mirroring at equal or better responsiveness.
PDD. PDD reweights persona traits per scenario at decoding time (PDD) and is the strongest existing persona-following method. It reports win rates above 60% against in-context prompting on BeyondDialogue. We run PDD with the affective anchor written as a text trait, and we match wall-clock compute to our $N$-sample reranking. Beating it means a head-to-head win rate above 50% under the same judge on BeyondDialogue, plus lower mirroring on our spoken set.
AMOUR-style forward state. This baseline gives the agent an explicit state that is updated with inertia before each reply (Amour). It tests whether "plan the state first" already solves the problem.
Forward classifier reranker. This is the key ablation. It uses the same $N$ samples and the same score, but $\hat a$ comes from a model trained to read agent affect from the reply alone, without the reverse model's joint reconstruction of the user's affect.
Two further ablations. One drops the mirroring term ($\mu=0$), which tests whether the direction-specific penalty matters. The other replaces continuous vectors with five discrete emotion labels, which tests whether implicit encoding matters.
Why not just…?
Why not just repeat the persona and the current emotion label every turn? The drift paper found prompt repetition Pareto-dominated by Split-Softmax, with degradation persisting over 8–16 turns (Measuring and Controlling Persona Drift). Role-play work reports that prompted traits stay superficial (Character-LLM). We still include repetition in the do-nothing tier rather than assume it fails in speech.
Why not just rerank with a forward persona classifier or an LLM judge? This is the real threat. A reverse reconstruction model is, mechanically, an inverse predictor, and a well-trained classifier might capture most of the gain. Our bet is narrower. Separating the agent's affect from the user's, and penalizing movement toward the user specifically, should catch mirroring that a single "is this on-persona" score misses. In particular, an on-persona-sounding reply can still drift partway toward the user. The classifier-reranker ablation at equal $N$ settles this. If it matches us, the contribution shrinks to the diagnostic finding, and we will report it that way.
Why not just track the state forward, as AMOUR does? AMOUR guarantees that the state stays sensible. It does not guarantee that the reply expresses that state, and the perceive-but-misuse literature (Plot Twist) suggests the gap between the two is exactly where failures happen. We also test the combination, using AMOUR's state as $a^{*}$. If AMOUR alone already closes the mirroring gap on AMOUR-REAL-EVAL scenarios, our verifier is unnecessary. No reported AMOUR metric value was found on Lacuna, so we will measure it ourselves.
Why not discrete emotion tags? Discretizing prosody discards intensity and mid-range nuance (Transferring Emphasis in Speech Translation). Mirroring is often partial, a drift rather than a switch, so labels may miss it. The discrete-label ablation tests this directly.
Proposed Experiments
Spoken drift test set. We construct 16-turn spoken conversations. The agent has an assigned affective baseline. The user's turns come from E-chat200 and the emotional speech dataset of Seen and Unseen emotional style transfer, arranged so that user emotion escalates away from the persona.
Metrics. We measure four things. Persona fidelity is the distance between the agent's affect and its anchor, measured by the held-out encoder on the reply rendered by an emotional TTS. The mirroring index is the lagged correlation between agent and user affect. Responsiveness is recovery of user affect plus human ratings of appropriateness. Text persona scores come from CharacterEval and BeyondDialogue.
Quality guardrail. For spoken quality we report semantic similarity and MOS against E-chat's reported SIM of 74.83 and MOS of 3.56 on E-chat200 (E-chat). Reranking must not fall below these.
Decisive checks.
- The mirroring index must fall relative to PDD and Split-Softmax without a drop in responsiveness. A drop in responsiveness would mean we simply built a flat agent.
- The gold-text-emotion condition must still show mirroring in the do-nothing agent. That outcome would confirm that this is a failure in using perceived emotion, not in perceiving it.
- The full method must beat the forward classifier reranker and the $\mu=0$ ablation at equal compute. Without that result, the reverse, two-sided reconstruction is not the active ingredient.
- Human raters blind to condition must agree with the held-out encoder on which replies mirror. This rules out the possibility that we are optimizing an artifact shared between the reverse model and the evaluator.
References
- Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation
- Eliciting Language Model Behaviors using Reverse Language Models
- E-Chat: Emotion-Sensitive Spoken Dialogue System with Large Language Models
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Measuring and Controlling Persona Drift in Language Model Dialogs
- Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agents
- Amour: A Latent Dynamical System for Controllable and Stateful Social Interaction in LLMs
- R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models
- DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling
- Generating Topical Poetry
- Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset
- Zero Shot Audio To Audio Emotion Transfer With Speaker Disentanglement
- Character-LLM: A Trainable Agent for Role-Playing
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Transferring Emphasis in Speech Translation Using Hard-Attentional Neural Network Models