When the Relationship Changes, Does the Model's Reading of Emotion Change Too?

People read an ambiguous expression through the relationship they have with the person showing it. A surprised face after a negative remark looks more unpleasant when it comes from a partner than from an opponent or a stranger. Relationships, however, change: allies fall out and rivals reconcile. A conversational AI that keeps a memory of a user's social world must therefore keep its record of who-stands-where-with-whom up to date. It must also reinterpret earlier emotional moments when that record changes. A separate line of work shows that language-model memory systems are bad at noticing when an earlier fact has been superseded. They anchor on old, often-repeated information even when a later turn plainly overrides it. Nobody has asked whether this failure also corrupts emotional interpretation, which is where it would do the most social harm.

We propose to measure and fix relational state drift: the model reads an emotional cue through a relationship that is no longer current. We build a benchmark of multi-session dialogues in which the relationship between two people shifts. Each history ends on a deliberately ambiguous emotional moment whose correct reading, judged by human raters, depends on the current relationship. We then test a memory design that, when a relationship changes, flags the emotional interpretations that were drawn under the old relationship and reappraises them, rather than deleting them. The point of reappraisal is that relational change is not a clean fact swap. A betrayal does not erase the warmth of past exchanges, and a reconciliation does not erase recent hurt. A model that simply forgets is wrong in a different way from a model that never updates.

Core Claim

Treating "the relationship between A and B" as one more fact in a fact-tracking memory is necessary but not sufficient. Emotional interpretations formed under an old relationship depend on that relationship and must be re-evaluated when it changes. Some of that earlier affect should persist as residue, and pure invalidation gets this wrong. We predict three outcomes. Long-context models and similarity-based retrieval will read ambiguous cues through the stale relationship at high rates, especially when the old relationship was mentioned often and the update only once. A relation-aware fact tracker will fix most stale readings but will over-correct when residual affect matters. Dependency-triggered reappraisal will reduce both errors.

Grounding in Prior Work

Two studies motivate the target behavior. Emotional Evaluations from Partners and Opponents Differentially Influence the Perception of Ambiguous Faces shows behaviorally and with brain-response timing that partner versus opponent framing changes how unpleasant a surprised face is judged. That study fixes roles per block, and we vary them across a history. Perceptual encoding of emotions in interactive bodily expressions shows that one person's perceived anger depends on the other person's reaction. Emotion is jointly encoded in the pair, and so a memory unit must be about the pair, not about one speaker. Dyer's BORIS system (Affect Processing for Narratives) formalized how friend versus enemy themes invert the emotional meaning of the same event. Its themes were hand-coded and never updated. We keep the inversion idea and make the relational theme a revisable state.

The obsolescence work supplies both the method and the warning. Can Agent Memory Systems Track Evolving State? introduces StateMem, which stores conversation content as structured state units with dependency links and runs a rechecker that flags dependent units when a base value changes. It shows that state drift persists even when all facts are in the prompt, that step-by-step reasoning can rationalize stale facts, and that models fall into a "salience trap" of anchoring on frequently mentioned values. All of its domains are transactional (research, shopping, personal finance), and invalidation there is a clean replacement. We reuse its dependency-and-recheck mechanism. Our change is to recheck by reappraising rather than overwriting. Mitigating Temporal Misalignment by Discarding Outdated Facts predicts how long a fact stays true and lowers confidence once that lifespan is exceeded. This is a scalar expiry, which assumes binary obsolescence, the assumption our residue condition tests.

The closest affect-side systems constrain the design. Amour: A Latent Dynamical System for Controllable and Stateful Social Interaction in LLMs keeps explicit continuous social variables (trust, stress, dominance) for the agent itself and updates them with hand-set smoothing and clipping. It does not track relationships between other people, and its decay is not triggered by evidence. Emotion Recognition in Conversation via Dynamic Personality infers fluctuating personality states from a speaker's own past turns, and Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction (CORECT) builds a graph over a window of 11 past and 9 future utterances. Both work within a single session and have no notion of superseded relational knowledge. Interpersonal Memory Matters: A New Task for Proactive Dialogue Utilizing Conversational History (MapDia) retrieves past topics by semantic similarity. That is exactly the retrieval rule we expect to resurface resolved grievances. DDRel: A New Dataset for Interpersonal Relation Classification in Dyadic Dialogues provides multi-session movie-script dialogues between speaker pairs, but it labels each pair's relationship as static. We re-annotate it.

Research Program

The benchmark (working name RelDrift). The first source is DDRel. Annotators read the sessions of each speaker pair in script order and mark per-session relational stance (cooperative, adversarial, intimate, distant) and the sessions where stance shifts. This turns static pair labels into relational trajectories drawn from real writing. The second source is a controlled set of LLM-drafted, human-edited multi-session histories built as minimal pairs. Two histories are identical except for one short update turn, such as "we patched things up last month" or "I found out she went behind my back." Every history ends on a target moment that is emotionally ambiguous by design, in the spirit of the surprised-face stimuli: a terse reply, a laugh, or "Wow, of course you'd say that."

The controlled histories cover five conditions. The no-change control keeps the relationship fixed. In supersede, the relationship flips. In flip-then-revert, it changes and then changes back. In residue, the relationship changes but recent affect should carry over. In salience-distractor, the old relationship is mentioned many times and the update only once. Ground truth is not hand-coded. At least five raters read each full history and give a categorical emotion reading plus valence and intensity for the target. The label is a human distribution, so residue is measured rather than assumed.

The method (relational ledger with reappraisal). We extend StateMem's state units with two types. The first is a pair-relation unit, with stance and trust for an ordered pair of people. The second is an emotional-episode unit, with who felt what toward whom, and a "derived_from" link to the relation unit current at the time. When a relation unit changes, the rechecker does not delete the dependent episodes. It marks them for reappraisal. A single model call then re-reads each flagged episode under the new relation and writes a revised interpretation together with a residue weight, the share of the original affect that still bears on the present. When the model interprets the target, it sees the current relation, the reappraised episodes, and their residue weights in time order, a "value chain" adapted from StateMemWrapper. The whole system wraps a frozen backbone, so it can be compared fairly against prompting-only methods on the same model.

Baselines

The naive reference is the backbone model given only the final session and the target. It shows how much reading the target depends on history at all. On minimal pairs it should score at chance on flip sensitivity, which confirms the benchmark actually requires memory.

The full-history long-context baseline places every session in the prompt, with and without a step-by-step reasoning instruction. This is the cheapest serious competitor, and it is the test of whether attention alone resolves relational updates. It runs on the same backbone with the same total token budget. To beat it, the proposed method must show a lower stale-interpretation rate on the supersede and salience-distractor conditions with non-overlapping bootstrap confidence intervals.

The semantic-retrieval baseline retrieves the top past turns by embedding similarity to the target, following MapDia's retrieval rule. It shares the same retriever, chunking, and number of retrieved items as our ledger's retrieval step.

The strongest existing method is StateMemWrapper, with the relationship encoded as an ordinary overwritable fact. Can Agent Memory Systems Track Evolving State? reports that StateMem raised current-state accuracy on StateMemBench from 0.205 to 0.363 on a DeepSeek-V4-Flash backbone. It also reports that the wrapper lifted six memory backends by +32 to +67 points. No reported result was found on Lacuna for any method on relational or affective state drift, which is the gap this benchmark fills. We run the wrapper with its published prompt structure, our relation and episode extraction, the same backbone, and the same number of model calls per history. Our method is allowed at most a matched budget increase, which we report. Beating it means equal or better performance on supersede, plus a clear reduction in over-forgetting on the residue condition.

We also run an AMOUR-style continuous-state baseline. It tracks trust and valence per pair with fixed smoothing and clipping and no dependency links. It is tuned on the same development split with the same number of hyperparameter trials as our residue weighting.

Two ablations isolate the key ingredient. "Ledger, delete" uses the same dependency links but drops flagged episodes instead of reappraising them. "Ledger, no links" keeps a current-relation slot but has no episode dependencies. If "Ledger, delete" matches the full method on the residue condition, then reappraisal is not doing the work.

For standard emotion recognition, we report that the ledger does not degrade single-session accuracy. The reference points are ERC-DP at 69.64% weighted F1 on IEMOCAP, 67.34% on MELD, and 40.10% on EmoryNLP (Emotion Recognition in Conversation via Dynamic Personality), and CORECT at 69.93% accuracy on IEMOCAP. For relation tracking on DDRel, the reference point is fine-tuned BERT at about 39.7% pair-level 13-class accuracy against 63.5% for humans (DDRel: A New Dataset for Interpersonal Relation Classification in Dyadic Dialogues).

Why not just…?

Why not just put the whole history in a long context window? StateMem's authors found that drift persists even with all relevant facts present. Models anchor on frequently mentioned values. Relational histories are worse on this axis because the old relationship is typically described at length and with emotional weight, while the change may be one sentence. Still, that evidence comes from transactional facts. If long-context prompting matches our method on the salience-distractor condition, we will report that relational drift is not a distinct problem for strong backbones. The full-history baseline with and without step-by-step reasoning settles this. It also tests whether reasoning helps here, as MapDia found for topic shifts, or hurts, as StateMem found for evolving facts.

Why not just treat the relationship as one more fact in StateMem? This is the real threat, and it may well fix most stale readings on the supersede condition. Our claim rests entirely on the residue and flip-then-revert conditions. In those, discarding earlier affect should produce readings that diverge from human raters, for example reading a recently hurt user's terse reply as neutral because "they reconciled." If the human distributions show little residue, meaning raters themselves read the target almost purely through the current relationship, then plain fact-overwrite is enough. In that case we would reframe the contribution as the benchmark plus the finding that relations can be handled as ordinary state.

Why not just ask the model to first state the current relationship, then interpret? A two-step "relation-first" prompt is nearly free and addresses the most obvious failure. We include it as a prompting baseline at matched call count. We expect it to help on supersede but to inherit the salience trap. The model may name the old relationship when the update was brief. It also does nothing to reinterpret earlier episodes, so it should look like "Ledger, no links."

Why not just fine-tune an emotion-recognition model on longer windows? CORECT's own ablation shows performance peaks at a bounded window, and a utterance window cannot span gaps of days or weeks between sessions. ERC-DP's dynamic state is built from one speaker's own turns, not from the pair. We run ERC-DP's personality-state conditioning over concatenated sessions as an additional check. We do not expect it to separate minimal pairs, because the update turn is a tiny fraction of its input.

Proposed Experiments and Decisive Checks

The primary metrics on RelDrift are three. The stale-interpretation rate is how often the model's reading matches the human reading of the same history with the update removed rather than the actual history. Agreement with the human rating distribution is measured by majority-label accuracy and divergence from the rater distribution. Flip sensitivity is the fraction of minimal pairs where the model's reading changes in the direction humans' readings change. On the residue condition we also report over-forgetting: how often the model's valence moves past the human mean toward the new relationship. Raters' split-half agreement sets a ceiling for every metric.

The decisive check is a two-by-two comparison on supersede and residue. We test whether the full ledger beats StateMemWrapper-as-fact on residue while tying it on supersede, and whether "Ledger, delete" loses that residue advantage. If both hold, dependency-triggered reappraisal is the active ingredient. If the advantage disappears in the deletion ablation's favor, the reappraisal idea is wrong.

The second check is the salience-distractor sweep. We vary how often the old relationship is mentioned (one, three, and ten times) against a single update. We plot stale-interpretation rate for each method, which shows whether structured state removes the dependence on repetition that long-context prompting is expected to show.

The third check is the reasoning tension. We run every prompting method with and without step-by-step reasoning to see whether verbal reasoning rationalizes the stale relationship, as StateMem reports for facts.

Finally, we run the ledger on the DDRel re-annotation to test transfer from controlled histories to naturally written ones. We also run it on IEMOCAP and MELD to confirm that single-session emotion recognition stays within about a point of the ERC-DP figures above. Together these show that the gain is about tracking change, not about sacrificing ordinary emotion recognition.