Can Agents Relive Past Experiences? Diagnosing Event-Affect Decoupling in Affective-Episodic Retrospection

Tingna Xie, Ming Wang, Shi Feng, Yongkang LiuarXiv 2026 · arXiv

TL;DR

This paper identifies Event-Affect Decoupling, a critical failure mode where AI agents recall the correct emotion from a past experience but link it to the wrong event. To solve this, the authors introduce EM-TRACE, a diagnostic framework that evaluates whether an agent's emotional response is truly grounded in the specific episode it claims to remember.

What Problem Does This Paper Address?

As we move toward AI assistants with long-term memory, we expect them to "relive" past experiences with us. If you ask, "How did I feel when I won my first award?", you don't just want a generic "happy"; you want the agent to recall the specific joy associated with that event.

Current evaluations often treat factual memory and emotional accuracy as two separate checkboxes. The authors argue this misses a "blind spot": Event-Affect Decoupling (EAD). As illustrated in Figure 1, an agent might correctly identify that you felt "joy" but mistakenly attribute it to "passing a math test" instead of "winning an award." The emotion is correct, but the "binding" to the event is broken.

Figure 1: An emotion-correct response can rest on the wrong stored experience.

The paper formalizes this challenge as Affective-Episodic Retrospection (AER)—the task of recovering a past event, its associated affect, and the correct correspondence between them.

Methods

The authors propose EM-TRACE, a stage-wise diagnostic framework that breaks down the recall process into three distinct steps:

  1. Episode Retrieval: Selecting the candidate memory from a bank.
  2. Affect Recovery and Assignment: Determining the emotion associated with that specific retrieved memory.
  3. Emotion Realization: Generating a natural language response that expresses both the event and the emotion.
Figure 2: EM-TRACE decomposes episodic emotion recall into Episode Retrieval, Affect Recovery and Assignment, and Emotion Realization, with oracle interventions and diagnostic signals for stage-wise analysis.

To pinpoint exactly where failures occur, they use "Oracle Interventions." By swapping in ground-truth events or emotions at different stages, they can see if a failure is due to bad retrieval, bad affect assignment, or poor generation. They also introduce two key metrics:

  • Binding@1: How often both the event and the emotion are recovered correctly.
  • Ungrounded Affect Rate (UAR): How often the agent gets the emotion right but attaches it to the wrong episode.

Datasets

The researchers constructed a controlled benchmark using GoEmotions, a dataset of Reddit comments with fine-grained emotion labels.

  • Memory Bank: 8,030 stored experiences. They used LLMs to strip away emotional wording from the original text to create "factual memories."
  • Queries: 1,103 test queries generated by GPT-5.6-Terra based on the original emotional texts.
  • Categories: Focused on 11 distinct emotion categories (e.g., admiration, joy, sadness, embarrassment).

Key Insights and Ideas

  • Emotion Accuracy is a Deceptive Metric: A system can have high emotion accuracy while being completely ungrounded. In their "Emotion-only" retrieval tests, the system achieved 28.65% affect recovery but retrieved the correct episode only 4.44% of the time. This results in a high Ungrounded Affect Rate (UAR), meaning the agent is "right for the wrong reasons."
  • Retrieval and Assignment are Distinct Bottlenecks: Even if an agent finds the perfect memory (successful retrieval), it can still fail to correctly identify the emotion associated with it (failed assignment). Conversely, a system might "hallucinate" the right emotion onto a wrong event.
  • Generation Control Doesn't Fix Grounding: Using advanced techniques like "Multilayer Steering" or "NPTI" to make the agent sound more emotional improves the expression of affect, but it doesn't magically fix the underlying memory errors. In fact, stronger emotional expression can sometimes lead to lower factual entailment, making the agent sound more confidently incorrect.

Key Results

The study compared five retrieval strategies (Fact-only, Emotion-only, etc.) and found that Fact-only retrieval performed best for grounding but still suffered from a 9.7% UAR.

Figure 3: Retrieval and Event-Affect Binding results across five strategies. Panel (a) shows metric estimates with 95% bootstrap confidence intervals, and panel (b) shows the four event-affect outcome proportions.

The "Oracle" analysis (Figure 5) provided the most striking evidence of the decoupling:

  • Correcting the emotion source significantly boosted emotion accuracy (up to +7.8 percentage points) but did almost nothing for factual grounding.
  • Correcting the event source significantly improved factual support (original-text entailment) but had a negligible effect on emotion accuracy.

This confirms that event and affect errors are largely independent bottlenecks in current architectures.

Figure 5: Mean gains over the Normal setting for different oracle interventions. Results are reported as absolute changes in percentage points (pp).

Limitations and Future Work

  • Controlled vs. Natural: The benchmark uses pre-built records from Reddit (GoEmotions). Real-world memory is dynamic, involving memory formation, updating, and evolving contexts which weren't captured here.
  • Discrete Emotions: The study treats affect as discrete categories (e.g., "Joy"). Human emotion is often continuous, mixed, and highly dependent on subtle context.
  • Semantic Consistency: The current metrics (Binding@1 and UAR) rely on record identity. Future work could use more fine-grained human evaluations to assess the semantic consistency between a recalled experience and the generated response.

In summary, the paper warns that as we build more "empathetic" agents, we must ensure their empathy is rooted in reality, not just a lucky guess at a label.

Summary updated

Abstract

As memory-augmented agents increasingly support long-term and personalized interactions, reliable recall of emotional experiences while preserving their associated affect becomes necessary. However, existing evaluations usually assess emotion accuracy and factual memory grounding separately, overlooking event-affect correspondence. In this work, we formulate Affective-Episodic Retrospection (AER) as a task that explicitly evaluates the recovery of past events, associated affects, and their correspondence, and define Event-Affect Decoupling as a failure mode where the correct emotion is recalled but associated with the wrong episode. To diagnose Event-Affect Decoupling, we propose EM-TRACE, a stage-wise framework that decomposes AER into retrieval, affect recovery, and realization. On a controlled GoEmotions-based benchmark, Emotion-only retrieval achieves 28.65% affect recovery but retrieves the target episode for only 4.44% of queries, yielding a UAR of 24.21%. Further analysis reveals that event retrieval and affect assignment introduce distinct binding failures. These findings show that reliable AER requires grounded recall of emotional experiences, where episode retrieval, affect recovery, event-affect binding, and generation quality should be evaluated jointly. Our code is available at https://anonymous.4open.science/r/A412/.
BibTeX
@misc{xie2026can,
  author = {Tingna Xie and Ming Wang and Shi Feng and Yongkang Liu},
  title = {{Can Agents Relive Past Experiences? Diagnosing Event-Affect Decoupling in Affective-Episodic Retrospection}},
  year = {2026},
  eprint = {2611.15655},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15655},
}
Figure