Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning

Jing Christine Ye, Xinpei Zhao, Lu Xiang, Yaping Zhang, Chengqing Zong

TL;DR

This paper introduces RAPO, a novel reinforcement learning framework for emotional support dialogue systems that moves beyond static, expert-defined rewards. By directly learning from simulated user reactions and providing fine-grained natural language feedback, RAPO significantly improves the quality and personalization of AI-driven emotional support, leading to more effective and empathetic interactions.

What Problem Does This Paper Address?

Current AI systems designed for emotional support conversations (ESC) often struggle to provide genuinely helpful and personalized assistance. They typically rely on expert-defined "scalar rewards" to guide their learning, which are essentially numerical scores assigned to responses based on predefined rubrics. However, these scalar rewards suffer from two major limitations:

  1. Information Sparsity: They provide very little detail about why a response failed or how to improve it. A low score doesn't explain what went wrong.
  2. Optimization Mismatch: Expert rubrics might not perfectly align with a user's actual experience. A response that scores high on "empathy" according to an expert might still feel hollow or unhelpful to the person seeking support, leading to "hallucinated empathy."

These limitations prevent ESC systems from truly adapting to dynamic user states and achieving the ultimate goal of facilitating positive emotional shifts. The authors argue that the most direct and reliable learning signal comes from the user's continuous reactions during the conversation itself.

Methods

The paper proposes Reaction Aware Policy Optimization (RAPO), a framework that optimizes dialogue policies based on user reactions rather than just rubric scores. RAPO achieves this through three core components:

  1. Hindsight Dialogue Selection (HDS): Instead of training on every turn of a dialogue, HDS identifies "pivotal turns" – those interactions that significantly influence the user's emotional trajectory. This helps the model focus on learning from impactful moments and avoid overfitting to generic, low-information empathy templates.
  2. Generative Hindsight Feedback (GHF): This component uses a simulated user and a Generative Reward Model (GRM) to create dense, natural-language feedback.
  3. User Simulation: A simulated user provides reactions to candidate responses generated by the policy.
  4. Contrastive Critique Generation: The GRM analyzes a group of simulated trajectories (dialogue context, candidate response, user reaction) and generates both a relative ranking score for each response and natural-language critiques. This contrastive approach helps overcome the ambiguity of isolated scores and provides detailed feedback.
  5. Scalar-Verbal Hybrid Policy Optimization: This is the core learning mechanism. It combines:
  6. Scalar RL (using GRPO): Optimizes the policy for the relative ranking scores provided by the GRM, ensuring global alignment with desired outcomes.
  7. Verbal RL (using On-Policy Distillation): Distills the natural-language critiques from the GRM into the policy. This allows for fine-grained semantic refinement, teaching the model how to improve specific aspects of its responses based on detailed feedback.

Datasets

  • ESConv: Used for training and evaluating emotional support conversations. The authors also introduce a subset derived from ESConv using Hindsight Dialogue Selection (HDS), which focuses on critical turns.
  • ESC-Eval: A benchmark for evaluating emotional support quality beyond simple lexical metrics, featuring dynamic role-playing scenarios and multi-dimensional evaluation.
  • EmoHarbor: An automated evaluation framework that simulates users with diverse profiles to assess personalized emotional support.
  • SOTOPIA: A multi-agent social intelligence framework used to evaluate the generalizability of RAPO in complex social negotiation and interaction scenarios.

Key Insights and Ideas

  • User Reactions are Gold: The paper strongly advocates for shifting from static, expert-defined rubrics to dynamic, user-reaction-driven learning signals. User reactions are presented as a more direct, information-dense, and reliable proxy for supportiveness.
  • Beyond "Hallucinated Empathy": By focusing on actual user responses and providing detailed verbal feedback, RAPO aims to move beyond generating responses that look empathetic according to a rubric but fail to resonate with the user.
  • The Power of Contrastive Feedback: Using a GRM to rank responses within a group, rather than assigning absolute scores, leads to more stable and informative learning signals.
  • Verbal Feedback for Fine-Grained Control: Natural language critiques from the GRM are crucial for teaching the model how to improve, addressing specific linguistic and strategic shortcomings that scalar rewards cannot capture.
  • Hindsight Dialogue Selection is Key: Focusing training on pivotal turns, rather than all turns, significantly improves efficiency and effectiveness by filtering out low-impact interactions.

Key Results

  • Superior Performance: RAPO consistently outperforms strong RL baselines (like GRPO and CPO) on both emotional support (ESConv, EmoHarbor, ESC-Eval) and social intelligence (SOTOPIA) benchmarks.
  • Significant Gains: For example, Qwen-2.5-RAPO improved over GRPO by 12.4% on EmoHarbor.
  • Robustness: RAPO maintains strong semantic consistency with human reference responses, as evidenced by high BERT-Score values.
  • Generalizability: The framework shows promise beyond ESC, demonstrating effectiveness in complex social interaction scenarios in SOTOPIA.
  • Ablation Studies Confirm Value: Removing either the User Reaction Modeling (URM) or the Self-Distillation (SD) components led to significant performance drops, highlighting the complementary nature of these mechanisms.

Limitations and Future Work

  • Computational Overhead: The use of an LLM-based user simulator and GRM introduces significant computational costs, leading to longer training times compared to simpler methods.
  • Dependency on Heavy Reward Models: The reliance on advanced LLMs as judges poses challenges for resource-constrained environments. Future work could explore distilling knowledge from these large judges into smaller, specialized models.
  • Simulator Sensitivity: The impact of different user model architectures on RAPO's performance was not exhaustively studied. Future research could explore the robustness of RAPO across various simulator backbones.
  • Domain Generalization: Current validation is limited to dialogue-based tasks. Scaling RAPO to broader sequential decision-making domains remains an open question.

Summary updated

Abstract

While current emotional support dialogue systems typically rely on expert-defined scalar rewards for alignment, these signals suffer from severe information sparsity. They cannot explain why a response failed or how to adapt to dynamic user states, often diverging from the actual goal of facilitating positive emotional shifts. In practice, the most direct and reliable learning signal emerges from the user's continuous reactions during ongoing interaction. We therefore propose Reaction Aware Policy Optimization (RAPO), a framework that optimizes over interaction consequences rather than rubric scores. RAPO treats dialogue as a reaction-driven process and utilizes simulated user responses to generate dense natural-language feedback through three core components: Hindsight Dialogue Selection, which isolates pivotal turns that meaningfully alter user emotional trajectories; Generative Hindsight Feedback, which transforms user reactions into contrastive ranking signals and natural-language critiques; and Scalar-Verbal Hybrid Policy Optimization, which couples scalar reward optimization for global alignment with verbal feedback distillation for fine-grained semantic refinement. Extensive experiments on ESC and Sotopia demonstrate that RAPO significantly outperforms strong reinforcement learning baselines in driving positive interaction outcomes.
BibTeX
@misc{ye2026listening,
  author = {Jing Christine Ye and Xinpei Zhao and Lu Xiang and Yaping Zhang and Chengqing Zong},
  title = {{Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning}},
  year = {2026},
  eprint = {2603.15434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.15434},
}

Loading related content...

Figure