Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents

Haojin Yang, Ai Jian, Yiwei Wang, Xinyue Huang, Weipeng Zhang, Ke Zeng, Xunliang Cai, Jingqing Ruan

TL;DR

Training AI sales agents is a balancing act between immediate conversational quality and long-term business goals. This paper introduces DuCA (Dual-Horizon Credit Assignment), a reinforcement learning framework that prevents high-stakes, sparse rewards (like closing a sale) from drowning out subtle, dense signals (like being polite and fluent), resulting in more effective and human-like industrial agents.

What Problem Does This Paper Address?

Deploying Large Language Models (LLMs) in professional sales is a "high-stakes" challenge. Unlike a general-purpose chatbot, a sales agent must navigate a temporal credit assignment dilemma.

On one hand, you have dense, turn-level signals: the agent needs to be fluent, non-repetitive, and compliant with regulations at every step. On the other hand, you have sparse, session-level signals: the ultimate goal is conversion (a sale), which only happens at the very end of a long conversation.

Standard Reinforcement Learning (RL) typically merges these goals into a single reward function. However, the high-magnitude, high-variance "sale" reward often overwhelms the subtle "fluency" rewards. This leads to gradient dominance, where the model either ignores conversational nuances to chase the sale or falls into reward hacking—becoming robotic or sycophantic just to stay "safe" and compliant while failing to actually persuade the customer.

Methods

The authors propose DuCA (Dual-Horizon Credit Assignment) to disentangle these conflicting time scales. The framework consists of three main components:

  1. Independent Advantage Estimation: Instead of one value head, the model maintains two: one for turn-level objectives and one for session-level objectives. This allows for tailored temporal dynamics (e.g., using different discount factors for short-term vs. long-term goals).
  2. Horizon-Independent Advantage Normalization (HIAN): This is the core innovation. Before fusing the signals, DuCA normalizes the advantages for the turn-level and session-level rewards separately within each mini-batch. This ensures that the gradients from both horizons contribute equally to the policy update, preventing the sparse "sale" signal from suppressing the learning of linguistic skills.
  3. High-Fidelity User Simulator: To train the agent safely, the authors built a simulator using a powerful LLM conditioned on specific "Personas" (e.g., price-sensitive, skeptical). This creates a dynamic environment that mimics real-world customer behavior better than static datasets.

Datasets

The study utilizes a robust data pipeline:

  • Real-world Foundation: 31,000 anonymized real-world business sales dialogues served as the base.
  • Simulator Training: 10,000 high-quality online samples were used to fine-tune the LLM-based user simulator.
  • Evaluation Set: A held-out test set of multi-turn trajectories was used for the final "LLM-as-a-Judge" evaluation.

Key Insights and Ideas

  • The Math of Gradient Suppression: The authors provide a theoretical justification for HIAN, showing that in naive scalarization, the effective gradient is scaled by total variance. Since sparse rewards have massive variance, the turn-level signal effectively drops to zero, halting the improvement of conversational skills.
  • Breaking the "Repetition Loop": A common failure mode in standard RL is the "repetition loop," where an agent repeats safe, compliant phrases to maximize local rewards. By decoupling the horizons, DuCA allows the agent to explore strategic moves without sacrificing local fluency.
  • The "Inverse" Turing Test: In a blind test, senior auditors identified 52% of actual human interactions as being generated by the simulator. This suggests the simulator captured the "gold standard" of professional sales behavior so well that it became the benchmark for the auditors.

Key Results

DuCA was tested against several baselines, including the state-of-the-art GRPO (Group-wise Relative Policy Optimization). The results were significant:

  • Conversion Rate (CVR): Achieved a 6.82% relative improvement over the strongest baseline.
  • Conversational Quality: Reduced inter-sentence repetition by a staggering 82.28%.
  • Human-likeness: Lowered the "Identity Detection Rate" (how easily a judge can tell it's an AI) by 27.35% compared to other independent normalization methods.
  • Stability: Training curves showed that DuCA maintained a stable upward trend in both compliance and conversion, whereas baselines like REINFORCE++ often suffered from "catastrophic collapse" or strategic stagnation.

Limitations and Future Work

The authors identify a few hurdles for broader adoption:

  • Data Dependency: The success of DuCA relies heavily on having a high-fidelity simulator, which in turn requires a massive amount of high-quality, domain-specific real-world data and professional auditing.
  • Domain Specificity: While effective for sales, the "dual-horizon" weights might need significant tuning for other fields (e.g., medical advice or technical support).
  • Ethical Risks: The authors acknowledge the risk of "bias amplification" from synthetic agents and the potential for AI to subtly manipulate user preferences without explicit consent. They suggest that future work should focus on more robust "compliance scoring" to mitigate these risks.

Summary updated

Abstract

Optimizing large language models for industrial sales requires balancing long-term commercial objectives (e.g., conversion rate) with immediate linguistic constraints such as fluency and compliance. Conventional reinforcement learning often merges these heterogeneous goals into a single reward, causing high-magnitude session-level rewards to overwhelm subtler turn-level signals, which leads to unstable training or reward hacking. To address this issue, we propose Dual-Horizon Credit Assignment (DuCA), a framework that disentangles optimization across time scales. Its core, Horizon-Independent Advantage Normalization (HIAN), separately normalizes advantages from turn-level and session-level rewards before fusion, ensuring balanced gradient contributions from both immediate and long-term objectives to the policy update. Extensive experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline, achieving a 6.82% relative improvement in conversion rate, reducing inter-sentence repetition by 82.28%, and lowering identity detection rate by 27.35%, indicating a substantial improvement for an industrial sales scenario that effectively balances the dual demands of strategic performance and naturalistic language generation.
BibTeX
@inproceedings{yang2026harmonizing,
  author = {Haojin Yang and Ai Jian and Yiwei Wang and Xinyue Huang and Weipeng Zhang and Ke Zeng and Xunliang Cai and Jingqing Ruan},
  title = {{Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.74},
}

Loading related content...

Figure