Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents

Haojin Yang, Ai Jian, Xinyue Huang, Yiwei Wang, Weipeng Zhang, Ke Zeng, Xunliang Cai, Jingqing Ruan

Abstract

Optimizing large language models for industrial sales requires balancing long-term commercial objectives (e.g., conversion rate) with immediate linguistic constraints such as fluency and compliance. Conventional reinforcement learning often merges these heterogeneous goals into a single reward, causing high-magnitude session-level rewards to overwhelm subtler turn-level signals, which leads to unstable training or reward hacking. To address this issue, we propose Dual-Horizon Credit Assignment (DuCA), a framework that disentangles optimization across time scales. Its core, Horizon-Independent Advantage Normalization (HIAN), separately normalizes advantages from turn-level and session-level rewards before fusion, ensuring balanced gradient contributions from both immediate and long-term objectives to the policy update. Extensive experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline, achieving a 6.82% relative improvement in conversion rate, reducing inter-sentence repetition by 82.28%, and lowering identity detection rate by 27.35%, indicating a substantial improvement for an industrial sales scenario that effectively balances the dual demands of strategic performance and naturalistic language generation.

Introduction

Deploying Large Language Models (LLMs) in professional sales environments presents a unique temporal credit assignment challenge. Agents must balance dense, turn-level signals—such as fluency, non-repetition, and regulatory compliance—with sparse, session-level signals, primarily the ultimate goal of conversion (a sale).

Standard Reinforcement Learning (RL) approaches often conflate these heterogeneous objectives into a single reward function. This typically leads to "gradient dominance," where high-magnitude, high-variance session-level rewards overwhelm subtle conversational signals. The result is often an agent that either ignores linguistic nuances to chase a sale or falls into "reward hacking"—adopting robotic or overly sycophantic behavior to maximize safety rewards while failing to persuade the customer.

Methodology: Dual-Horizon Credit Assignment (DuCA)

To address these conflicting time scales, we introduce DuCA (Dual-Horizon Credit Assignment). The framework disentangles optimization across different horizons through three primary components:

  1. Independent Advantage Estimation: The model utilizes separate value heads for turn-level and session-level objectives, allowing for distinct temporal dynamics and discount factors tailored to short-term vs. long-term goals.
  2. Horizon-Independent Advantage Normalization (HIAN): This core innovation normalizes advantages for turn-level and session-level rewards separately within each mini-batch before fusion. This ensures balanced gradient contributions, preventing sparse signals from suppressing the learning of essential linguistic skills. Mathematically, this prevents the effective gradient from being scaled solely by the total variance, which is often dominated by sparse rewards.
  3. High-Fidelity User Simulator: A simulator based on a powerful LLM, conditioned on diverse customer "Personas" (e.g., price-sensitive, skeptical), provides a safe and dynamic training environment that mimics real-world behavior.
The DuCA framework architecture, illustrating the disentanglement of turn-level and session-level advantages.

Experimental Setup

The study's data pipeline is grounded in real-world industrial contexts. It utilizes 31,000 anonymized real-world business sales dialogues as a base, with 10,000 high-quality online samples used to fine-tune the LLM-based user simulator. Evaluation is conducted on a held-out test set of multi-turn trajectories via "LLM-as-a-Judge" and senior professional auditors. Notably, in a blind test, senior auditors identified 52% of actual human interactions as being generated by the simulator, validating the high fidelity of the training environment.

Key Results and Discussion

DuCA was benchmarked against several baselines, including GRPO (Group-wise Relative Policy Optimization). As shown in the results table, DuCA achieves superior performance across both commercial and conversational metrics.

Performance comparison of DuCA against foundation models and various training methods across multiple metrics.

Key findings include:

  • Conversion Rate (CVR): A 6.82% relative improvement over the strongest baseline (GRPO).
  • Conversational Quality: A staggering 82.28% reduction in inter-sentence repetition, effectively breaking the "repetition loop" common in standard RL.
  • Human-likeness: The "Identity Detection Rate" (IDR) was lowered by 27.35% compared to other independent normalization methods, indicating more natural interactions.
  • Training Stability: DuCA maintained stable growth in both compliance and conversion, avoiding the "catastrophic collapse" seen in methods like REINFORCE++.

Conclusion

DuCA effectively harmonizes the dual demands of strategic sales performance and naturalistic language generation. By decoupling reward horizons, it prevents the strategic "sale" signal from degrading conversational quality. Future work will address the heavy dependency on high-fidelity simulators and explore methods to mitigate potential bias amplification in synthetic agents.

Summary updated

Abstract

Optimizing large language models for industrial sales requires balancing long-term commercial objectives (e.g., conversion rate) with immediate linguistic constraints such as fluency and compliance. Conventional reinforcement learning often merges these heterogeneous goals into a single reward, causing high-magnitude session-level rewards to overwhelm subtler turn-level signals, which leads to unstable training or reward hacking. To address this issue, we propose Dual-Horizon Credit Assignment (DuCA), a framework that disentangles optimization across time scales. Its core, Horizon-Independent Advantage Normalization (HIAN), separately normalizes advantages from turn-level and session-level rewards before fusion, ensuring balanced gradient contributions from both immediate and long-term objectives to the policy update. Extensive experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline, achieving a 6.82% relative improvement in conversion rate, reducing inter-sentence repetition by 82.28%, and lowering identity detection rate by 27.35%, indicating a substantial improvement for an industrial sales scenario that effectively balances the dual demands of strategic performance and naturalistic language generation.
BibTeX
@inproceedings{yang2026harmonizing,
  author = {Haojin Yang and Ai Jian and Xinyue Huang and Yiwei Wang and Weipeng Zhang and Ke Zeng and Xunliang Cai and Jingqing Ruan},
  title = {{Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.74},
}
Figure