ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agents

Lei Ding, Bin He, Chenguang Wang, Yang Liu

TL;DR

This paper introduces ProActor, a novel framework for training AI agents to proactively schedule tasks in conversations. By developing an automated annotation pipeline, new proactiveness metrics, and an efficient RL training system (ART-F), ProActor significantly improves an agent's ability to anticipate user needs and act at the optimal moment, without disrupting the conversation flow.

What Problem Does This Paper Address?

Modern AI assistants are moving beyond simply reacting to user commands. The goal is to create "proactive" agents that can anticipate user needs, identify opportunities, and initiate actions before being explicitly asked. This is particularly challenging in conversational task scheduling, where timing is crucial. Existing methods struggle to:

  • Measure and optimize proactivity: It's hard to define and quantify "good" proactive behavior, especially when there are multiple valid timings for an action.
  • Generate training data at scale: Manually annotating proactive opportunities and their ideal timing is labor-intensive and doesn't scale.
  • Balance timing with accuracy: Agents that are too eager to act might miss crucial information or make mistakes, while those that wait too long lose the proactive advantage.
  • Train large models efficiently: Reinforcement learning (RL) for large language models (LLMs) is computationally expensive.

Methods

ProActor tackles these challenges with a three-pronged approach:

  1. Automated Annotation Pipeline: This system generates "reference actions" and their "opportunity time windows" (instead of rigid point labels). It's domain-agnostic, meaning it can be adapted to different task scheduling scenarios.
  2. Proactiveness Metrics: The paper introduces new metrics to quantify both the timing quality of an agent's actions and their alignment with reference actions. This includes metrics for action consistency and various aspects of proactive timing (like fault trigger rates).
  3. Reinforcement Learning Optimization (GRPO with Composite Rewards): ProActor uses a turn-level RL approach with a novel reward design. They found that combining metrics like action consistency (RAC) with RULER-based rewards (which use rubrics to evaluate proactivity) is key. Stage-aware composite rewards help balance timing and consistency.
  4. Efficient Training Framework (ART-F): To address the computational cost of RL training for LLMs, they developed ART-F. This framework optimizes GPU utilization by dynamically managing inference clusters and using asynchronous training, enabling significant speedups (4-8x) for training large models.

Datasets

The paper introduces two newly auto-annotated datasets:

  • ABCD+: An extension of the ABCD dataset, incorporating proactive annotations for customer support scenarios. This dataset includes historical action execution logs, allowing for validation of annotation quality.
  • Home Loan: A dataset derived from mortgage consultation transcripts, representing domains without observed software action triggers. This dataset is crucial for evaluating agents in scenarios where proactive timing must be inferred purely from conversation context.

Key Insights and Ideas

  • Timing is as important as action: For proactive agents, when an action is taken is as critical as what action is taken.
  • Automated annotation is key for scale: Manually labeling proactive opportunities is a bottleneck; an automated, domain-agnostic pipeline is essential for training robust agents.
  • Composite rewards are crucial for balance: Simply optimizing for action consistency or timing alone leads to suboptimal or polarized behaviors. Combining these through carefully designed rewards is necessary.
  • Turn-level rewards are more stable: While trajectory-level rewards can offer better performance in some cases, turn-level rewards provide more stable training, especially with large context windows that can cause out-of-memory issues.
  • Efficient training infrastructure matters: ART-F demonstrates that optimizing GPU utilization through adaptive inference and asynchronous training can dramatically speed up RL for LLMs.

Key Results

  • ProActor achieves significant improvements in proactive timing while maintaining action consistency comparable to state-of-the-art baselines.
  • The ART-F framework enables 4-8x training speedups for large, quantized LLMs (e.g., Qwen2.5-14B-ProActor-Q4) on multi-GPU systems.
  • Ablation studies validate the effectiveness of their stage-aware composite reward designs.
  • ProActor models demonstrate a better balance between action consistency and proactive timing, avoiding the polarized failure modes seen in prompting-based baselines.

Limitations and Future Work

  • Action Observation Scope: Current datasets primarily rely on observed triggers. Future work aims to incorporate human annotation protocols to capture a broader range of valid proactive opportunities, including those that might not have been triggered in the original dialogue.
  • Multilingual Evaluation: The current experiments focus on English-language datasets. Extending ProActor to other languages is a natural next step, though it may introduce challenges related to data sparsity and language-specific conversational norms.
  • Model and Training Constraints: While ART-F is model-agnostic, experiments were primarily conducted on 4-bit quantized LLMs. Future work will explore generalization to other model families, scales, and quantization schemes.
  • Longer Conversations: The current framework limits training to dialogues with up to 50 turns; performance on longer conversations remains to be evaluated.
  • Ethical Considerations: The authors emphasize the need for rigorous fairness testing and human-in-the-loop oversight before real-world deployment, especially in sensitive domains, to mitigate potential biases and ensure user consent.

Summary updated

Abstract

Proactive task-oriented agents must autonomously anticipate user needs, identify actionable opportunities, and trigger software actions at appropriate moments—fundamentally shifting from reactive systems that await explicit instructions. However, existing approaches lack generalizable end-to-end solutions for measuring and optimizing such anticipatory behaviors. This paper introduces ProActor, a unified framework for conversational task scheduling that integrates: (1) a domain-agnostic automated annotation methodology that enables scalable proactiveness reinforcement learning (RL) by generating full opportunity time windows instead of rigid point labels, (2) systematic proactiveness metrics capturing both timing quality and reference action alignment, and (3) RL optimization using GRPO with various reward designs. Our insight is that RULER-based rewards with proactiveness rubrics are crucial for improving timing quality, and that proactiveness optimization enabled by stage-aware composite rewards is key to balancing timing quality and reference action alignment. Furthermore, we introduce ART-F, an adaptive RL framework that combines request-adaptive inference clusters with asynchronous training for better GPU utilization, enabling LoRA training of 4-bit Qwen2.5-14B-ProActor-Q4 models on 4×H200 and 8×H100 GPUs with substantial speedups. Experiments on two newly auto-annotated datasets demonstrate significant improvements in proactive timing while maintaining action consistency comparable to state-of-the-art baselines. Ablations validate the effectiveness of distinct composite reward variations.
BibTeX
@inproceedings{ding2026proactor,
  author = {Lei Ding and Bin He and Chenguang Wang and Yang Liu},
  title = {{ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agents}},
  booktitle = {AIWILD @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=k5BQhj9Qrz},
}
Figure