Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use

Zixiang Chen, Wenting Zhao, Zhepeng Cen, Akshara Prabhakar, Jielin Qiu, Jianguo Zhang, Zhiwei Liu, Tulika Manoj Awalgaonkar, Liangwei Yang, Shelby Heinecke, Silvio Savarese, Huan WangarXiv 2026 · arXiv

TL;DR

When LLM agents fail during complex, multi-turn tool use, trajectory-level rewards cannot pinpoint which specific turn caused the failure. Critical-State RL introduces a training-free diagnostic that uses nested sampling to isolate action-dependent reward variance from downstream noise, identifying the exact model call that will benefit from training. By optimizing only these diagnosed "critical states" via localized contextual-bandit training, the method significantly boosts agent performance while avoiding the degradation caused by training the wrong turns.

What Problem Does This Paper Address?

In multi-turn interactions—such as an LLM agent calling multiple tools to solve a user request—errors often hinge on a single bad decision. However, standard Reinforcement Learning (RL) struggles with credit assignment in these scenarios. If you only have a final trajectory-level reward (success or failure), how do you know which turn to blame or train?

Simply looking at reward variation across rollouts isn't enough. If an agent makes a decision at turn 2, the final outcome still depends on random variations or choices made in turns 3, 4, and 5 (continuation noise). As a result, a good action might end in failure, or a bad action might be bailed out later. Standard RL methods either update every turn indiscriminately (adding massive noise) or rely on rigid heuristics that target the wrong states, which can stall or even degrade model performance.

Gradient placement comparison between trajectory-level RL, fixed-role local RL, and Critical-State RL.

Methods

The authors propose a two-step framework: Diagnose where to train, then Optimize locally.

  1. The Critical-State Diagnostic (Training-Free): Before updating any weights, the framework evaluates candidate turns based on three criteria: action-sufficiency (the local reward aligns with final task success), headroom (the current policy has room to improve), and trainability. To measure trainability, they use nested within-prefix sampling:

  2. Fix the history (prefix) up to the candidate turn.

  3. Sample multiple alternative actions at this turn.
  4. For each action, resample multiple downstream continuations.

By applying the law of total variance, this nested sampling mathematically separates the variance caused by the current action from the downstream continuation noise. Turns with high action-dependent variance are selected as "critical states."

  1. Occurrence-Local RL: Once a critical turn is selected, the policy is optimized only at that specific call using contextual-bandit training. Gradients are applied strictly to the tokens generated during this turn, while the surrounding turns merely act as frozen context or reward providers.
Figure 2. Task structure supplies the candidate phase and occurrence-local label; policy rollouts then measure headroom and action-dependent variance before local training. The BFCL four-cell study in §4 tests this measured selection step in Gemma; App. D.2 presents a contrasting label construction in a memory application.

Datasets

The method was evaluated across several models and multi-turn configurations:

  • Berkeley Function Calling Leaderboard (BFCL) v4 (multi_turn suite): Testing specific failure modes including miss_func (withholding a required tool until later turns) and miss_param (withholding a required argument).
  • Logged Interaction Traces: Real-world interaction logs from Nemotron-Super-120B evaluated for repeat-call avoidance.
  • Memory Management Tasks: Evaluated on xLAM and Gemma models using the BFCL v4 agentic/memory sub-tasks (Key-value store, Vector store, and Recursive summary).

Key Insights and Ideas

  • Variance $\neq$ Trainability: High reward variance at a specific turn does not mean that turn is trainable. The variance could be entirely driven by downstream randomness.
  • Critical States are Model- and Task-Dependent: There is no universal "correct turn" to train. For instance, on the missing-function task (miss_func), the diagnostic determined that Gemma needed training on its recovery turn (after the tool becomes available), whereas Nemotron benefited from training on its decision turn (before tool availability).
  • Training the Wrong Turn Destroys Performance: Blindly applying local RL to non-critical turns can actively harm the model. For example, training Gemma on the decision turn for miss_func caused a 4.5 percentage point drop in accuracy because it forced the model to over-bias toward firing tools prematurely.

Key Results

The authors conducted a controlled four-cell study using Gemma-4-26B-A4B to validate their diagnostic choices against alternative turns:

Table 1. Base-policy diagnostics and training outcomes for Critical-State RL on BFCL v4 multi_turn.
  • Missing-Function (miss_func): Training the diagnostic-selected recovery turn improved BFCL accuracy from 14% to 28.3% (+14.3 pp), while training the alternative decision turn degraded performance.
  • Missing-Argument (miss_param): Training the selected decision turn improved accuracy from 43.5% to 47.3% (+3.8 pp), while the alternative turn left performance flat.
  • Comparison to Baselines: Critical-State RL outperformed standard multi-turn RL using Monte-Carlo Return-to-Go (RTG), which only achieved 45% accuracy on miss_param.
Table 2. Monte-Carlo return-to-go (RTG) comparison on the non-saturated miss_param cell.
  • Memory Management: On the xLAM memory sub-task, the agent's mean score jumped from 34.54% to 50.54%. On Gemma, it rose from 38.1% to 52.7%.

Limitations and Future Work

  • Heuristic Candidate Phases: The framework still requires humans or task structures to define the initial "candidate phases" and local reward functions before the diagnostic can rank them.
  • Simplified Recurrence ($K=1$): The experimental validation focused primarily on settings where the critical state occurs once per episode ($K=1$). Extending this to highly recurrent states within a single trajectory remains an open area for deeper empirical exploration.
  • Threshold Calibration: The study evaluated categorical turn selection (choosing turn A over turn B) rather than establishing absolute, calibrated variance thresholds for when to trigger training across arbitrary tasks.

Summary updated

Abstract

Multi-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions. We introduce Critical-State RL to identify trainable states in multi-turn interactions. Given task-defined candidate calls and local rewards, the method assesses whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible. It then uses nested sampling to separate action-dependent reward variation from continuation noise and optimizes the policy at the selected states using contextual-bandit training. Experiments on the Berkeley Function Calling Leaderboard (BFCL) v4 compare training at diagnostic-selected states with training at alternative states. For missing-function tasks, the diagnostic selects the response after the tool becomes available; for missing-argument tasks, it selects the response before the missing argument is supplied. Training the selected responses improves performance, including about 14 percentage points on the missing-function task, while training the alternatives leaves performance flat or worse. We further apply the recipe across models and tasks, including logged repeat-call avoidance and memory management.
BibTeX
@misc{chen2026critical,
  author = {Zixiang Chen and Wenting Zhao and Zhepeng Cen and Akshara Prabhakar and Jielin Qiu and Jianguo Zhang and Zhiwei Liu and Tulika Manoj Awalgaonkar and Liangwei Yang and Shelby Heinecke and Silvio Savarese and Huan Wang},
  title = {{Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use}},
  year = {2026},
  eprint = {2609.24985},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.24985},
}
Figure