Success-Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success

Daniel RussoICML 2026 · OpenReview

TL;DR

This paper reveals that a common, heuristic policy improvement technique called "success conditioning" (used in areas like LLM alignment and RL) is actually an exact solution to a specific type of trust-region optimization problem. This provides a principled understanding of what success conditioning optimizes for, showing it's a conservative approach that balances policy improvement with stability.

What Problem Does This Paper Address?

In reinforcement learning (RL) and related fields like large language model (LLM) alignment, a popular method for improving a policy (how an agent acts) is "success conditioning." This involves collecting data (trajectories) from an existing policy, filtering for those that achieved a desired outcome (e.g., a high reward, a correct answer), and then training a new policy to imitate the actions taken in those successful trajectories. This technique goes by many names, including rejection sampling with supervised fine-tuning (SFT), goal-conditioned RL, and Decision Transformers.

Despite its widespread use, it's been unclear what optimization problem success conditioning actually solves. It doesn't explicitly compute policy gradients or value functions, which are standard in RL. This lack of a clear theoretical foundation makes it hard to understand its limitations and guarantees. The paper aims to bridge this gap by providing a rigorous mathematical explanation for success conditioning.

Methods

The core contribution of this paper is a theoretical analysis that frames success conditioning as a trust-region policy optimization (TRPO) problem. Specifically, they show that success conditioning is the exact solution to a TRPO problem with the following characteristics:

  • Objective: Maximizing the first-order policy improvement, which measures how much the policy is expected to improve based on the current policy's advantages.
  • Constraint: A $\chi^2$ (chi-squared) divergence constraint, rather than the more common KL (Kullback-Leibler) divergence used in standard TRPO. This constraint limits how much the new policy can deviate from the old one.
  • Radius: The "trust region" radius (how far the policy is allowed to change) is not a hyperparameter but is automatically determined by a quantity called "action-influence."

The paper also introduces the concept of action-influence, a measure of how much random variation in actions under the behavior policy affects the probability of success. They prove a key identity: at every state, the relative policy improvement, the magnitude of policy change (measured by $\chi^2$ divergence), and the action-influence are all equal.

Datasets

Not discussed in the paper. The paper focuses on theoretical analysis and uses synthetic examples (like a two-armed bandit) to illustrate concepts.

Key Insights and Ideas

  • Success Conditioning is TRPO in Disguise: The most significant insight is that success conditioning isn't just a heuristic; it's a principled optimization method solving a specific trust-region problem. This provides a formal understanding of its behavior.
  • $\chi^2$ Divergence is Key: The use of $\chi^2$ divergence, instead of KL, makes success conditioning a more "conservative" policy improvement operator. It strongly penalizes exploring actions that were very rare under the original policy, preventing drastic distribution shifts.
  • Action-Influence as a Diagnostic: The "action-influence" quantity is a crucial finding. It quantifies the sensitivity of success rates to action choices and directly relates to how much policy improvement is possible. When action-influence is low, success conditioning makes minimal changes, explaining its failure mode.
  • Automatic Trust Region: Unlike standard TRPO where the trust region radius is a hyperparameter, success conditioning derives this radius automatically from the data via action-influence. This makes it more data-driven and less reliant on manual tuning.
  • Failure Mode Explained: The paper explains why success conditioning sometimes yields little improvement: it's not due to catastrophic distribution shifts (like in some RL methods), but rather due to low action-influence, leading to an overly conservative policy update.

Key Results

  • Exact Equivalence: Success conditioning is proven to be the exact solution to a specific trust-region optimization problem with $\chi^2$ divergence and an automatically determined radius.
  • Action-Influence Identity: At every state, the following three quantities are proven to be equal:
  • Relative policy improvement.
  • Magnitude of policy change (in $\chi^2$ divergence).
  • Action-influence.
  • Conservative Operator: Success conditioning is shown to be a conservative policy improvement operator that cannot degrade performance or cause dangerous distribution shifts. Its failure mode is observable as minimal policy change.
  • Return Thresholding Analysis: The paper provides a theoretical framework to analyze proxy rewards like return thresholding, showing they can amplify action-influence but risk misalignment with the true objective.

Limitations and Future Work

  • Finite Data and Function Approximation: The analysis primarily deals with exact success conditioning and assumes access to the true behavior policy and its value functions. The paper acknowledges that understanding the effects of finite data, function approximation, and large-scale optimization is an important, distinct question.
  • Iterated Success Conditioning: While the paper analyzes a single step of success conditioning, future work could explore the convergence properties of iterated success conditioning.
  • Forced Exploration: The current framework doesn't inherently include forced exploration, which could lead to policy collapse. Future work could investigate mechanisms to increase action-influence.
  • Algorithmic Choices: The paper suggests that understanding how policy movement diagnostics behave in realistic training regimes and the impact of specific algorithmic choices is an open area.

Summary updated

Abstract

A widely used technique for improving policies is success conditioning, in which one collects trajectories, identifies those that achieve a desired outcome, and updates the policy to imitate the actions taken along successful trajectories. This principle appears under many names---rejection sampling with SFT, goal-conditioned RL, Decision Transformers---yet what optimization problem it solves, if any, has remained unclear. We prove that success conditioning exactly solves a trust-region optimization problem, maximizing policy improvement subject to a $\chi^2$ divergence constraint whose radius is determined automatically by the data. This yields an identity: relative policy improvement, the magnitude of policy change, and a quantity we call action-influence---measuring how random variation in action choices affects success rates---are exactly equal at every state. Success conditioning thus emerges as a conservative improvement operator. Exact success conditioning cannot degrade performance or induce dangerous distribution shift, but when it fails, it does so observably, by hardly changing the policy at all. We apply our theory to the common practice of return thresholding, showing this can amplify improvement, but at the cost of potential misalignment with the true objective.
BibTeX
@inproceedings{russo2026success,
  author = {Daniel Russo},
  title = {{Success-Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=FEmXFeqYNZ},
}
Figure