In reinforcement learning (RL), evaluating or optimizing a new policy using data collected by a different policy is a notoriously difficult challenge. This problem, known as off-policy evaluation (OPE) and offline policy optimization, underpins high-stakes applications like healthcare, robotics, and autonomous driving, where real-world trial and error is too dangerous or expensive.
Traditional approaches to this problem suffer from two fundamental failure modes:
- The Curse of Horizon: Standard importance sampling multiplies action probability ratios step-by-step along a trajectory. Over long horizons, this product causes the variance of the estimator to explode exponentially, rendering it useless for infinite-horizon tasks.
- Bootstrapping Breakdown: When optimizing a policy offline using temporal difference (TD) learning, the algorithm evaluates actions outside the dataset. Without real-world feedback to correct it, the value function wildly overestimates these out-of-distribution (OOD) actions, causing policy performance to collapse.
Stationary distribution correction bypasses both issues. Instead of tracking sequential trajectories or bootstrapping ungrounded value functions, it directly estimates the long-term, steady-state occupancy frequencies of the target policy relative to the data-collection policy. By shifting the focus from trajectories to stationary distributions, this framework transforms unstable sequential estimation into a stable, often convex, optimization problem.
The Core Insight: Marginalization
To understand why stationary distribution correction works, we must contrast it with standard importance sampling.
Suppose we want to evaluate the expected discounted reward of a target policy $\pi$ using a dataset generated by a behavior policy $\mu$. Standard importance sampling computes a weight for an entire trajectory $\tau = (s_0, a_0, s_1, a_1, \dots)$ by multiplying the step-by-step action ratios:
$$W(\tau) = \prod_{t=0}^{T} \frac{\pi(a_t | s_t)}{\mu(a_t | s_t)}$$
If $\pi$ and $\mu$ disagree even slightly on a few actions, this product either vanishes to zero or explodes to infinity as the horizon $T$ grows.
However, the total expected reward does not actually depend on the exact sequential order of the states; it only depends on the total discounted frequency with which the target policy visits each state-action pair. This frequency is called the discounted stationary distribution (or occupancy measure), denoted $d_\pi(s, a)$.
If we can determine the ratio between the target policy's occupancy measure and the behavior policy's occupancy measure, we can define a marginalized weight:
$$w(s, a) = \frac{d_\pi(s, a)}{d_\mu(s, a)}$$
As Qiang Liu et al. demonstrated, evaluating the policy using this marginalized density ratio completely eliminates the exponential variance growth caused by long horizons. The variance of the estimator becomes independent of the trajectory length, effectively breaking the curse of horizon.
Trajectory Importance Sampling (Explodes with Horizon):
(s0, a0) -> (s1, a1) -> (s2, a2) -> ... -> (sT, aT) [Multiply weights at every step]
Marginalized Importance Sampling (Horizon-Independent):
Data Distribution d_μ(s,a) ───[ Multiply by w(s,a) ]───> Target Distribution d_π(s,a)
Why It Works: Flow Conservation and Duality
The obvious challenge is: how do we calculate $w(s, a)$ if we cannot run the target policy $\pi$ in the environment, and we do not know the underlying transition dynamics?
The solution lies in a fundamental property of stationary distributions: flow conservation. For any state $s'$, the probability of the target policy landing in $s'$ must equal the probability of starting there plus the probability of transitioning there from a previous state $s$ via an action $a$. This relationship is governed by the adjoint Bellman equation:
$$d_\pi(s') = (1-\gamma) \rho_0(s') + \gamma \sum_{s, a} P(s' | s, a) d_\pi(s, a)$$
where $\rho_0$ is the initial state distribution and $\gamma$ is the discount factor.
By dividing this entire equation by the behavior distribution $d_\mu(s')$, we can reframe the flow conservation constraint in terms of our weight $w(s, a)$. However, directly solving this equation is still challenging because the transition probabilities $P(s' | s, a)$ are unknown.
To bypass this, the foundational DICE (Distribution Correction Estimation) framework leverages mathematical duality. As introduced by Bo Dai et al. in their DualDICE algorithm, we can apply Fenchel duality to transform the constrained flow-conservation problem into an unconstrained minimax saddle-point problem.
This optimization introduces an auxiliary function $\nu(s)$—which acts as a smooth, backward Bellman-like operator—to enforce the flow constraints. The density ratios are derived directly from the residuals of this auxiliary function:
$$\min_{\nu} \max_{w} \mathbb{E}_{(s,a,s') \sim d_\mu} \left[ w(s,a) \left( \nu(s) - \gamma \nu(s') - r(s,a) \right) \right] - \zeta(w)$$
where $\zeta(w)$ is a regularizer (such as a f-divergence).
This formulation is incredibly powerful for three reasons:
- Behavior-Agnostic: It does not require explicit knowledge of the behavior policy $\mu$ that collected the data, as shown by Bo Dai et al.. The expectations are evaluated purely using samples from the offline dataset.
- No Simulators Required: It avoids compounding errors from explicit transition modeling. As Junfeng Wen et al. noted, direct ratio estimation is significantly more robust than building a generative simulator of complex environments.
- Convex Stability: By choosing appropriate regularizers, the optimization objective becomes convex-concave, avoiding the training pathologies common to adversarial networks.
From Evaluation to Optimization
While stationary distribution correction was originally designed for off-policy evaluation, researchers quickly realized it could revolutionize offline policy optimization.
In standard offline RL, algorithms try to learn a value function $Q(s,a)$. If the policy chooses an action $a$ not present in the dataset, the $Q$-network has to guess its value, frequently leading to catastrophic overestimation.
Algorithms like OptiDICE, developed by Jongmin Lee et al., completely flip this paradigm. Instead of optimizing a value function over unseen actions, OptiDICE directly estimates the stationary distribution ratio of the optimal policy relative to the behavior data.
Because the optimization is formulated strictly as a reweighting of the existing dataset, the agent is mathematically constrained to the support of the offline data. It finds the optimal policy by deciding how much to value the trajectories it already has, completely sidestepping the need to evaluate out-of-distribution actions.
Extensions and Variations
The core mechanism of stationary distribution correction has been adapted across several frontiers in reinforcement learning:
- Alternative Representations: Rather than using saddle-point optimization, Scott Fujimoto et al. introduced SR-DICE, which reformulates density ratio estimation as a reward optimization problem using the Successor Representation (SR). This allows the model to maintain highly precise density ratios across thousands of changing reward functions.
- Multi-Agent Coordination: In multi-agent settings, the joint action space grows exponentially, making value estimation highly unstable. Algorithms like AlberDICE (Daiki E. Matsunaga et al.) and ComaDICE (The Viet Bui et al.) apply stationary distribution regularization to coordinate multiple agents, preventing them from drifting into uncoordinated, out-of-distribution joint behaviors.
- Safe Reinforcement Learning: For safety-critical systems, Jongmin Lee et al. developed COptiDICE, which uses distribution correction ratios to compute provable upper bounds on policy costs. This enables the extraction of optimal policies that strictly adhere to safety constraints without requiring dangerous real-world exploration.
By shifting the fundamental representation of off-policy learning from sequential trajectories to global occupancy ratios, stationary distribution correction provides a mathematically rigorous foundation for stable, reliable, and horizon-independent offline AI.