In reinforcement learning (RL), evaluating a new "target" policy using data collected by a "behavior" policy is a foundational task. The standard tool for this is Importance Sampling (IS), which reweights observed rewards by the ratio of the target policy's action probabilities to the behavior policy's action probabilities.
Conventionally, researchers assumed that if the behavior policy is known—for instance, if we programmed the robot ourselves—we should use that ground-truth probability in our calculations. However, a growing body of research reveals a counter-intuitive paradox: using an estimated behavior policy, derived from the observed data itself, often produces significantly more accurate results than using the true, known policy.
The Problem of Sampling Error
The core issue stems from "sampling error." Even if we know a behavior policy has a 50% chance of taking an action, a finite dataset might only show that action 45% of the time due to random chance. Hanna et al. (2019) and Hanna et al. (2021) argue that standard estimators fail to account for these accidental fluctuations. If the estimator uses the "true" 50% probability, it incorrectly weights the rewards, leading to high variance.
This discrepancy is not limited to off-policy settings. Zhong et al. (2022) and Corrado and Hanna (2025) demonstrate that even "on-policy" evaluation—where the agent follows the target policy exactly—suffers from sampling error in finite batches. The empirical distribution of actions rarely matches the theoretical distribution, introducing noise into gradient estimates and value functions.
The Solution: Empirical Estimation as a Filter
By substituting the known behavior policy with a Maximum Likelihood Estimate (MLE) calculated from the dataset, researchers can "cancel out" this sampling noise. Liu et al. (2019) refer to this as a "surrogate policy." Because the surrogate policy is built from the same data used for evaluation, the importance weights ($\pi / \hat{b}$) naturally adjust to the specific random fluctuations present in the sample.
Zhou et al. (2025) provide a theoretical bridge for this intuition, showing that using an estimated policy is mathematically equivalent to projecting the return onto a lower-dimensional functional space. This projection acts as an implicit control variate, filtering out noise that does not contribute to the true expectation. Similarly, Chen et al. (2020) show that when data comes from multiple sources, estimating a "virtual" mixture policy serves as an inductive bias that prevents the model from overfitting to rare transitions.
Beyond Estimation: Designing Better Behavior
If estimating the behavior policy helps, can we go a step further and design the behavior policy to be more informative?
Standard practice assumes the target policy is the best at collecting its own data. However, Hanna et al. (2017) prove this is often false. They propose the Behavior Policy Gradient (BPG) to search for a behavior policy that specifically minimizes the mean squared error of the evaluation.
This "evaluation-optimized" approach is particularly powerful when combined with historical data. Zhang (2023) and Liu et al. (2024) suggest that offline data should be used to identify high-variance regions of the state space. By directing online exploration toward these informative areas, practitioners can achieve stable performance estimates with a fraction of the usual online interaction.
Applications and Constraints
The benefits of empirical behavior estimation extend across several RL subfields:
- Temporal Difference Learning: Pavse et al. (2020) introduced PSEC-TD(0) to correct sampling error in batch TD updates, ensuring the value function converges to the true expected return rather than a batch-specific artifact.
- Learning to Rank: Oosterhuis and de Rijke (2021) show that conditioning on the aggregate history of all logging policies—rather than just the active one—reduces variance when items' exposure changes drastically.
- Causal Inference: Kato et al. (2021) demonstrate that estimating the logging policy produces more stable estimates in adaptive experiments like multi-armed bandits.
However, this technique must be used carefully. Liu and Brunskill (2022) warn of "propensity overfitting," where a policy might learn to avoid low-reward states by manipulating importance weights. They suggest that constraining the policy to actions actually observed in the data is necessary to maintain reliable estimates.
Ultimately, the shift toward empirical behavior estimation represents a move from treating data as a perfect reflection of a policy to treating it as a noisy signal that requires active correction. As Li et al. (2015) established in their minimax analysis, while importance sampling is a standard tool, its efficiency is fundamentally tied to how well we account for the specific distribution of the data at hand.
Key Foundational Work
Data-Efficient Policy Evaluation Through Behavior Policy Search (2017) — contributed 2 concept elements to this direction.
Toward Minimax Off-policy Value Estimation (2015) — contributed 1 concept elements to this direction.