In reinforcement learning (RL) and counterfactual decision-making, we rarely have the luxury of perfect information. Whether we are evaluating a new recommendation policy using historical logs (Off-Policy Evaluation, or OPE) or trying to deduce an expert's hidden motives from their actions (Inverse Reinforcement Learning, or IRL), we must rely on intermediate approximations: reward models and dynamics models.
The shared insight across this body of research is that these proxy models introduce a dangerous vulnerability. Because downstream policies optimize against or are evaluated by these proxies, even minor, localized errors or misspecifications do not just create small inaccuracies—they propagate catastrophically, biasing the entire learning or evaluation pipeline.
The Reward Model Bottleneck in Off-Policy Evaluation
When evaluating a target policy using historical data, the Direct Method (DM) trains a supervised model to predict rewards for any given state-action pair. However, as shown in Doubly Robust Policy Evaluation and Learning by Miroslav Dudík et al., relying solely on these predictive reward models introduces severe estimation bias if the model fails to perfectly capture the true data-generating process.
Why does this happen? A reward model is trained on the distribution of the historical logging policy. When we evaluate a new target policy, it may select actions where the reward model's errors are magnified. Jacek Golebiowski et al. note in Learning Action Embeddings for Off-Policy Evaluation that even minor misspecifications in the reward function systematically degrade evaluation reliability, especially in large action spaces. This sensitivity to model misspecification makes it difficult to rely solely on regression-based approaches for high-stakes decision-making, a challenge highlighted in Counterfactual Learning with General Data-Generating Policies by Yusuke Narita et al.
To mitigate this, hybrid estimators combine importance sampling with reward modeling. Yet, as Yi Su et al. point out in CAB: Continuous Adaptive Blending for Policy Evaluation and Learning, these blended approaches remain highly sensitive to the underlying reward model's accuracy; if the model used to impute missing data is poorly specified, the evaluation inherits significant bias.
[True Data Distribution] ──> [Logging Policy Data] ──> [Supervised Reward Model]
│
Target Policy ──> [Evaluation] ──> Catastrophic Bias
(Exploits out-of-distribution errors)
Real-world benchmarks, such as the Open Bandit Dataset analyzed by Yuta Saito et al. in A Large-scale Open Dataset for Bandit Algorithms, confirm that the quality of the reward predictor acts as the primary bottleneck for reliable policy assessment. When data is sparse, this bottleneck tightens further: Ritam Majumdar et al. show in Reward Shaping Control Variates for Off-Policy Evaluation Under Sparse Rewards that a lack of reward signals prevents models from learning accurate value functions, injecting substantial bias into the final performance estimate.
This fragility extends beyond average returns to uncertainty quantification. Yingying Zhang et al. note in Conformal Off-Policy Prediction that small errors in modeling the reward distribution can lead to catastrophic failures in the reliability of prediction intervals. Furthermore, in Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders, Andrew Bennett et al. demonstrate that constructing explicit reward models in the presence of hidden confounders propagates substantial bias when estimating the overall value of a target policy.
The Unavoidable Coupling of Rewards and Dynamics
The problem compounds when we move from static contextual bandits to sequential environments where we must model the world's transitions. In Inverse Reinforcement Learning (IRL), the goal is to recover an expert's reward function. However, an agent's behavior is a function of both what they want (rewards) and how they think the world works (dynamics).
As Michael Herman et al. establish in Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics, the accuracy of a recovered reward function is strictly dependent on the accuracy of the transition model. If the environment's dynamics are poorly estimated, the reward estimation process will absorb that error, generating a biased reward function to explain away the mismatch between the model and the observed trajectories. The authors of Model-Free IRL Using Maximum Likelihood Estimation. similarly demonstrate that if the environment model is a poor approximation of reality, the resulting reward estimate will fail to capture the true intent of the observed agent.
Expert Demonstrations ──> [IRL Algorithm]
▲
│ (Absorbs transition errors)
[Inaccurate Dynamics Model] ──> Biased Reward Function
This coupling creates a severe challenge in offline settings. Siliang Zeng et al. explain in When Demonstrations meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning that because expert preferences are intrinsically linked to environment dynamics, a world model built on limited data cannot distinguish between an expert's true intent and the model's own transition errors.
This error propagation isn't unique to IRL. In model-based RL frameworks, Motoya Ohnishi et al. show in Continuous-time Value Function Approximation in Reproducing Kernel Hilbert Spaces that value function approximations directly inherit errors from inaccurate transition models, leading to suboptimal control policies. Even when deriving rewards implicitly through probabilistic inference, Tim G. J. Rudner et al. find in Outcome-Driven Reinforcement Learning via Variational Inference that the derived signals remain strictly constrained by the accuracy of the underlying dynamics model.
The Credit-Bias Tradeoff in Policy Optimization
When we use learned models to guide policy learning online, proxy errors manifest as a fundamental tradeoff between variance reduction (or credit assignment) and policy bias.
Consider the multi-agent credit assignment problem explored by Yueheng Li et al. in Difference Advantage Estimation for Multi-Agent Policy Gradients. To determine an individual agent's contribution to a global reward, algorithms use a reward network to calculate a localized "difference advantage." While this precise credit assignment accelerates learning by reducing environmental noise, it introduces a dangerous vulnerability:
$$\text{High Precision Credit Assignment} \implies \text{High Sensitivity to Reward Model Errors}$$
If the reward model is imperfect, agents will optimize for the inaccuracies in the model rather than the actual task objectives—a phenomenon known as policy bias. Attempting to maximize individual credit using imperfect reward models causes agents to deviate from the collective team objective.
A similar dynamic occurs when smoothing erratic user feedback in recommender systems. Shenliao Bao et al. demonstrate in Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems that while replacing noisy raw rewards with predictions from a supervised model stabilizes training, any systematic bias or poor calibration in that reward model directly misleads the RL agent, reinforcing incorrect behaviors.
Across all these domains, the takeaway is identical: proxy models are not passive tools for variance reduction. Because optimization processes aggressively exploit model blind spots, the quality of reward and dynamics predictions remains the primary bottleneck for reliable policy learning and evaluation.
Key Foundational Work
Doubly Robust Policy Evaluation and Learning (2011) — contributed 1 concept elements to this direction.
Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics (2016) — contributed 1 concept elements to this direction.