Introduction
Off-policy reinforcement learning (RL) is a critical paradigm that enables an agent to learn about a target policy while following a different behavior policy. This capability is vital for learning from historical data, human demonstrations, or safety-critical environments where exploration must be constrained. However, off-policy learning is notoriously difficult due to high variance introduced by importance sampling (IS) ratios ($\rho_t$), which account for the discrepancy between policies.
In standard "per-decision" IS theory, only the target (reward and next-state value) is typically scaled. Yet, many practical implementations scale the entire Temporal-Difference (TD) error. This paper investigates why this "theoretically inconsistent" placement is widely used and demonstrates its benefits as a variance reduction technique.
Theoretical Framework: The Control Variate Interpretation
The authors compare two primary methods of applying IS ratios to the TD error:
- The "Strict" Version ($\delta_t$): Only scales the target components (reward + discounted next-state value).
- The "Common" Version ($\tilde{\delta}_t$): Scales the entire TD error, including the current state's value estimate.
Mathematical analysis reveals that the "Common" version is equivalent to the "Strict" version plus a control variate. In statistics, a control variate uses a known quantity to reduce the variance of an estimate without introducing bias. By scaling the current state's value estimate by $\rho_t$, the algorithm cancels out significant noise because the IS ratio and the TD target are often highly correlated.
Experimental Setup: The Collision Task
To validate these theoretical insights, the authors utilized the Collision Task, a standard benchmark for off-policy prediction.
In this environment, an agent navigates an 8-state grid toward a potential "crash" reward. The target policy is aggressive (always moving forward), while the behavior policy is more cautious or random. The objective is to accurately estimate the value function despite the policy mismatch.
Empirical Results
The study benchmarked 10 different off-policy algorithms, including Gradient TD (GTD, GTD2), Emphatic TD, Vtrace, and Tree Backup.
Faster Convergence and Stability
Across nearly all tested algorithms, the "Common" version (blue curves) reached lower error states significantly faster than the "Strict" version (red curves). Furthermore, the "Common" implementations often plateaued at a lower total error, indicating superior stability.
Step-size Robustness
A significant challenge in RL is hyperparameter tuning, particularly the learning rate (step-size). Sensitivity analysis showed that scaling the entire TD error makes algorithms much less sensitive to the step-size parameter. The "Common" version remained stable over a significantly broader range of log-scale learning rates compared to the "Strict" version.
Conclusion
The findings suggest that scaling the entire TD error by the importance sampling ratio acts as an effective control variate. This simple implementation choice provides universal benefits across a wide array of off-policy algorithms, leading to faster convergence and improved robustness to hyperparameters. For practitioners and researchers developing new off-policy methods, correcting the whole TD error should be considered the preferred default design choice.
Summary updated