Off-policy Distributional Q(λ): Distributional RL without Importance Sampling

Yunhao Tang, Mark Rowland, Rémi Munos, Bernardo Ávila Pires, Will Dabney

Abstract

We introduce off-policy distributional Q($\lambda$), a new addition to the family of off-policy distributional evaluation algorithms. Off-policy distributional Q($\lambda$) does not apply importance sampling for off-policy learning, which introduces intriguing interactions with signed measures. Such unique properties distinguish distributional Q($\lambda$) from other existing alternatives such as distributional Retrace. We characterize the algorithmic properties of distributional Q($\lambda$) and validate theoretical insights with tabular experiments. We show how distributional Q($\lambda$)-C51, a combination of Q($\lambda$) with the C51 agent, exhibits promising results on deep RL benchmarks.

Introduction

In reinforcement learning (RL), off-policy learning allows an agent to learn about a target policy using data generated by a different behavior policy. Traditionally, these methods rely on Importance Sampling (IS) to correct for the discrepancy between policies. However, IS often suffers from high variance, leading to unstable updates, and requires precise knowledge of the behavior policy probabilities, which may not be available in real-world scenarios.

While Distributional RL—which models the full distribution of future rewards—has proven highly effective, existing multi-step off-policy variants typically still depend on IS. This work introduces Off-policy Distributional $Q(\lambda)$, a multi-step distributional algorithm that operates entirely without importance sampling by leveraging the mathematical framework of signed measures.

Methodology

The core of the proposed approach is the Off-policy Distributional $Q(\lambda)$ operator. Unlike algorithms like Retrace that use policy ratios to reweight updates, this operator utilizes a fixed trace coefficient $\lambda \in [0, 1]$ to interpolate between one-step temporal difference learning and Monte-Carlo sampling.

Signed Measures and Categorical Projection

A key technical innovation is the use of signed measures. In standard distributional RL, intermediate learning steps are constrained to be valid probability distributions (non-negative values summing to 1). Off-policy $Q(\lambda)$ relaxes this, allowing intermediate "probabilities" to become negative. To implement this in deep learning, the authors adapted the C51 architecture, modifying the projection step to handle these signed measures by removing the non-negativity constraint during updates.

Categorical projection of a signed measure η into a discrete representation, allowing for negative mass.

To maintain stability, the algorithm incorporates Trust Region Mixing, which mixes the target policy with the behavior policy to ensure the "off-policyness" remains within a contractive mathematical radius.

Theoretical Insights: The Signed Measure Detour

The most significant theoretical insight is the necessity of the "signed measure detour." The authors prove that allowing for signed measures is essential for convergence when avoiding importance sampling. If the algorithm were forced to remain within the space of valid probability distributions at every step, it would fail to converge to the correct target distribution.

Convergence of the distributional Q(λ) operator over iterations, illustrating the temporary emergence of signed measures (green) before reaching a valid probability distribution.

The paper also establishes a "contraction radius," providing a theoretical budget for the divergence between target and behavior policies. This justifies the use of mixing heuristics to keep updates stable.

Experiments and Results

The algorithm was evaluated on tabular Markov Decision Processes (MDPs) to validate theoretical convergence and on the Atari-57 benchmark to test scalability.

Key Results

  • Performance: On Atari-57, $Q(\lambda)$-C51 outperformed both the original C51 and Retrace-C51, showing significant gains in Inter-Quartile Mean (IQM) and Median scores.
  • Empirical Validation: Tabular experiments confirmed that with an appropriately tuned $\lambda$ (e.g., $\lambda=0.4$), the algorithm converges faster and more accurately than competing methods.
  • Stability: By avoiding importance sampling, the algorithm maintained stability in complex deep RL settings where IS-based methods often fail, provided the policy mismatch was managed via target mixing.

Discussion and Future Work

While promising, the performance of $Q(\lambda)$-C51 is sensitive to hyperparameters like $\lambda$ and the mixing coefficient $\alpha$. Furthermore, the current theoretical contraction bounds are conservative; the algorithm often converges in practice even when theoretical limits are exceeded. Future work may explore tighter theoretical analysis and the application of this approach to other distributional representations, such as Quantile Regression.

Summary updated

Abstract

We introduce off-policy distributional Q($\lambda$), a new addition to the family of off-policy distributional evaluation algorithms. Off-policy distributional Q($\lambda$) does not apply importance sampling for off-policy learning, which introduces intriguing interactions with signed measures. Such unique properties distributional Q($\lambda$) from other existing alternatives such as distributional Retrace. We characterize the algorithmic properties of distributional Q($\lambda$) and validate theoretical insights with tabular experiments. We show how distributional Q($\lambda$)-C51, a combination of Q($\lambda$) with the C51 agent, exhibits promising results on deep RL benchmarks.
BibTeX
@inproceedings{tang2026off,
  author = {Yunhao Tang and Mark Rowland and Rémi Munos and Bernardo Ávila Pires and Will Dabney},
  title = {{Off-policy Distributional Q(λ): Distributional RL without Importance Sampling}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=zQT82QJh8c},
}
Figure