To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning

Yuda Song, Dhruv Rohatgi, Aarti Singh, J. Andrew BagnellarXiv 2025 · arXiv · OpenReview

Abstract

Partial observability is a notorious challenge in reinforcement learning (RL), due to the need to learn complex, history-dependent policies. Recent empirical successes have used privileged expert distillation—which leverages availability of latent state information during training (e.g., from a simulator) to learn and imitate the optimal latent, Markovian policy—to disentangle the task of "learning to see" from "learning to act". While expert distillation is more computationally efficient than RL without latent state information, it also has well-documented failure modes. In this paper—through a simple but instructive theoretical model called the perturbed Block MDP, and controlled experiments on challenging simulated locomotion tasks—we investigate the algorithmic trade-off between privileged expert distillation and standard RL without privileged information. Our main findings are: (1) The trade-off empirically hinges on the stochasticity of the latent dynamics, as theoretically predicted by contrasting approximate decodability with belief contraction in the perturbed Block MDP; and (2) The optimal latent policy is not always the best latent policy to distill. Our results suggest new guidelines for effectively exploiting privileged information, potentially advancing the efficiency of policy learning across many practical partially observable domains.

Introduction

Reinforcement Learning (RL) in real-world scenarios is often hampered by partial observability, where agents lack a complete view of the environment's state. Common solutions involve privileged expert distillation, where an expert agent is trained with full access to latent states in a simulator, and a student agent is then trained to imitate this expert using only limited observations. While this approach is computationally efficient, its reliability in noisy environments remains a subject of debate. This paper investigates the fundamental trade-off between this distillation approach and standard RL using frame-stacking, which learns directly from observation history.

Methodology

The study employs three primary methodologies to analyze the distillation-RL trade-off:

  1. Theoretical Modeling: The authors introduce the Perturbed Block MDP, a model where observations are generally informative but occasionally contain pure noise. This framework allows for the mathematical analysis of "decodability" (interpreting current observations) versus "belief contraction" (using history to resolve uncertainty).
  2. Algorithmic Comparison: The performance of Behavior Cloning (offline), DAgger (online imitation), and MrQ (standard RL) is compared across varying levels of environmental stochasticity.
  3. Expert Smoothing: A novel technique is proposed where the latent expert is trained with intentional motor noise. This creates a "smoother" policy that is more robust and easier for a partially-sighted student to imitate than a perfectly optimal, but "twitchy," expert.

Key Insights: Stochasticity and Compounding Errors

A critical finding is that environmental stochasticity dictates the success of distillation. In deterministic settings, distillation methods like DAgger are highly efficient. However, as noise increases, the error in decoding latent states from observations compounds over the task horizon.

Suboptimality vs. Horizon for the Humanoid task, showing how distillation errors grow linearly with time.

Furthermore, the authors discover that the most "optimal" expert is often a poor teacher. A perfectly optimal policy may rely on high-frequency adjustments to minute state changes that a student cannot perceive. By "smoothing" the expert through noise injection during its training, the resulting policy becomes more learnable for the student, significantly improving performance in partially observable tasks.

Empirical Results

The researchers evaluated these insights on the DeepMind Control Suite using Walker, Dog, and Humanoid locomotion tasks.

Reward vs. Training Time across three tasks, comparing Behavior Cloning (BC), DAgger, and Reinforcement Learning (RL).
  • Walker: In low-noise environments, DAgger converged significantly faster and achieved higher rewards than standard RL.
  • Dog and Humanoid: As motor noise increased, the performance of distilled agents (DAgger) declined. In the Dog task, standard RL eventually surpassed DAgger in total reward, albeit requiring more training time.
  • Smoothing Benefits: In the Humanoid task with 0.2 motor noise, students imitating a "smoothed" expert (trained at 0.5 noise) outperformed those imitating an expert trained at the task's native noise level.

Conclusion

The choice between distilling an expert and deciding via standard RL hinges on the balance between computational efficiency and robustness to noise. While distillation excels in near-deterministic worlds, standard RL's ability to utilize history makes it superior as stochasticity increases. These findings suggest that practitioners should favor "smoother" experts and carefully consider environmental noise when choosing their learning strategy.

Summary updated

Abstract

Partial observability is a notorious challenge in reinforcement learning (RL), due to the need to learn complex, history-dependent policies. Recent empirical successes have used privileged expert distillation--which leverages availability of latent state information during training (e.g., from a simulator) to learn and imitate the optimal latent, Markovian policy--to disentangle the task of "learning to see" from "learning to act". While expert distillation is more computationally efficient than RL without latent state information, it also has well-documented failure modes. In this paper--through a simple but instructive theoretical model called the perturbed Block MDP, and controlled experiments on challenging simulated locomotion tasks--we investigate the algorithmic trade-off between privileged expert distillation and standard RL without privileged information. Our main findings are: (1) The trade-off empirically hinges on the stochasticity of the latent dynamics, as theoretically predicted by contrasting approximate decodability with belief contraction in the perturbed Block MDP; and (2) The optimal latent policy is not always the best latent policy to distill. Our results suggest new guidelines for effectively exploiting privileged information, potentially advancing the efficiency of policy learning across many practical partially observable domains.
BibTeX
@misc{song2025distill,
  author = {Yuda Song and Dhruv Rohatgi and Aarti Singh and J. Andrew Bagnell},
  title = {{To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning}},
  year = {2025},
  eprint = {2510.03207},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.03207},
}
Figure