Nonparametric Bayesian Policy Priors for Reinforcement Learning

Finale Doshi-Velez, David Wingate, Nicholas Roy, Joshua B. TenenbaumNeurIPS 2010 · OpenReview

TL;DR

This paper introduces a Bayesian framework that simultaneously learns a world model and a policy by treating expert demonstrations as evidence of both. By applying nonparametric priors that favor "simple" world dynamics and "simple" controllers, the agent can learn effectively in complex, partially observable environments even with limited or imperfect expert data.

What Problem Does This Paper Address?

Reinforcement Learning (RL) in partially observable environments (POMDPs) is notoriously difficult because the agent never sees the true state of the world—only noisy observations. Traditionally, researchers tackle this in two ways:

  1. Model-based RL: The agent explores the world to learn the physics (dynamics) and then plans a path.
  2. Imitation Learning: The agent watches an expert and tries to mimic their policy, often assuming the world's physics are already known.

The gap lies in how to use both types of data—independent exploration and expert trajectories—simultaneously. Expert actions are more than just "good moves"; they contain "hidden" information about what the expert believes the world looks like. The challenge is creating a mathematical bridge that allows an agent to update its belief about the world's dynamics based on the expert's choices.

Methods

The authors propose a Bayesian approach where the world model ($M$) and the expert's policy ($\pi_e$) are linked. Their core contribution is a joint prior that biases the agent toward models that are both simple to represent and easy to control.

Graphical models comparing standard inference (A) and the proposed joint model-policy inference (B)

Key technical innovations include:

  • The "Role Reversal" Trick: To create a policy prior, the authors take the infinite POMDP (iPOMDP)—a nonparametric model usually used to learn world dynamics—and swap the roles of actions and observations. In this setup, observations become inputs and actions become outputs, effectively turning the world-modeling machinery into a Finite State Controller (FSC) learner.
  • Joint Model-Policy Inference: They propose inference algorithms that link the policy and world model. The most robust is an alternating sampling scheme, which alternates between sampling a policy (given the current world model) and sampling a world model (given the current policy and expert data). This avoids the "brittleness" of standard importance sampling in high-dimensional spaces.
  • Nonparametric Priors: By using Dirichlet Process-based priors, the agent doesn't need to know the number of "states" in the world or "nodes" in the controller in advance; the model grows in complexity only as the data demands it.

Datasets

The researchers evaluated their approach on a wide variety of benchmarks:

  • Toy/Standard POMDPs: Tiger, Network, Shuttle, Hallway, Tag, and RockSample.
  • Complex Domains: A 16,321-state Image-search task and a "Beach" problem involving tracking a ball on a 2D grid.
  • Custom Scenarios: A "Multicolored Gridworld" designed to test simplicity biases and a simulated Snake Robot with a 40-dimensional state space, where the "expert" data was generated by a non-optimal randomized planner (RRT).

Key Insights and Ideas

  • The Expert as a Mirror of the World: The most profound idea here is that an expert's policy is a reflection of the world's dynamics. If an expert behaves simply, it suggests the underlying world might have a simple, controllable structure.
  • Simplicity as a Regularizer: By using priors that favor fewer states and fewer controller nodes, the model resists overfitting to the noise inherent in partial observations.
  • Robustness to "Mediocre" Experts: Unlike many imitation learning algorithms that fail if the expert isn't perfect, this Bayesian framework treats expert data as a "bias" rather than a hard constraint. In the Snake Robot experiment, the agent actually outperformed the "expert" it learned from.

Key Results

The proposed joint inference approaches consistently outperformed standard iPOMDPs and EM-based learners across almost all tasks.

Cumulative rewards in the Multicolored Gridworld and Snake domains, alongside a diagram of the simulated snake robot
  • Performance Gain: In the "Network" and "Gridworld" tasks, the proposed methods achieved significantly higher cumulative rewards compared to baselines that ignored the expert's policy structure.
  • Sample Efficiency: The simplicity bias allowed the agent to find "good enough" policies much faster than agents trying to learn the full, complex dynamics of the environment from scratch.
  • Scalability: The policy-based approach was particularly successful in larger problems (like Image-search and Tag), where traditional model-based planning often becomes computationally prohibitive.

Limitations and Future Work

  • Computational Cost: While more sample-efficient in terms of environment interactions, the MCMC (Markov Chain Monte Carlo) sampling required for nonparametric Bayesian inference is computationally heavy.
  • Discrete Focus: The current work focuses primarily on discrete action and observation spaces. Extending this to fully continuous domains (beyond the motor primitives used in the Snake example) remains a challenge.
  • Expert Assumptions: The framework assumes the expert is "near-optimal." If the expert is intentionally deceptive or completely random, the prior might bias the agent toward incorrect world models.

Summary updated

Abstract

We consider reinforcement learning in partially observable domains where the agent can query an expert for demonstrations. Our nonparametric Bayesian approach combines model knowledge, inferred from expert information and independent exploration, with policy knowledge inferred from expert trajectories. We introduce priors that bias the agent towards models with both simple representations and simple policies, resulting in improved policy and model learning.
BibTeX
@inproceedings{doshivelez2010nonparametric,
  author = {Finale Doshi-Velez and David Wingate and Nicholas Roy and Joshua B. Tenenbaum},
  title = {{Nonparametric Bayesian Policy Priors for Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=r1ZmCLbObH},
}
Figure