Nonparametric Model-Based Reinforcement Learning

Christopher G. AtkesonNeurIPS 1997

In model-based reinforcement learning (MBRL), a common assumption is that the planner should strictly follow the learned model of the world. However, this paper argues the opposite: during the early stages of learning, being "too consistent" with a flawed model can lead to catastrophic failure. By treating the model's predictions as "soft" suggestions rather than hard constraints, robots can successfully learn complex, dynamic tasks—such as swinging up an inverted pendulum—from simple human demonstrations.

The Problem with "Perfect" Planning

MBRL is a powerful paradigm where an agent learns a simulator (a model) of the world and then uses a planner to find the optimal sequence of actions within that simulator. The catch is that early in the learning process, the model is inevitably inaccurate.

Traditional "consistent" planners, such as Differential Dynamic Programming (DDP), treat the model as absolute truth. When the model is flawed, this rigid adherence leads to three major headaches:

  1. Initialization Hell: The system requires a starting trajectory that perfectly obeys a model that isn't even correct yet.
  2. Numerical Explosions: Integrating an unstable or inaccurate model forward in time often causes the underlying mathematics to "blow up."
  3. Derivative Noise: Planners frequently rely on the model's derivatives. In nonparametric models (which lack a fixed parametric shape), these derivatives are often noisy, leading the planner down nonsensical paths.

Inconsistent Local Planning

To make planning robust when the model is still a "work in progress," the author introduces Inconsistent Local Planning. Instead of forcing the state at the next time step ($x_{k+1}$) to be exactly what the model predicts ($f(x_k, u_k)$), this requirement is relaxed into a soft constraint.

The core innovation is a modified cost function: $$C_{new} = \sum [c(x_k, u_k) + \lambda |x_{k+1} - f(x_k, u_k)|^2]$$

Here, $\lambda$ acts as a "trust" parameter:

  • If $\lambda$ is exceptionally large, the planner must obey the model perfectly, reverting to the traditional problem.
  • If $\lambda$ is small, the planner prioritizes the task goal (like minimizing effort or reaching a target) over strictly following a potentially wrong model.

To implement this, the trajectory is represented using B-splines rather than step-by-step integration, allowing state and control histories to be optimized independently. For the model itself, the system uses Locally Weighted Learning (LWL), a nonparametric technique that builds a local model on the fly by querying nearby data points in a database.

Experimental Setup: The Pendulum Swing-Up

Rather than relying on standard digital benchmarks, the approach was evaluated on a real-world robotics task. The hardware consisted of a SARCOS seven-degree-of-freedom hydraulic anthropomorphic robot arm.

SARCOS robot arm performing the pendulum swing-up task

The objective was the "Pendulum Swing-Up": the robot holds a pendulum hanging downward and must move its hand horizontally to swing the pendulum into an inverted, balanced position. A single human demonstration, captured via stereo vision, served as the initial data point.

Key Insights

  • Planners are "Error Seekers": If a model has a slight flaw that suggests a "free lunch" (e.g., a trajectory that seemingly costs zero energy), the planner will find and exploit it. Strict planners amplify modeling errors rather than smoothing them over.
  • The "Inconsistency" Paradox: To eventually achieve a perfect plan, you must start with an imperfect one. By allowing the plan to be slightly "unphysical" (inconsistent with the current model), the robot can explore trajectories that lead to better data collection and, ultimately, a better model.
  • Derivatives are Dangerous: While nonparametric models offer excellent flexibility, their gradients are often jagged. Traditional planners that rely heavily on these gradients fail early on; the proposed soft-constraint method is significantly more forgiving.

Results and Performance

The experiments yielded several compelling results demonstrating the efficacy of the soft-constraint approach.

Pendulum angle over time across multiple learning trials
  • Direct Imitation Failed: Simply having the robot mimic the human's hand movement was insufficient. Because the robot's physical dynamics differed from the human's, the pendulum barely reached halfway up during the first trial.
  • Rapid Success: Using the inconsistent planner, the robot learned a successful swing-up trajectory by the third trial, closely matching the desired inverted state.
  • The $\lambda$ Sweet Spot: The exact value of the trust parameter $\lambda$ wasn't hyper-critical—it functioned well across a broad range (100 to 100,000). However, the extremes were fatal. If $\lambda = 0$, the model was entirely ignored and learning failed. If $\lambda$ was too high, the planner became too rigid to find a viable solution.

Limitations and Future Directions

While successful, the approach has limitations. The optimization relied on Powell’s method to tune the spline knot points, which is computationally inefficient. Furthermore, while the method worked well for the pendulum task, scaling it to model the full 21-dimensional dynamics of the SARCOS arm would require an infeasible amount of data for rapid learning. Future work may explore dynamic $\lambda$ tuning, where the trust parameter adapts over time or varies based on the model's confidence in specific regions of the state space.

Summary updated

BibTeX
@inproceedings{atkeson1997nonparametric,
  author = {Christopher G. Atkeson},
  title = {{Nonparametric Model-Based Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}
Figure