Agnostic System Identification for Model-Based Reinforcement Learning

Stéphane Ross, J. Andrew Bagnell

Introduction

Model-Based Reinforcement Learning (MBRL) offers a promising path toward sample-efficient robot learning. By learning a predictive model of environment dynamics, an agent can "plan" its actions offline, significantly reducing the need for expensive real-world interactions. However, MBRL faces a persistent challenge: the mismatch between the data used to train the model and the states visited by the resulting policy. This paper introduces an iterative algorithm that frames MBRL as a no-regret online learning problem, providing robust performance even when the model cannot perfectly capture the true system dynamics.

The Challenge: Covariate Shift in MBRL

The standard approach to MBRL involves a batch process: collect data using an exploration policy, fit a model, and then derive an optimal policy. This often fails due to covariate shift. As the learned policy improves, it enters regions of the state space that were not covered during the initial data collection. Because the model's predictions are unreliable in these new regions, the policy can fail catastrophically.

Illustration of covariate shift in the helicopter domain. The training distribution (red circle) fails to cover the states visited during the test trajectory (dashed red line).

This problem is exacerbated in "agnostic" settings, where the chosen model class is a simplified approximation of the complex real-world system. Traditional methods often assume the true dynamics lie within the model class (the realizable setting), an assumption that rarely holds in practice.

Proposed Method: DAgger for MBRL

To address this, the authors adapt the DAgger (Dataset Aggregation) algorithm to the MBRL framework. Instead of a single batch of data, the algorithm iteratively refines the model by aggregating data from the states the current policy actually visits.

The process follows these steps:

  1. Initialization: Train an initial model and policy using a base exploration distribution.
  2. Iterative Data Collection: In each iteration, collect new data by executing a mixture of the current learned policy and exploratory actions.
  3. Aggregation and Update: Add the new data to a cumulative dataset and update the model using a no-regret online learning algorithm, such as Follow-The-Regularized-Leader.
  4. Policy Synthesis: Solve the optimal control problem using the updated model to produce the next policy.

By reducing MBRL to online learning, the authors prove that if the model-learning component has "no regret," the resulting sequence of policies will converge toward optimal performance relative to the best possible model in the chosen class.

Experimental Evaluation

The algorithm was tested on a high-dimensional Helicopter Domain, involving tasks like hovering and complex aerobatic maneuvers (e.g., the "nose-in funnel").

Performance comparison on the helicopter hovering task. DAgger variants (solid lines) consistently achieve lower total cost compared to batch methods (dashed lines) and prior iterative approaches.

Key findings include:

  • Superior Convergence: In the hovering task, DAgger variants reached high performance within 50-100 iterations, while batch baselines and previous iterative methods plateaued at much higher costs.
  • Handling Complexity: For the "nose-in funnel" maneuver, DAgger successfully learned the task, whereas standard batch methods failed entirely.
  • Agnostic Robustness: The method proved effective even when the model class was misspecified (e.g., ignoring mechanical delays), demonstrating its practicality for real-world systems.

Conclusion

By framing system identification as an online learning problem, this work provides a theoretically grounded and practically robust solution to the train-test mismatch in MBRL. The approach eliminates the need for perfect model realizability and offers sample complexity guarantees that scale with model complexity rather than state-space dimensionality, marking a significant step toward scalable and reliable model-based control.

Summary updated

BibTeX
@inproceedings{ross2012agnostic,
  author = {Stéphane Ross and J. Andrew Bagnell},
  title = {{Agnostic System Identification for Model-Based Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2012},
  url = {https://openreview.net/forum?id=Hy4gssbO-B},
}
Figure