Agnostic System Identification for Model-Based Reinforcement Learning

Stéphane Ross, Drew Bagnell

TL;DR

This paper introduces an iterative algorithm that provides strong performance guarantees for model-based reinforcement learning (MBRL), even when the "true" system cannot be perfectly represented by the chosen model class (the "agnostic" case). By reducing system identification to a no-regret online learning problem, the authors demonstrate a robust way to overcome the classic "train-test mismatch" that often causes autonomous systems to fail when encountering states unseen during training.

What Problem Does This Paper Address?

Model-Based Reinforcement Learning is a powerful paradigm where an agent learns a mathematical model of how the world works (system identification) and then uses that model to plan its actions. However, it suffers from a fundamental flaw: train-test mismatch.

Illustration of train-test mismatch in a helicopter trajectory task.

Usually, we collect data using an "exploration policy" (like a human expert or random actions) and fit a model to this data. But when we use that model to derive a new "optimal" policy, that policy might lead the agent into entirely new parts of the state space where the model has never seen data. If the model is even slightly wrong in those new regions, the errors compound, leading to catastrophic failure.

Traditional control theory often assumes the "true" system belongs to the class of models we are using (e.g., assuming a non-linear helicopter is perfectly linear). When this assumption is false—which it almost always is in the real world—standard "batch" learning methods provide no guarantees that the resulting controller will actually work.

Methods

The authors propose an iterative approach based on DAgger (Dataset Aggregation), originally developed for imitation learning, and adapt it for MBRL.

The algorithm works in a loop:

  1. Execute: Run the current best policy (derived from the current model) to collect new state-transition data.
  2. Aggregate: Add this new data to a growing dataset containing all transitions seen in previous iterations.
  3. Update: Train a new model on the entire aggregated dataset.
  4. Solve: Solve the Optimal Control (OC) problem for this new model to get an updated policy.

What’s new? Unlike "Batch" methods that collect data once and hope for the best, this approach treats system identification as a no-regret online learning problem. By training on data generated by its own previous mistakes, the model "learns" where it is weak and corrects itself. The authors provide a formal reduction showing that if the online learning algorithm has "low regret" (meaning it performs nearly as well as the best possible model in the class in hindsight), the resulting policy is guaranteed to perform well relative to the best policy that could have been found using that model class.

Datasets

The researchers evaluated their approach on a high-fidelity simulated helicopter domain.

  • State Space: 21-dimensional continuous state (position, velocity, orientation, etc.).
  • Control Space: 4-dimensional continuous control (cyclic, collective, and tail rotor inputs).
  • Tasks:
    • Hovering: Maintaining a stationary position in the presence of noise and aerodynamic delays.
  • Nose-in Funnel: A complex aerobatic maneuver where the helicopter circles a point while keeping its nose pointed at the center.

Key Insights and Ideas

  • Embracing the "Agnostic" Reality: We don't need a "perfect" model of the universe. We just need a model that is accurate in the specific regions of the state space that our policy actually visits.
  • The Power of Aggregation: By keeping all old data, the algorithm ensures that the model doesn't "forget" how the system behaves in earlier regions while trying to master new ones.
  • Robustness to Exploration: In batch learning, if your exploration data is bad, your policy is bad. In the DAgger-based approach, the algorithm is much more robust; even with mediocre initial exploration, the iterative feedback loop allows it to recover and find a high-performing policy.

Key Results

  • Superior Performance in Noise: In the helicopter hovering task with a 1-step delay (which introduces unmodeled high-order dynamics), DAgger significantly outperformed Batch learning. DAgger converged to a stable, low-cost policy, while Batch methods often failed to stabilize the helicopter.
Performance comparison showing DAgger methods achieving lower average total cost compared to Batch methods over sampled transitions.
  • Outperforming the Expert: Interestingly, DAgger eventually learned a policy that outperformed the "expert" controller used for initial data collection. This happened because the expert was based on a local linearization that didn't account for the specific noise and delays present in the simulation, whereas DAgger learned a model tailored to the actual closed-loop behavior.
  • Complex Maneuvers: For the "nose-in funnel," DAgger successfully completed four full rotations. In contrast, the Batch method failed to produce a controller capable of completing the maneuver regardless of the exploration distribution used.

Limitations and Future Work

  • Computational Overhead: The algorithm requires solving an Optimal Control (OC) problem at every iteration. For complex, non-linear models, this can be computationally expensive. The authors suggest using "warm starts" (starting the next optimization from the previous solution) to mitigate this.
  • Exploration Requirements: While more robust than batch methods, the theory still assumes access to a "good" exploration distribution (like an expert) to get started. Finding such a distribution for entirely unknown tasks remains a challenge.
  • Model Class Selection: DAgger guarantees you'll do as well as the best model in your class, but if your model class is fundamentally too simple (e.g., trying to model a car with a single constant), no amount of data aggregation will result in a good policy. Selecting the right "agnostic" model class remains an engineering art.

Summary updated

Abstract

A fundamental problem in control is to learn a model of a system from observations that is useful for controller synthesis. To provide good performance guarantees, existing methods must assume that the real system is in the class of models considered during learning. We present an iterative method with strong guarantees even in the agnostic case where the system is not in the class. In particular, we show that any no-regret online learning algorithm can be used to obtain a near-optimal policy, provided some model achieves low training error and access to a good exploration distribution. Our approach applies to both discrete and continuous domains. We demonstrate its efficacy and scalability on a challenging helicopter domain from the literature.
BibTeX
@inproceedings{ross2012agnostic,
  author = {Stéphane Ross and Drew Bagnell},
  title = {{Agnostic System Identification for Model-Based Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2012},
  url = {https://openreview.net/forum?id=Hy4gssbO-B},
}
Figure