Inverse optimal control from incomplete trajectory observations

Wanxin Jin, Dana Kulic, Shaoshuai Mou, Sandra Hirche

TL;DR

This paper introduces a novel "recovery matrix" to enable learning an agent's objective function from incomplete trajectory data, a significant limitation of prior methods. This approach allows for efficient, incremental learning and can even identify irrelevant features, making it more practical for real-world applications where full trajectory data is often unavailable.

What Problem Does This Paper Address?

Inverse Optimal Control (IOC) aims to infer an agent's underlying objective function (e.g., cost or reward) by observing its behavior. A common assumption in existing IOC methods is that the entire optimal trajectory of the agent's states and actions over a complete time horizon is available. However, in many real-world scenarios, such as due to sensor limitations, failures, or occlusions, only segments or partial observations of the trajectory are accessible. This gap prevents existing IOC techniques from being applied effectively. This paper addresses this challenge by developing a method to learn objective functions from these incomplete trajectory observations.

Methods

The core contribution of this paper is the recovery matrix. This matrix, defined for any trajectory segment and a set of candidate features, establishes a linear constraint on the unknown objective function weights and the system's costates (which represent unseen future information).

Key aspects of the methodology include:

  • Recovery Matrix Definition: A novel matrix is formulated that captures the relationship between observed trajectory segments and candidate features.
  • Rank-based Sufficiency: The rank of the recovery matrix is shown to be a critical indicator of whether the available incomplete trajectory data is sufficient to uniquely determine the objective function weights.
  • Incremental Learning: An incremental IOC algorithm is proposed that automatically determines the minimal required observation length to achieve a successful estimate. This is achieved by iteratively updating the recovery matrix as more data becomes available and checking its rank.
  • Handling Unseen Future: The recovery matrix explicitly accounts for the "unseen future" information through costate variables, allowing it to work with partial observations where traditional methods would fail.
  • Feature Selection: The framework naturally allows for identifying and discarding irrelevant features from the candidate set.

Datasets

Not discussed in the paper. The paper uses simulated systems (LQR and a two-link robot arm) for evaluation.

Key Insights and Ideas

  • The "Recovery Matrix" is the Key: This novel mathematical construct is the central innovation, bridging the gap between incomplete trajectory data and objective function learning.
  • Rank as a Sufficiency Indicator: The rank of the recovery matrix directly tells you if you have "enough" information from your partial observation to learn the objective function. This is a powerful diagnostic tool.
  • Unseen Future is Accounted For: By incorporating costates, the method doesn't ignore what happens after the observed segment, which is crucial for learning from partial data.
  • Incremental Learning is Efficient: The algorithm can start learning from any point and automatically figure out how much more data it needs, making it adaptive and potentially more computationally efficient.
  • Irrelevant Features Don't Break It: The method can handle candidate feature sets that include features not actually present in the true objective function, and it can even identify them.

Key Results

The paper demonstrates the effectiveness of its proposed method through numerical experiments on an LQR system and a simulated two-link robot arm.

  • Successful Learning from Incomplete Data: The proposed method successfully learns objective function weights from trajectory segments, outperforming existing inverse-KKT methods that require full trajectories.
  • Minimal Observation Length: The incremental algorithm automatically determines the minimal observation length required for successful estimation, which is significantly shorter than using the full trajectory in many cases. For example, in the LQR experiment, successful estimation was achieved with approximately 4 data points out of a total horizon of 50.
  • Robustness to Noise: The method shows robustness to Gaussian noise in the trajectory data, with increased observation length compensating for noise to maintain estimation accuracy.
  • Handling Irrelevant Features: Including irrelevant features in the candidate set increases the required observation length but does not significantly degrade estimation accuracy.
  • Infinite-Horizon Control: The method is shown to be applicable to infinite-horizon optimal control problems.

Limitations and Future Work

  • Assumption of Candidate Features: The method assumes that the true relevant features are a subset of the provided candidate features (Assumption 1). If relevant features are entirely missing from the candidate set, learning will fail.
  • Near-Optimality and Noise: While the method is robust to noise, the paper acknowledges that practical implementation requires careful handling of data noise, near-optimality of demonstrations, and computational errors, often through setting a rank threshold parameter ($\gamma$).
  • Computational Cost: While incremental, the computation of the recovery matrix itself can still be demanding for very high-dimensional systems or long trajectories.
  • Future Work: The paper suggests exploring how to further improve estimation accuracy by adjusting the rank threshold parameter ($\gamma$) and investigating the method's applicability to more complex, non-linear, and high-dimensional systems.

Summary updated

Abstract

This article develops a methodology that enables learning an objective function of an optimal control system from incomplete trajectory observations. The objective function is assumed to be a weigh...
BibTeX
@article{jin2021inverse,
  author = {Wanxin Jin and Dana Kulic and Shaoshuai Mou and Sandra Hirche},
  title = {{Inverse optimal control from incomplete trajectory observations}},
  journal = {Int. J. Robotics Res.},
  year = {2021},
  doi = {10.1177/0278364921996384},
}
Figure