Data-driven Koopman operators for model-based shared control of human-machine systems

Alexander Broad, Ian Abraham, Todd D. Murphey, Brenna Argall

TL;DR

This paper introduces Model-based Shared Control (MbSC), a framework that uses data-driven Koopman operators to assist humans in controlling complex, nonlinear systems without needing a predefined mathematical model of the machine or the user. By learning dynamics directly from observation, the system can provide effective, real-time assistance that generalizes across different users and can even be learned "on the fly" in under two minutes of interaction.

What Problem Does This Paper Address?

Shared control aims to combine human intuition with robotic precision. However, most existing shared control systems rely on an a priori (pre-existing) model of the system’s dynamics. This is a major hurdle because:

  1. Complexity: Many systems (like exoskeletons or high-performance vehicles) have nonlinear dynamics that are difficult to model analytically.
  2. The Human Element: Humans interact with machines in idiosyncratic ways. Modeling the "joint" human-machine system is notoriously hard.
  3. Generalization: A model tuned for one user might not work for another, leading to a "cold start" problem where a system needs hours of data before it becomes helpful.

The authors tackle these issues by asking: Can we learn the dynamics of a human-machine system purely from data and use that model to provide helpful assistance in real-time?

Methods

The authors' approach, MbSC, consists of three main components:

  1. Model Learning (The Koopman Operator): The core innovation is using the Koopman operator to represent nonlinear dynamics. While the system's physics are nonlinear, the Koopman operator maps these states into a higher-dimensional "Hilbert space" where the evolution of the system becomes linear. They use Extended Dynamic Mode Decomposition (EDMD) to approximate this operator from snapshots of user interaction data.
  2. Policy Generation (MPC/SAC): Once they have a linear-in-feature model, they use it within an optimal control framework. For the nonlinear variant, they use Sequential Action Control (SAC), a real-time algorithm that finds short bursts of control to maximize performance based on the learned model.
  3. Control Allocation (Maxwell’s Demon): To merge human and autonomous input, they use the Maxwell’s Demon Algorithm (MDA). This is a "lenient" filter: it only allows human input if it moves the system in a direction that aligns with the autonomous "optimal" solution (i.e., they are in the same half-plane). Crucially, the autonomy never adds its own force; it only filters out "bad" human inputs.

Datasets

The researchers conducted two human-subject studies with a total of 32 participants (16 per study).

  • Environment: A modified version of the OpenAI Gym Lunar Lander (simulated in Box2D). This environment is notoriously difficult for novices due to its unstable, low-gravity dynamics.
  • Data Collection: They recorded state variables (position, heading, velocities) and control inputs (main and side thrusters) at 10Hz.
  • Conditions: They compared "User Only" control against three types of MbSC models:
  • Individual: Learned from the current user's own data.
  • General: Learned from three other novice users.
  • Expert: Learned from a highly experienced pilot.

Key Insights and Ideas

  • Linearity is not enough: The study compared a linear basis (LQR-based) with a nonlinear basis (SAC-based). While the linear version showed a positive trend, only the nonlinear variant provided a statistically significant boost in task success. This highlights that for complex tasks like landing a rocket, capturing nonlinearities is essential.
  • Models Generalize Surprisingly Well: One of the most important findings was that the "General" and "Expert" models performed just as well as the "Individual" models. This suggests that the Koopman operator captures the fundamental physics of the human-machine interaction, meaning we don't necessarily need to train a new model for every single user.
  • Extreme Data Efficiency: In the online learning trials, the system became helpful after only 5 trials (roughly 90 seconds of data). This is significantly faster than many deep reinforcement learning approaches, which often require thousands of iterations.
  • Ergodicity as a Metric: Beyond just "success vs. failure," the authors used ergodicity to show that MbSC helps users spend more time in "desirable" states (near the goal) even when they don't ultimately succeed in the landing.

Key Results

  • Success Rates: In the nonlinear study, users under MbSC achieved significantly higher success rates (approx. 60-80% depending on the model) compared to the "User Only" condition (approx. 10-20%).
  • Agreement: Users "agreed" with the autonomous controller (i.e., their input was accepted by the MDA filter) about 65-70% of the time, regardless of whether the model was trained on their own data or an expert's.
  • Learning Curve: In the online condition, performance started poor (matching "User Only") but converged to the level of offline-trained models within 7 to 8 trials.

Limitations and Future Work

  • Simulation vs. Reality: The study was conducted in a 2D simulation. Real-world systems involve sensor noise, latency, and 3D dynamics that might challenge the Koopman approximation.
  • The "Demon" is Passive: The Maxwell’s Demon filter is safe because it never adds energy, but it can be frustrating if it constantly rejects user input without explaining why. Future work could explore "haptic" feedback to guide the user toward the optimal half-plane.
  • Basis Function Selection: While EDMD is powerful, it still requires choosing a set of basis functions (e.g., polynomials, sines, cosines). The authors suggest that automated "Sparsity Promoting" methods could help the system choose these functions automatically in the future.

Summary updated

Abstract

We present a data-driven shared control algorithm that can be used to improve a human operator's control of complex dynamic machines and achieve tasks that would otherwise be challenging, or impossible, for the user on their own. Our method assumes no a priori knowledge of the system dynamics. Instead, both the dynamics and information about the user's interaction are learned from observation through the use of a Koopman operator. Using the learned model, we define an optimization problem to compute the autonomous partner's control policy. Finally, we dynamically allocate control authority to each partner based on a comparison of the user input and the autonomously generated control. We refer to this idea as model-based shared control (MbSC). We evaluate the efficacy of our approach with two human subjects studies consisting of 32 total participants (16 subjects in each study). The first study imposes a linear constraint on the modeling and autonomous policy generation algorithms. The second study explores the more general, nonlinear variant. Overall, we find that model-based shared control significantly improves task and control metrics when compared to a natural learning, or user only, control paradigm. Our experiments suggest that models learned via the Koopman operator generalize across users, indicating that it is not necessary to collect data from each individual user before providing assistance with MbSC. We also demonstrate the data-efficiency of MbSC and consequently, it's usefulness in online learning paradigms. Finally, we find that the nonlinear variant has a greater impact on a user's ability to successfully achieve a defined task than the linear variant.
BibTeX
@article{broad2020data,
  author = {Alexander Broad and Ian Abraham and Todd D. Murphey and Brenna Argall},
  title = {{Data-driven Koopman operators for model-based shared control of human-machine systems}},
  journal = {Int. J. Robotics Res.},
  year = {2020},
  url = {https://openreview.net/forum?id=AkuWfhGj9V},
}

Loading related content...

Figure