Learning to Reach, Swim, Walk and Fly in One Trial: Data-Driven Control with Scarce Data and Side Information

Franck Djeumou, Ufuk Topcu

TL;DR

This paper introduces a learning-based control algorithm that can master unknown dynamical systems—like a walking robot or a diving aircraft—using data from just a single, ongoing trial. By cleverly incorporating "side information" from basic physics, the method achieves performance comparable to state-of-the-art reinforcement learning models that require millions of training interactions.

What Problem Does This Paper Address?

Most modern reinforcement learning (RL) algorithms are "data-hungry," often requiring millions of interactions with an environment to learn a stable control policy. While this works in simulation, it is a deal-breaker for real-world physical systems where trials are expensive, time-consuming, or dangerous.

Imagine an aircraft that suddenly loses an engine. It doesn't have the luxury of "crashing a million times" to learn how to stay airborne; it needs to adapt immediately using only the data from its current, failing trajectory. The authors address this "scarce data" challenge by moving away from pure black-box learning and instead leveraging what we already know about the world (e.g., the laws of physics) to guide the learning process on the fly.

Methods

The authors propose a three-step framework for "on-the-fly" control:

  1. Data-Driven Differential Inclusions: Instead of trying to find one perfect mathematical model for the system, the algorithm constructs a "differential inclusion"—essentially a set of all possible models that are consistent with the observed noisy data and known physics.
  2. Reachable Set Over-Approximation: Using an interval Taylor-based method, the algorithm calculates a "safety envelope" (reachable set) of where the system might go next, given the uncertainty in the model.
  3. Sequential Convex Programming (SCP): To decide on control actions, the algorithm solves an optimization problem. Since the true dynamics are unknown and likely non-convex, it uses SCP to iteratively linearize the problem and solve it within "trust regions" where the approximations are reliable.

The "secret sauce" is Side Information. This includes known Lipschitz constants (bounds on how fast the system changes), rigid-body dynamics structures, or algebraic constraints like friction laws. This information "prunes" the set of possible models, making the reachable sets tighter and the control more precise.

The algorithm's performance on various MuJoCo environments and an F-16 simulator.

Datasets

The researchers evaluated their approach on several challenging benchmarks:

  • MuJoCo Environments: Standard robotics simulations including the Reacher, Swimmer, and Half-Cheetah.
  • F-16 Aircraft Simulator: A high-fidelity, 13-state nonlinear simulation of an F-16 fighter jet.
  • Baselines: They compared their results against high-performing RL algorithms like D4PG and SAC, as well as SINDYc (a popular system identification and control technique).

Key Insights and Ideas

  • Physics as a Prior: You don't need to learn gravity from scratch. By telling the algorithm the basic structure of rigid-body dynamics, you drastically reduce the "search space" for the controller.
  • Tightening the Envelope: The algorithm is designed such that as more data streams in during the trial, the "differential inclusion" shrinks. This means the controller becomes more confident and precise over time.
  • Robustness to Novelty: Because the method relies on reachability analysis rather than just mimicking past data, it can handle abrupt changes (like a simulated engine failure) better than traditional RL.
  • Real-Time Feasibility: Unlike many complex optimization schemes, the SCP approach used here is fast enough to run in real-time, with computation times consistently lower than the system's control frequency.

Key Results

  • Single-Trial Success: In MuJoCo tasks, the algorithm achieved near-optimal rewards within a single episode of 1,000 steps. In contrast, the RL baselines (D4PG and SAC) were pre-trained on 10 million interactions to reach similar performance.
  • F-16 Ground Collision Avoidance: In a high-stakes scenario where an F-16 is diving toward the ground, the proposed algorithm successfully pulled the jet up to avoid a crash. Both the standard LQR controller and the SINDYc baseline failed, resulting in a collision.
  • Data Efficiency: The method proved that incorporating even minimal side information (like the inertial matrix of a robot) is the difference between failing to learn and achieving expert-level control in one go.
Reward curves showing the algorithm matching RL baselines in just one episode.

Limitations and Future Work

While powerful, the method has some noted constraints:

  • Dependency on Side Information: Without at least some structural knowledge (like the inertial matrix in Cheetah), the reachable sets can become too large, causing the controller to fail.
  • Lipschitz Bounds: The current implementation relies on estimated Lipschitz constants. If these are significantly under-estimated, the "safety envelope" might not actually contain the true system behavior.
  • Polynomial Control: The theoretical guarantees currently focus on systems with polynomial dependency on control inputs. Extending this to more arbitrary control structures is a potential future direction.

The authors suggest that future work could involve automatically discovering side information from data or integrating more complex forms of "qualitative" knowledge, such as monotonicity.

Summary updated

Abstract

We develop a learning-based control algorithm for unknown dynamical systems under very severe data limitations. Specifically, the algorithm has access to streaming and noisy data only from a single and ongoing trial. It accomplishes such performance by effectively leveraging various forms of side information on the dynamics to reduce the sample complexity. Such side information typically comes from elementary laws of physics and qualitative properties of the system. More precisely, the algorithm approximately solves an optimal control problem encoding the system's desired behavior. To this end, it constructs and iteratively refines a data-driven differential inclusion that contains the unknown vector field of the dynamics. The differential inclusion, used in an interval Taylor-based method, enables to over-approximate the set of states the system may reach. Theoretically, we establish a bound on the suboptimality of the approximate solution with respect to the optimal control with known dynamics. We show that the longer the trial or the more side information is available, the tighter the bound. Empirically, experiments in a high-fidelity F-16 aircraft simulator and MuJoCo's environments illustrate that, despite the scarcity of data, the algorithm can provide performance comparable to reinforcement learning algorithms trained over millions of environment interactions. Besides, we show that the algorithm outperforms existing techniques combining system identification and model predictive control.
BibTeX
@inproceedings{djeumou2022learning,
  author = {Franck Djeumou and Ufuk Topcu},
  title = {{Learning to Reach, Swim, Walk and Fly in One Trial: Data-Driven Control with Scarce Data and Side Information}},
  booktitle = {L4DC},
  year = {2022},
}
Figure