Few-Shot System Identification for Reinforcement Learning

Karim Farid, Nourhan Sakr

Introduction

Reinforcement Learning (RL) has shown remarkable success in simulated environments, yet its deployment in the real world remains challenging due to the "reality gap." A policy optimized for a specific simulation often fails when faced with slight variations in physical parameters, such as mass, friction, or gravity. Traditionally, bridging this gap requires System Identification (SysID)—a process of estimating a system's mathematical parameters through extensive data collection. However, standard SysID is often too slow for dynamic environments where parameters might change on the fly.

This paper introduces a framework for few-shot system identification, enabling model-based RL agents to adapt to novel environments almost instantaneously. By integrating variational inference with Neural Ordinary Differential Equations (NODEs), the proposed "oracle" can identify a system's underlying physical properties from just a few frames of observation, facilitating robust control without intensive retraining.

Methodology

The core of the proposed framework is the Variational Controlled Neural ODE (VCNODE). This architecture is designed to disentangle the instantaneous state of a system from its underlying dynamics.

Disentangled Embeddings and Neural ODEs

The system utilizes a Recurrent Neural Network (RNN) encoder to process a short history of state-action pairs. This encoder generates a probability distribution over the dynamics parameters, effectively performing "few-shot" identification. Unlike standard discrete-time models, VCNODE employs Neural ODEs to model transitions. This continuous-time approach is inherently better suited for physical systems, allowing the model to handle irregular sampling and provide a more natural representation of motion.

Comparison of the standard VRNN architecture (a) and the proposed time-variant dynamics embedding architecture (b).

Latent Linearization and Efficient Control

A significant innovation in this work is Latent Linearization. The model "lifts" complex, non-linear dynamics into a latent space where transitions are approximately linear ($x_{t} \approx A x_{t-1} + B u_{t-1}$). This linearization allows the use of efficient Model Predictive Control (MPC) solvers, specifically those utilizing Quadratic Programming (QP). By bypassing the need for computationally expensive non-linear optimization at every timestep, the framework achieves high-frequency control suitable for real-time applications.

Experimental Evaluation

The framework was evaluated on two distinct benchmarks:

  1. Toy Spiral Dataset: Consisting of 12,000 trajectories of synthetic spiral motion with randomized matrices, testing the model's ability to infer global dynamics from partial trajectories.
  2. OpenAI Gym Classic Control: The Pendulum Swing-up and Cart-pole environments were modified to include randomized physical parameters (mass, length, gravity) varying by ±30% across 10,000 instances.

Key Results

The VCNODE framework demonstrated superior performance in both accuracy and control robustness. In the pendulum task, the model achieved a Relative Root Mean Squared Error (RMSE) of 0.0016, significantly outperforming baseline variational autoencoder (VAE) approaches.

Progression of pendulum states (cosθ, sinθ, and angular velocity ω) over time, illustrating the model's predictive accuracy.

Furthermore, the framework successfully solved the pendulum swing-up across the randomized test suite with a 95.32% success rate. Efficiency was a key highlight: using a simple Euler solver, inference required only 3ms, well within the requirements for high-frequency robotic control. Even with more sophisticated solvers like ODE45, inference times remained practical at approximately 60ms.

Conclusion and Future Work

While highly effective for low-dimensional classic control problems, the authors identify scaling to high-dimensional state spaces—such as humanoid robotics or raw pixel input—as a primary challenge for future research. Future work will also explore "lifted structure loss" to further refine latent embeddings and the integration of separate temporal encoders for high-dimensional sequential observations.

Summary updated

Abstract

Learning by interaction is the key to skill acquisition for most living organisms, which is formally called Reinforcement Learning (RL). RL is efficient in finding optimal policies for endowing complex systems with sophisticated behavior. All paradigms of RL utilize a system model for finding the optimal policy. Modeling dynamics can be done by formulating a mathematical model or system identification. Dynamic models are usually exposed to aleatoric and epistemic uncertainties that can divert the model from the one acquired and cause the RL algorithm to exhibit erroneous behavior. Accordingly, the RL process loses its generality because it is sensitive to operating conditions and changes in the model parameters. To address these problems, Intensive system identification for modeling purposes is needed for each system even if the model dynamics structure is the same, as the slight deviation in the model parameters can render the model useless in RL. The existence of an oracle that can adaptively predict the rest of the trajectory regardless of the uncertainties can help resolve the issue. The target of this work is to present a framework for facilitating the online system identification of different instances of the same dynamics class by learning a probability distribution of the dynamics conditioned on observed data with variational inference and show its reliability in robustly solving different instances of control problems without extra training in model-based RL with maximum sample efficiency.
BibTeX
@inproceedings{farid2021few,
  author = {Karim Farid and Nourhan Sakr},
  title = {{Few-Shot System Identification for Reinforcement Learning}},
  booktitle = {ACIRS},
  year = {2021},
  url = {https://openreview.net/forum?id=_nmrdcyBJn},
}
Figure