Introduction
Reinforcement Learning (RL) has shown remarkable success in simulated environments, yet its deployment in the real world remains challenging due to the "reality gap." A policy optimized for a specific simulation often fails when faced with slight variations in physical parameters, such as mass, friction, or gravity. Traditionally, bridging this gap requires System Identification (SysID)—a process of estimating a system's mathematical parameters through extensive data collection. However, standard SysID is often too slow for dynamic environments where parameters might change on the fly.
This paper introduces a framework for few-shot system identification, enabling model-based RL agents to adapt to novel environments almost instantaneously. By integrating variational inference with Neural Ordinary Differential Equations (NODEs), the proposed "oracle" can identify a system's underlying physical properties from just a few frames of observation, facilitating robust control without intensive retraining.
Methodology
The core of the proposed framework is the Variational Controlled Neural ODE (VCNODE). This architecture is designed to disentangle the instantaneous state of a system from its underlying dynamics.
Disentangled Embeddings and Neural ODEs
The system utilizes a Recurrent Neural Network (RNN) encoder to process a short history of state-action pairs. This encoder generates a probability distribution over the dynamics parameters, effectively performing "few-shot" identification. Unlike standard discrete-time models, VCNODE employs Neural ODEs to model transitions. This continuous-time approach is inherently better suited for physical systems, allowing the model to handle irregular sampling and provide a more natural representation of motion.
Latent Linearization and Efficient Control
A significant innovation in this work is Latent Linearization. The model "lifts" complex, non-linear dynamics into a latent space where transitions are approximately linear ($x_{t} \approx A x_{t-1} + B u_{t-1}$). This linearization allows the use of efficient Model Predictive Control (MPC) solvers, specifically those utilizing Quadratic Programming (QP). By bypassing the need for computationally expensive non-linear optimization at every timestep, the framework achieves high-frequency control suitable for real-time applications.
Experimental Evaluation
The framework was evaluated on two distinct benchmarks:
- Toy Spiral Dataset: Consisting of 12,000 trajectories of synthetic spiral motion with randomized matrices, testing the model's ability to infer global dynamics from partial trajectories.
- OpenAI Gym Classic Control: The Pendulum Swing-up and Cart-pole environments were modified to include randomized physical parameters (mass, length, gravity) varying by ±30% across 10,000 instances.
Key Results
The VCNODE framework demonstrated superior performance in both accuracy and control robustness. In the pendulum task, the model achieved a Relative Root Mean Squared Error (RMSE) of 0.0016, significantly outperforming baseline variational autoencoder (VAE) approaches.
Furthermore, the framework successfully solved the pendulum swing-up across the randomized test suite with a 95.32% success rate. Efficiency was a key highlight: using a simple Euler solver, inference required only 3ms, well within the requirements for high-frequency robotic control. Even with more sophisticated solvers like ODE45, inference times remained practical at approximately 60ms.
Conclusion and Future Work
While highly effective for low-dimensional classic control problems, the authors identify scaling to high-dimensional state spaces—such as humanoid robotics or raw pixel input—as a primary challenge for future research. Future work will also explore "lifted structure loss" to further refine latent embeddings and the integration of separate temporal encoders for high-dimensional sequential observations.
Summary updated