Symbolic-Model-Based Reinforcement Learning

Pierre-Alexandre Kamienny, sylvain lamprierAI4Science @ NeurIPS 2022 · OpenReview

TL;DR

This paper introduces Symbolic-Model-Based Reinforcement Learning (Symbolic-MBRL), a novel approach that uses symbolic regression to learn interpretable mathematical expressions for environment dynamics, rather than relying on data-hungry neural networks. This method demonstrates significantly improved sample efficiency and extrapolation capabilities, leading to faster learning and better performance in control tasks.

What Problem Does This Paper Address?

Reinforcement Learning (RL) aims to train agents to make optimal decisions in complex environments. Model-Based Reinforcement Learning (MBRL) is a promising paradigm that tackles this by first learning a model of the environment's dynamics. However, current MBRL methods predominantly use over-parameterized neural networks (NNs) to learn these dynamics models. While NNs are powerful function approximators, they require vast amounts of data to train effectively and are prone to overfitting, especially in low-data regimes. This data inefficiency and lack of interpretability hinder MBRL's application in real-world scenarios where data collection can be expensive or dangerous. The authors observe that many physical systems, common in RL benchmarks, are governed by underlying mathematical laws. They propose to leverage this inherent structure by using Symbolic Regression (SR) to discover these laws as interpretable mathematical expressions, aiming to overcome the limitations of NN-based dynamics models.

Methods

The core innovation is the integration of Symbolic Regression (SR) into the model-learning step of MBRL. Instead of training a neural network to approximate the transition function $f(s_t, a_t) \rightarrow s_{t+1}$, the authors use SR to search for an interpretable mathematical expression within a predefined set of operators (e.g., +, -, *, /, sin, cos, exp) and variables that best fits the observed state transitions.

The proposed framework, dubbed Symbolic-Model-Based RL (Symbolic-MBRL), uses an SR algorithm (specifically, the genetic programming-based Operon) to learn the dynamics model. This learned symbolic model is then used by a standard MBRL algorithm (Probabilistic Ensembles with Trajectory Sampling - PETS) for planning and policy optimization. The authors compare their Symbolic-PETS approach against the standard MLP-PETS.

Datasets

The paper evaluates the proposed method on two environments:

  • Toy Example: A simple one-dimensional state MDP with dynamics defined by $s_{t+1} = s_t + a_t$ and a reward function $r_t = \cos(2\pi s_{t+1}) \exp(|s_{t+1}|/3)$. This environment is used to illustrate the concept and demonstrate the learning of reward functions.
  • CartPole: The continuous version of the classic CartPole environment, where the state includes position, velocity, angle, and angular velocity. This is a standard benchmark for evaluating control algorithms.

For training the dynamics models, the paper uses data generated by a random policy to ensure good state-action coverage, particularly for the initial exploration phase. Evaluation datasets are also collected.

Key Insights and Ideas

  • Leveraging Physical Laws: The paper highlights that many environments in RL are governed by underlying physical laws expressible as mathematical equations. This inherent structure can be exploited for more efficient and robust model learning.
  • SR for Interpretability and Extrapolation: Symbolic regression offers a powerful alternative to neural networks for learning dynamics models. It naturally produces interpretable mathematical expressions and has demonstrated superior extrapolation capabilities on unseen data, a critical advantage for RL agents operating in dynamic or partially explored environments.
  • Sample Efficiency Gains: By learning a more accurate and generalizable model with fewer data points, Symbolic-MBRL significantly reduces the sample complexity required for effective learning, making it more practical for real-world applications.
  • Model Overfitting Mitigation: The tendency of neural networks to overfit in low-data regimes is a major bottleneck. SR, by constraining the model space to structured expressions, inherently avoids this issue, leading to more reliable dynamics models.

Key Results

  • Faster Learning: In the toy example, Symbolic-PETS learns the reward function and achieves better performance with significantly fewer environment interactions compared to MLP-PETS. Figure 2 shows that Symbolic-PETS agents can reach better rewarded states, while MLP-PETS's models overfit the training data.
  • CartPole Performance: Symbolic-PETS solves the CartPole task in as few as 20 environment interactions, achieving state-of-the-art performance. This is achieved despite the MLP-PETS model exhibiting significant error, suggesting that the symbolic model's better generalization is key. Figure 3 demonstrates this rapid convergence.
  • Model Accuracy: Figure 5 shows that the symbolic regressor achieves lower model error than the MLP regressor on data generated by a random policy, especially at lower numbers of environment interactions. This indicates that SR can learn more accurate models with less data.

Limitations and Future Work

  • Scalability to High Dimensions: The authors acknowledge that scaling SR to environments with very high-dimensional state spaces (e.g., pixel observations) remains a challenge. This includes parallelizing the training of regressors for multiple output dimensions.
  • Modularity of SR Tools: Current SR tools are noted as being less modular than neural network frameworks.
  • Handling Uncertainties: Future work aims to design SR algorithms that can express distributions to handle epistemic and aleatoric uncertainties, allow for initialization with known expressions, support multi-step predictions, and preprocess raw pixel inputs.
  • Broader RL Applications: The authors suggest that Symbolic-MBRL's sample efficiency and interpretability could significantly impact areas like partially-observable environments, meta-RL, constrained RL, and Sim2Real transfer.

Summary updated

Abstract

We investigate using symbolic regression (SR) to model dynamics with mathematical expressions in model-based reinforcement learning (MBRL). While the primary promise of MBRL is to enable sample-efficient learning, most popular MBRL algorithms rely, in order to learn their approximate world model, on black-box over-parametrized neural networks, which are known to be data-hungry and are prone to overfitting in low-data regime. In this paper, we leverage the fact that a large collection of environments considered in RL is governed by physical laws that compose elementary operators e.g $\sin{},\sqrt{\phantom{x}}, \exp{}, \frac{\text{d}}{\text{dt}}$, and we propose to search a world model in the space of interpretable mathematical expressions with SR. We show empirically on simple domains that MBRL can benefit from the extrapolation capabilities and sample efficiency of SR compared to neural models.
BibTeX
@inproceedings{kamienny2022symbolic,
  author = {Pierre-Alexandre Kamienny and sylvain lamprier},
  title = {{Symbolic-Model-Based Reinforcement Learning}},
  booktitle = {AI4Science @ NeurIPS},
  year = {2022},
  url = {https://openreview.net/forum?id=yeF6cyYU7W},
}
Figure