Model-based Policy Optimization using Symbolic World Model

Andrey Gorodetskiy, Konstantin Mironov, Aleksandr Panov

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in complex control tasks, yet it remains notoriously data-hungry, often requiring millions of interactions with an environment to converge on an optimal policy. Model-Based Reinforcement Learning (MBRL) offers a solution by learning a "world model"—a predictive simulator of the environment's dynamics—which allows an agent to supplement real-world experience with "imaginary" rollouts.

Traditionally, these world models are implemented as deep neural networks. While flexible, neural networks are high-parameter "black boxes" that often struggle to extrapolate beyond their training distribution and lack human interpretability. This paper introduces a novel MBRL framework that replaces these black-box models with transparent symbolic expressions. By leveraging transformer-based symbolic regression, the authors achieve superior sample efficiency and provide models that are both physically grounded and human-readable.

Methodology

The proposed framework integrates Symbolic Regression (SR) into the Model-Based Policy Optimization (MBPO) pipeline. The architecture consists of four primary stages:

1. Transformer-based Symbolic Regression

Traditional symbolic regression often relies on genetic algorithms, which are computationally expensive and slow. This framework utilizes a pre-trained Transformer model to treat equation discovery as a sequence-to-sequence translation task. The model takes observed state transitions as input and generates mathematical formulas that describe the underlying dynamics.

2. The Symbolic World Model

The environment's transition dynamics are represented as a collection of symbolic expressions, where each coordinate of the state vector is modeled by a distinct equation. This representation is inherently compact compared to neural networks.

3. Numerical Refinement via BFGS

To ensure the discovered equations are numerically precise, the authors employ the Broyden–Fletcher–Goldfarb–Shanno (BFGS) optimization algorithm. This step fine-tunes the constants within the Transformer-generated formulas to minimize the error between predicted and observed state transitions.

4. Policy Training

With the symbolic world model established, the agent’s policy is trained using the Soft Actor-Critic (SAC) algorithm. The agent learns primarily within the "imaginary" environment defined by the symbolic equations, significantly reducing the need for real-world samples.

Experimental Evaluation

The method was evaluated on three continuous control simulation environments within the MuJoCo physics engine:

  • Pendulum: A classic control problem requiring the agent to swing a pole into an upright position.
  • Reacher-v2: A two-link planar manipulator task where the tip must reach a target location.
  • Car2d: A navigation task requiring a car to reach a target on a 2D plane.

The Transformer model used for SR was pre-trained on a massive synthetic dataset of mathematical functions, enabling it to generalize to the physics-based dynamics of these environments without task-specific training.

Key Insights and Results

Superior Sample Efficiency

In benchmarks against model-free SAC and the state-of-the-art Dreamer-v3 algorithm, the symbolic approach demonstrated significantly higher sample efficiency. In the Pendulum task, for instance, the symbolic method reached performance thresholds much earlier than Dreamer-v3, which required approximately 40,000 transitions to achieve comparable results.

Extrapolation and Robustness

Because symbolic expressions can capture the fundamental functional forms of physics (e.g., trigonometric functions in pendulum dynamics), they generalize better than neural networks. When the agent encounters states slightly outside the initial training distribution, the symbolic model maintains higher predictive accuracy. The inclusion of BFGS optimization further boosted prediction quality, increasing the $r^2$ metric by approximately 0.2 across various candidates.

Interpretability and Safety

The symbolic world model allows for direct human inspection. In the Pendulum task, the model successfully recovered the actual equations of motion. This transparency is a critical safety feature, allowing engineers to verify the physical soundness of a model before deploying a policy in sensitive real-world applications.

Limitations and Future Work

While the results are promising, several challenges remain for scaling symbolic world models:

  • High Dimensionality: The search space for symbolic expressions grows exponentially with the number of state variables, making high-DOF (degrees of freedom) robots difficult to model.
  • Inference Constraints: The Transformer’s fixed context window limits the number of historical data points it can process simultaneously.
  • Coordinate Independence: Currently, each state coordinate is modeled independently. Future iterations could benefit from modeling coupled dynamics where sub-expressions are shared across different coordinates.
  • Model Plasticity: The current framework requires re-inferring the model rather than updating it incrementally, which can lead to discontinuities in the world model during the training process.

Conclusion

By bridging the gap between classical symbolic physics and modern reinforcement learning, this framework offers a path toward more efficient, robust, and interpretable AI. The transition from black-box neural networks to transparent symbolic equations marks a significant step in making MBRL viable for complex, real-world engineering tasks.

Summary updated

BibTeX
@inproceedings{gorodetskiy2024model,
  author = {Andrey Gorodetskiy and Konstantin Mironov and Aleksandr Panov},
  title = {{Model-based Policy Optimization using Symbolic World Model}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801405},
}
Figure