Improving Controller Generalization with Dimensionless Markov Decision Processes

Valentin Charvet, Sebastian Stein, Roderick Murray-SmitharXiv 2025 · DOI · arXiv

Abstract

Reinforcement Learning (RL) agents often struggle with distribution shifts in environment dynamics. This paper introduces the Dimensionless Markov Decision Process (Π-MDP), a framework that leverages the Buckingham-Π theorem from fluid mechanics to create scale-invariant policies. By transforming physical states and actions into dimensionless groups, the Π-MDP allows for zero-shot generalization across varying physical parameters such as mass, length, and gravity.

Introduction

A persistent challenge in Reinforcement Learning is the brittleness of learned controllers when faced with changes in physical scale. A policy trained to balance a one-meter pole typically fails when applied to a two-meter pole, despite the underlying physical laws remaining identical. While techniques like domain randomization attempt to bridge this gap, they often require extensive training data and can lead to sub-optimal, overly conservative behavior. This work proposes a more principled approach: embedding the rules of dimensional analysis directly into the MDP framework to achieve inherent equivariance to physical scaling.

Methods: The Π-MDP Framework

The core contribution of this work is the application of the Buckingham-Π theorem to the MDP structure. The theorem states that any physically meaningful equation involving $d$ variables with $k$ independent fundamental dimensions can be rewritten in terms of $d-k$ dimensionless parameters, or Π-groups.

The Π-MDP framework: Transforming physical states and actions into dimensionless groups.

In a Π-MDP, the agent utilizes a context $c$ (representing physical constants like mass $M$ or length $L$) to transform the raw state $s_t$ into a dimensionless state $s_{Π,t}$. The policy $\pi_\Pi$ operates entirely within this dimensionless space, outputting a dimensionless action $a_{Π,t}$. This action is then re-scaled using the context to produce the physical control signal $a_t$. This architecture ensures that the agent's internal representation remains invariant to the absolute scale of the system. The authors integrated this framework into Π-PILCO, a model-based RL algorithm using Gaussian Processes for high data efficiency.

Experimental Results

The researchers evaluated the Π-MDP framework on classic continuous control tasks, including the underactuated pendulum and the cartpole balancer. The primary metric for evaluation was the "Controllable Area"—the region in the parameter space (e.g., mass vs. length) where the controller maintains near-optimal performance.

Generalization performance (Controllable Area) for natural vs. Buckingham controllers across varying mass (M) and length (L).

As shown in the heatmaps, the "natural" controller trained on standard physical units exhibits a very narrow range of success, failing quickly as parameters deviate from training values. In contrast, the Buckingham controller demonstrates massive generalization gains, maintaining high success rates across a wide spectrum of physical contexts. This zero-shot transfer capability highlights the power of using physics-based inductive biases to improve robustness.

Discussion and Limitations

The Π-MDP framework provides a robust method for achieving scale-invariant control. However, it currently assumes that the physical dimensions of state/action variables are known and that the context parameters are observable at test time. Furthermore, while effective for second-order systems, further research is required to assess performance in environments with complex, non-conservative forces like non-linear friction or air resistance. Future work may focus on learning these dimensionless transformations directly from data to broaden the framework's applicability to high-dimensional sensor inputs.

Summary updated

Abstract

Controllers trained with Reinforcement Learning tend to be very specialized and thus generalize poorly when their testing environment differs from their training one. We propose a Model-Based approach to increase generalization where both world model and policy are trained in a dimensionless state-action space. To do so, we introduce the Dimensionless Markov Decision Process ($Π$-MDP): an extension of Contextual-MDPs in which state and action spaces are non-dimensionalized with the Buckingham-$Π$ theorem. This procedure induces policies that are equivariant with respect to changes in the context of the underlying dynamics. We provide a generic framework for this approach and apply it to a model-based policy search algorithm using Gaussian Process models. We demonstrate the applicability of our method on simulated actuated pendulum and cartpole systems, where policies trained on a single environment are robust to shifts in the distribution of the context.
BibTeX
@misc{charvet2025improving,
  author = {Valentin Charvet and Sebastian Stein and Roderick Murray-Smith},
  title = {{Improving Controller Generalization with Dimensionless Markov Decision Processes}},
  year = {2025},
  eprint = {2504.10006},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.10006},
}
Figure