Abstract
Reinforcement Learning (RL) agents often struggle with distribution shifts in environment dynamics. This paper introduces the Dimensionless Markov Decision Process (Π-MDP), a framework that leverages the Buckingham-Π theorem from fluid mechanics to create scale-invariant policies. By transforming physical states and actions into dimensionless groups, the Π-MDP allows for zero-shot generalization across varying physical parameters such as mass, length, and gravity.
Introduction
A persistent challenge in Reinforcement Learning is the brittleness of learned controllers when faced with changes in physical scale. A policy trained to balance a one-meter pole typically fails when applied to a two-meter pole, despite the underlying physical laws remaining identical. While techniques like domain randomization attempt to bridge this gap, they often require extensive training data and can lead to sub-optimal, overly conservative behavior. This work proposes a more principled approach: embedding the rules of dimensional analysis directly into the MDP framework to achieve inherent equivariance to physical scaling.
Methods: The Π-MDP Framework
The core contribution of this work is the application of the Buckingham-Π theorem to the MDP structure. The theorem states that any physically meaningful equation involving $d$ variables with $k$ independent fundamental dimensions can be rewritten in terms of $d-k$ dimensionless parameters, or Π-groups.
In a Π-MDP, the agent utilizes a context $c$ (representing physical constants like mass $M$ or length $L$) to transform the raw state $s_t$ into a dimensionless state $s_{Π,t}$. The policy $\pi_\Pi$ operates entirely within this dimensionless space, outputting a dimensionless action $a_{Π,t}$. This action is then re-scaled using the context to produce the physical control signal $a_t$. This architecture ensures that the agent's internal representation remains invariant to the absolute scale of the system. The authors integrated this framework into Π-PILCO, a model-based RL algorithm using Gaussian Processes for high data efficiency.
Experimental Results
The researchers evaluated the Π-MDP framework on classic continuous control tasks, including the underactuated pendulum and the cartpole balancer. The primary metric for evaluation was the "Controllable Area"—the region in the parameter space (e.g., mass vs. length) where the controller maintains near-optimal performance.
As shown in the heatmaps, the "natural" controller trained on standard physical units exhibits a very narrow range of success, failing quickly as parameters deviate from training values. In contrast, the Buckingham controller demonstrates massive generalization gains, maintaining high success rates across a wide spectrum of physical contexts. This zero-shot transfer capability highlights the power of using physics-based inductive biases to improve robustness.
Discussion and Limitations
The Π-MDP framework provides a robust method for achieving scale-invariant control. However, it currently assumes that the physical dimensions of state/action variables are known and that the context parameters are observable at test time. Furthermore, while effective for second-order systems, further research is required to assess performance in environments with complex, non-conservative forces like non-linear friction or air resistance. Future work may focus on learning these dimensionless transformations directly from data to broaden the framework's applicability to high-dimensional sensor inputs.
Summary updated