Generalizing Reinforcement Learning to Unseen Physical Dynamics

This direction is merged into another direction: policy brittleness. Its last version stays readable and citable here.

Version 1 of 142 papers42 conceptsfrozen 2026-10-04

Reinforcement learning (RL) agents are notoriously "brittle." An agent trained to walk in a simulator with a friction coefficient of 1.0 often collapses when deployed on a real-world surface where the friction is 0.9. As noted by Kevin P. Murphy in Reinforcement Learning: An Overview, this lack of robustness to distribution shifts—minor changes in physics, sensor calibration, or environmental geometry—prevents autonomous systems from moving safely from controlled simulations to the unpredictable real world.

The core of the problem is overfitting: the policy learns to exploit the specific, deterministic quirks of its training environment rather than mastering the underlying task. To solve this, researchers are moving away from training single, static policies toward frameworks that anticipate and adapt to environmental variation.

Generalization as an Adversarial Game

One powerful way to build intuition for robustness is to frame generalization as a game. If an agent can succeed even when a "hostile" force is trying to make it fail, it will likely succeed when the real world is simply different.

Xinrun Wang and colleagues in A Game-Theoretic Perspective of Generalization in Reinforcement Learning formalize this as a two-player game between a policy-seeking agent and a task-manipulating adversary. Instead of training on a fixed distribution, the agent faces an adversary that chooses the most difficult environmental parameters (like gravity or mass).

However, standard adversarial training is often unstable. To fix this, Parameswaran Kamalaruban introduced MixedNE-LD, which uses Langevin Dynamics to seek a "mixed" strategy—a distribution of behaviors—rather than a single point. Similarly, Aryaman Reddi proposed QARL, which uses a "bounded rationality" curriculum. By starting with a weak adversary and gradually increasing its "intelligence," the agent learns to handle increasingly difficult physical perturbations without the training process collapsing.

Probing and Rapid Adaptation

A second school of thought argues that we shouldn't expect a single policy to be optimal everywhere. Instead, an agent should be able to "feel out" its environment and adjust its behavior on the fly.

Lerrel Pinto developed Environment Probing Interaction Policies (EPI), where the agent performs a specific set of "probing" actions to identify hidden physical parameters before attempting the main task. This is similar to a human nudging a box to see how heavy it is before trying to lift it.

Once the environment is identified, the agent needs a way to change its strategy. Jean-Baptiste Gaya proposed learning a Line of Policies (LoP)—a continuous subspace of behaviors. At test time, the agent can quickly navigate this subspace to find the specific policy that matches the current friction or gravity. This decoupling of environment representation from policy behavior, also explored by Tong Sang in PAnDR, allows for near-instant adaptation to new dynamics without retraining from scratch.

Implicit Robustness through Entropy and Pessimism

Interestingly, robustness can sometimes be achieved without explicit adversarial training. Benjamin Eysenbach proved in Maximum Entropy RL (Provably) Solves Some Robust RL Problems that simply maximizing the entropy of a policy—making it as random as possible while still achieving the goal—is mathematically equivalent to optimizing against a budget of environmental disturbances. By being "intentionally diverse," the agent implicitly prepares for a range of possible worlds.

In the context of offline data, where we cannot interact with the environment to test its limits, researchers use "pessimism." Yue Wang and Li Xia have developed frameworks for Distributionally Robust Offline RL. These methods assume the deployment environment will be the "worst-case" version of the training data, forcing the agent to learn conservative, high-reliability strategies that are less likely to fail when the transition dynamics shift.

Bridging the Gap

Whether through adversarial curricula, environment probing, or mathematical pessimism, the goal remains the same: moving RL from a "lab curiosity" to a reliable tool. By treating environmental parameters not as constants, but as variables to be navigated, these researchers are building agents that don't just solve a specific simulator, but master the underlying physics of the task itself.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generalizing_reinforcement_learning_to_u_9dd770c8,
  title = {Generalizing Reinforcement Learning to Unseen Physical Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalizing-reinforcement-learning-to-unseen-physical-dynamics/rdr_9dd770c8d7c1423b9445d3da4c1c8a86},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_generalizing_reinforcement_learning_to_u_9dd770c8_v1,
  title = {Generalizing Reinforcement Learning to Unseen Physical Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalizing-reinforcement-learning-to-unseen-physical-dynamics/rdr_9dd770c8d7c1423b9445d3da4c1c8a86/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}