In reinforcement learning (RL) and control theory, the "rules of the game" are typically divided into two parts: the reward function, which tells an agent what to achieve, and the transition dynamics, which describe how the environment responds to actions. A significant body of research operates under the known transition dynamics assumption, where the agent is provided with the underlying physics or probability matrices of the environment a priori.
While this assumption simplifies the mathematical landscape, it creates a fundamental gap between theoretical planning and real-world deployment. Understanding why researchers rely on this assumption—and where it fails—is central to modern sequential decision-making.
The Utility of a Known World
The primary reason for assuming known dynamics is computational and analytical tractability. When the transition function $P(s'|s, a)$ is known, the agent does not need to explore to understand the environment's "physics"; it only needs to optimize its behavior within those rules.
For example, in Inverse Reinforcement Learning (IRL), the goal is to recover a reward function from expert demonstrations. Boularias and Chaib-draa (2010) explain that knowing the dynamics allows an agent to "fill in the gaps" of an expert’s trajectory, inferring what the expert would have done in states they never actually visited. Without this knowledge, as Dimitrakakis and Rothkopf (2011) note, the observer cannot easily distinguish whether a demonstrator's action was due to their intent (the reward) or the constraints of the environment (the dynamics).
In planning scenarios, known dynamics enable algorithms to "look ahead." Shoham (2021) demonstrates this in complex puzzles like Sokoban, where a known model allows for backward-looking agents to provide hints to forward-looking ones. Similarly, El Shar and Jiang (2020) use known transitions to "sandwich" value functions between upper and lower bounds, significantly accelerating convergence.
Guarantees for Robustness and Risk
The assumption is particularly prevalent in research focused on safety and robustness. To guarantee that a robot will not fall or that a system will remain stable under pressure, researchers often require the system's "matrices"—the mathematical description of its motion.
- Robust Control: Zhang et al. (2020) show that policy optimization can globally solve robust control problems, but their framework assumes full knowledge of system matrices to compute gradients.
- Risk Sensitivity: When optimizing for metrics like Value-at-Risk (VaR), Marthe et al. (2025) and Chow et al. (2015) highlight that current frameworks rely on perfectly known dynamics to calculate the "Optimality Front" or perform value iteration.
- Privacy: Even in privacy research, Russo and Proutiere (2021) argue that protecting the privacy of system transitions requires complete knowledge of the transition probabilities to ensure an eavesdropper cannot detect abrupt changes.
Breaking the Assumption: The Model-Free Gap
The transition from "known dynamics" to "unknown dynamics" is not just a change in parameters; it is a change in the fundamental complexity of the problem. When dynamics are unknown, the agent faces the exploration-exploitation trade-off in its most difficult form: it must spend time learning how the world moves before it can even begin to optimize its rewards.
Cohen et al. (2018) point out that in online Linear Quadratic control, moving from known to unknown dynamics introduces a requirement to estimate transition matrices online, which complicates regret bounds. Hashemi and Moon (2024) further observe that when dynamics are unknown, the error in estimating the "physics" of the environment often dominates the performance, overshadowing the benefits of advanced reward predictors.
This limitation is a recurring theme across diverse subfields:
- Resource Allocation: Mahajan and Akbarzadeh (2022) note that calculating priority indices for restless bandits usually assumes known transitions, leaving the "model-free" version of the problem largely open.
- Conformal Prediction: Kuipers et al. (2024) explain that providing safety "uncertainty bubbles" for multi-agent systems requires precise knowledge of transition probabilities to correct for off-policy data.
Towards Model-Free Robustness
Recent work has begun to bridge this gap by developing algorithms that require less prior knowledge. Derman et al. (2023) introduced "Twice Regularized MDPs," which achieve robustness without the expensive optimization loops that typically require a known nominal transition matrix. Similarly, Shani (2020) provided some of the first theoretical regret guarantees for policy optimization in settings where transitions are entirely unknown and feedback is limited.
The field is gradually moving away from the "known dynamics" crutch, seeking methods that can learn the transition model and the optimal policy simultaneously, or bypass the need for an explicit model altogether while maintaining the rigorous guarantees of the model-based era.