Most theoretical guarantees in reinforcement learning (RL) rest on a comforting but fragile assumption: the world is smooth. Mathematically, this is usually expressed as Lipschitz continuity. The intuition is simple: if two states are close together, their optimal values, rewards, and next-state distributions should also be close.
However, as Peter Ramadge and Ting-Han Fan point out in A Contraction Approach to Model-based Reinforcement Learning, this assumption is frequently violated in the real world. A robot balancing on a beam experiences a smooth transition of states until a tiny perturbation causes it to fall—a catastrophic, discontinuous drop in performance. When smoothness breaks, the mathematical "glue" holding RL algorithms together often dissolves.
The Problem with Spikes and Cliffs
In standard RL theory, smoothness allows an agent to generalize. If I know the value of being at coordinates $(x, y)$, I can reasonably guess the value of $(x+0.01, y)$. But many environments are "spiky" or "jumpy."
- Binary and Goal-Based Rewards: In Coarse-Grained Smoothness for RL in Metric Spaces, Kavosh Asadi and colleagues explain that goal-based rewards create sharp discontinuities. A robot gets a reward of 1 if it touches a button and 0 otherwise; the "cliff" between these two states makes the gradient infinite, rendering standard smoothness constants useless.
- Hard Physical Boundaries: Corentin Tallec and Yann Ollivier show in Making Deep Q-learning methods robust to time discretization that hard boundaries and discrete transitions cause instabilities in algorithms designed for continuous-time smoothness.
- Reward Hacking and Corruption: Jason Mancuso identifies "spiky" corruption in Detecting Spiky Corruption in Markov Decision Processes, where false rewards appear as sharp, localized discontinuities that lead agents astray because they lack a mechanism to distinguish "smooth" true rewards from "spiky" noise.
Weakening the Assumption: Hölder and Weak Smoothness
If global Lipschitz continuity is too restrictive, how do we maintain theoretical rigor? Researchers are increasingly turning to weak smoothness or Hölder continuity.
In Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settings, Matthew S. Zhang and Animesh Garg relax the requirement for bounded gradients. By assuming the objective is merely Hölder continuous—meaning it can have "sharper" turns than Lipschitz functions allow—they provide the first convergence guarantees for Policy Gradient methods in environments that better reflect real-world complexity.
Similarly, Alberto Maria Metelli and Marcello Restelli demonstrate in Tight Performance Guarantees of Imitator Policies with Continuous Actions that while value functions in continuous MDPs are rarely Lipschitz, they are inherently Hölder continuous. This shift allows for tighter performance bounds in Behavioral Cloning without pretending the environment is smoother than it actually is.
Algorithmic Solutions for Discontinuous Worlds
Recognizing that the world is non-smooth has led to several practical innovations:
- Branched Rollouts: Peter Ramadge proves that in model-based RL, short "branched" simulations starting from real-world states are mathematically necessary to prevent errors from exploding when dynamics are non-smooth.
- Flipping Policies: For safety-constrained RL, Xun Shen and Sebastien Gros show in Flipping-based Policy for Chance-Constrained Markov Decision Processes that the optimal policy in non-smooth environments often involves a "flip" between two discrete actions rather than a smooth interpolation.
- Wasserstein Metrics: Instead of measuring the "average" difference between states, researchers like Kavosh Asadi and Michael Littman use the Wasserstein distance (Earth Mover's Distance). As shown in Equivalence Between Wasserstein and Value-Aware Model-based Reinforcement Learning, this metric is more robust to the "jumps" found in value-aware model learning.
Why It Works: Intuition over Continuity
The core intuition of this research direction is that local structure matters more than global bounds. While a robot falling is a global discontinuity, the physics leading up to the fall and the physics after the fall are still structured.
By using tools like "zooming" algorithms (Ahmed Touati in Zooming for Efficient Model-Free Reinforcement Learning in Metric Spaces) or risk-sensitive scoring functions (Seunghyun Han in Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions), agents can learn to navigate "jumpy" environments like financial markets or robotics. They do this by acknowledging that while they cannot assume global smoothness, they can still bound their regret by how much the environment deviates from that ideal.