When training a reinforcement learning (RL) agent or control policy, it is tempting to start every training episode from the exact same configuration. In a simulation, we have access to "oracle resets"—the ability to instantly teleport an agent back to a perfect, pristine starting state $s_0$.
However, optimizing a policy from a single fixed starting point introduces a severe vulnerability: the agent learns to optimize for a narrow "tube" of trajectories. Because the state encountered at time $t$ is highly correlated with the sequence of actions that came before it, the policy can cheat. Instead of learning the underlying physics or rules of the environment, it memorizes open-loop action sequences or path-dependent trajectories. The moment the agent encounters a minor external perturbation, it is pushed outside of this narrow tube into unfamiliar state space, leading to catastrophic failure.
To build truly autonomous and robust systems, researchers across continuous control, robotics, and game-playing agents broaden the distribution of initial states during training. By intentionally initializing episodes from diverse, randomized, or off-nominal states, we force policies to learn closed-loop recovery behaviors rather than static path reproduction.
The Illusion of Competence from Fixed Starts
The brittleness of fixed-state initialization is well-documented across diverse control tasks. In early work on simulated car racing, Togelius et al. discovered that evaluating and training controllers from varied starting positions and orientations was essential to prevent the optimization process from converging on fragile, open-loop sequences that only work for a single trajectory. Without this variation, the evolved strategies failed to generalize to any deviation from the ideal racing line.
Similarly, in robotic navigation, Walter demonstrated that training neural controllers from fixed starting positions results in path memorization. Introducing stochasticity into the initial conditions forces the learning process to prioritize generalized obstacle avoidance behaviors over specific trajectory recall.
This phenomenon creates an illusion of competence. An agent trained on a narrow range of initial states can achieve exceptionally high performance scores within its training distribution. However, as Tao et al. showed in locomotion tasks, these high-scoring policies fail to generalize to even minor environmental perturbations. Broadening the distribution of initial states during training initially increases the learning difficulty, but it is a necessary design choice to force the agent to learn robust recovery behaviors from diverse poses.
Narrow Initialization:
[s_0] ----> [ s_1 ] ----> [ s_2 ] ----> [ Goal ] (Fragile trajectory tube)
| (Perturbation)
v
[Unfamiliar State] ----> Failure
Broad/Randomized Initialization:
[s_random] ---\
[s_random] ----+---> [ Robust Closed-Loop Policy ] ----> [ Goal ]
[s_random] ---/
Why It Works: Forcing Recovery Behaviors
Broadening the initial state distribution fundamentally alters what the policy learns. When an agent is dropped into an arbitrary, sub-optimal, or "off-nominal" state, it cannot rely on a pre-baked plan. It must evaluate the current state intrinsically and figure out how to steer back toward a successful path.
Consider the task of autonomous vehicle drifting at the handling limits. Zhou et al. demonstrated that randomizing the starting position and velocity of the vehicle during training is critical for developing robust control policies. Without this initial state randomization, the controller cannot correct for real-world deviations from the reference trajectory and quickly spins out. By experiencing off-nominal states during training, the policy explicitly learns how to recover.
We see the same mechanism in physics-based character animation. Peng et al. integrated randomized initial states—such as starting a simulated character in a falling position—into their training framework. This forced the learning agent to develop robust self-righting skills. Crucially, these learned recovery behaviors become foundational motor primitives that make the character highly resilient when executing downstream tasks in complex environments.
This principle applies equally well to discrete spatial puzzles and games. In the game of Tetris, approximate dynamic programming historically struggled to match simpler optimization methods. Gabillon et al. resolved this by ensuring the rollout distribution included states of varying board heights and difficulties. This forced the agent to learn recovery strategies from sub-optimal, cluttered board configurations rather than only optimizing for ideal, clean slates.
Generalization Beyond Continuous Control
While highly intuitive in robotics and physics, initial state randomization is a universal mechanism for breaking memorization in abstract reasoning and multi-agent coordination.
Evaluating Language Models as Agents
As Large Language Models (LLMs) are increasingly deployed as interactive agents, evaluating them from static starting points risks measuring memorized text sequences rather than active reasoning. To address this, the SmartPlay benchmark introduced by Wu et al. initializes environments from random intermediate states. This forces the models to generalize their understanding of rules to arbitrary configurations within the state space, exposing the "agentic gap" between rote memorization and true planning.
Multi-Agent Coordination
In cooperative multi-agent settings, agents often co-adapt by learning highly specific, brittle communication or movement sequences. Gessler et al. showed that the popular Overcooked benchmark primarily tests state robustness rather than actual teamwork. By implementing random variations in agent starting positions and orientations in OvercookedV2, the authors prevented agents from relying on memorized, open-loop coordination sequences, forcing the development of truly adaptive, reactive policies.
Value Function Stability
In deep Q-learning, agents often suffer from overoptimism and value overestimation. To combat this in Double DQN, van Hasselt et al. initiated training and evaluation episodes from states sampled from human play or after a random number of initial no-op actions. This simple injection of initial state diversity ensures that the value function generalizes smoothly across the state space instead of overfitting to a single deterministic starting sequence.
Structuring the Initial Distribution
If broadening the initial state distribution is so effective, how should we structure it? Simply injecting massive uniform noise can make the learning problem too difficult, causing the policy to fail to converge entirely (Kumar et al.). Researchers use several structured paradigms to manage this distribution.
1. Autonomous RL (No Resets)
The most natural way to experience a broad initial state distribution is to eliminate oracle resets entirely. In Autonomous Reinforcement Learning (ARL), the agent must operate continuously without human intervention or teleportation. As Sharma et al. and Zhu et al. argue, training without external resets forces the agent to navigate back to goal regions from diverse, unintended, and often sub-optimal parts of the state space. This organic exposure to initialization noise naturally builds a global understanding of the environment and yields highly robust policies.
2. Subgoal and Variance Curricula
To prevent the agent from being overwhelmed by high-variance initial states early in training, we can build a curriculum. Sharma introduced VaPRL, which automatically generates a curriculum of starting positions that gradually expands from the goal region back to the initial state distribution. Similarly, Levy et al. incrementally increased the variance of starting and goal states around a single expert demonstration. This incremental expansion allows the agent to bootstrap its learning, ensuring it always knows how to navigate back to a successful path from increasingly unfamiliar positions.
3. Targeted Critical States
Instead of uniform randomization, we can target states where policy robustness matters most. Cheng et al. used explanation methods to identify critical decision points where a pre-trained agent frequently fails. By restarting training from a balanced mixture of these high-impact critical states and standard initial states, the agent is forced to re-evaluate high-impact decisions without overfitting to specific failure cases.
Ultimately, whether through simple noise injection, autonomous exploration, or structured curricula, broadening the initial state distribution shifts the learning objective from path memorization to space coverage. By forcing agents to master the art of recovery, we obtain policies that remain stable when the real world pushes back.
Key Foundational Work
Evolving controllers for simulated car racing (2006) — contributed 1 concept elements to this direction.
Approximate Dynamic Programming Finally Performs Well in the Game of Tetris (2013) — contributed 1 concept elements to this direction.