Introduction
In traditional model-based Reinforcement Learning (RL), agents typically learn a "world model" through a supervised loss: the model observes the current state and action, and is penalized if its prediction of the next state deviates from reality. While effective, this approach is computationally expensive and often forces the model to waste capacity on irrelevant environmental details.
However, in nature, biological brains likely evolved predictive capabilities as a byproduct of survival, rather than by optimizing a specific mathematical prediction loss. This paper investigates whether a functional world model can emerge simply as a side effect of solving an RL task under severe sensory constraints, without ever explicitly training the agent to predict the future.
Observational Dropout
To test this hypothesis, the authors introduce a training regime called observational dropout.
At every timestep, the agent is only allowed to observe the "ground truth" environment with a specific probability $p$ (the "peek probability"). When the agent is denied a view of the real world (probability $1-p$), it must rely on the output of its own internal model from the previous step as its current input.
Instead of training the predictive model and the action policy separately, both are trained simultaneously using a population-based version of REINFORCE. Crucially, there is no "prediction error" term in the objective function. The only signal the system receives is the standard task reward. To succeed while effectively "blind" most of the time, the model is forced to learn to generate a sequence of internal observations—hallucinations—that are accurate enough for the policy to make correct decisions.
Experimental Environments
The researchers evaluated this approach across three distinct environments:
- Swing-up Cart-pole: A non-linear control task requiring an agent to swing a hanging pole upright and balance it.
- Grid World: A 2D environment where a stateless agent navigates to collect rewards and avoid penalties.
- Car Racing: A high-dimensional pixel-based environment where a pre-trained Variational Autoencoder (VAE) compresses images into a latent vector for the world model to process.
Key Insights: Function Over Form
The models that emerge from observational dropout are not perfect simulators of reality. Because the policy and model are trained jointly, the model only learns the dynamics strictly necessary for the task, and the policy learns to accommodate the model's specific "hallucinations."
For instance, the model might learn to balance a pole at a slight angle rather than perfectly vertical. This demonstrates a "just enough" principle: the model ignores irrelevant physics (like the forward motion of a car on a straightaway) and focuses on critical information (like upcoming curves).
Furthermore, architectural inductive biases play a massive role in this emergent learning. In Grid World experiments, a simple Convolutional Neural Network (CNN) vastly outperformed a high-capacity Fully Connected model, as the CNN's inherent spatial translation allowed it to easily grasp the logic of movement.
Key Results and Robustness
The most striking result is the agents' robustness to extreme data loss. In the Swing-up Cart-pole task, agents successfully solved the environment even when they only observed the real world 5% to 10% of the time.
Similarly, in the Car Racing environment, the agent maintained successful performance while being blind for 70% of the frames. Remarkably, world models trained via observational dropout (at very low peek probabilities) were often better at training new policies from scratch than models of the same size trained with a standard, explicit forward-prediction loss.
Limitations and Future Work
While observational dropout successfully coerces the emergence of world models, it is not without limitations. Because the model is not explicitly grounded in reality at every step, errors can compound over long "imagined" trajectories, eventually causing the agent's internal state to diverge from the true environment. Additionally, the current reliance on evolutionary strategies like REINFORCE is computationally heavy compared to gradient-based supervised learning. Future work will need to explore how to scale this emergent modeling to highly stochastic, complex multi-object environments while improving optimization efficiency.
Summary updated