In the quest to build autonomous agents that can explore complex environments without human guidance, researchers often turn to "curiosity." The standard recipe for curiosity is simple: reward the agent for going to places where its internal model makes poor predictions. If the agent is surprised, it must have found something new to learn.
However, this simple recipe has a fatal flaw known as the Noisy TV problem. Imagine an agent in a room with a television displaying static. Because the static is fundamentally unpredictable, the agent’s prediction error will remain high forever. A "curious" agent will become mesmerized by the screen, wasting its life trying to predict the next random pixel rather than exploring the rest of the world.
To solve this, a growing body of research focuses on disentangling two distinct types of uncertainty: epistemic uncertainty (what we don't know yet but could learn) and aleatoric uncertainty (what is inherently random and unlearnable).
The Two Faces of Uncertainty
The core intuition behind this research direction is that not all surprises are created equal. As Bertrand Charpentier and colleagues argue in Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning, an agent must distinguish between these two to be efficient:
- Epistemic Uncertainty: This is "reducible" uncertainty. It stems from a lack of data. If the agent visits a state enough times, this uncertainty should vanish. This is the true signal for exploration.
- Aleatoric Uncertainty: This is "irreducible" uncertainty. It is the inherent noise in the environment—the static on the TV or the bounce of a die. No amount of experience will make this predictable.
When an agent conflates these two, it treats all variance as a signal for exploration. This leads to the "Pitfall of Optimism" described by Taehyun Cho in Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion, where agents become trapped in high-variance, sub-optimal states because they interpret inherent noise as a potential for high future rewards.
Filtering the Noise
How do we actually separate these signals? Several mechanisms have emerged to help agents "ignore" the Noisy TV.
Aleatoric Mapping
One direct approach is to explicitly model the noise. In How to Stay Curious while avoiding Noisy TVs using Aleatoric Uncertainty Estimation, Augustine N. Mavor-Parker and Lewis Griffin introduce Aleatoric Mapping Agents (AMAs). These agents use a double-headed network: one head predicts the next state, while the other predicts the variance (the noise) of that state. By subtracting the predicted noise from the total prediction error, the agent is left with a "pure" curiosity reward that only reflects what is actually learnable.
Learning Progress Monitoring
Instead of measuring how surprised they are now, agents can measure how much their surprise is decreasing. Zhibo Hou and Wan Du propose Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring. If an agent watches a Noisy TV, its prediction error stays high but never improves. By rewarding Learning Progress (LPM)—the derivative of the error—the agent quickly loses interest in the TV and moves on to things it can actually master.
Random Network Distillation
A different strategy is to change the target of prediction. In Exploration by Random Network Distillation, Yuri Burda and Amos Storkey have the agent predict the output of a fixed, random neural network. Because the target network is deterministic, any prediction error must come from the agent's lack of familiarity with the input (epistemic), not from randomness in the target itself (aleatoric).
Principled Optimism and Safety
Disentangling uncertainty isn't just about avoiding distractions; it's about making exploration safer and more principled.
In continuous action spaces, Amarildo Likmeta and Marcello Restelli developed Wasserstein Actor-Critic (WAC), which uses distributional value functions to propagate epistemic uncertainty. Similarly, Jinyi Liu suggests in OVD-Explorer that "optimism should not be the sole pursuit." By separating the unknown from the noisy, agents can be "optimistic" about new knowledge while remaining "risk-averse" toward inherent environmental volatility.
This distinction is critical for robotics. Marin Vlastelica and Georg Martius demonstrate in Risk-Averse Zero-Order Trajectory Optimization that by penalizing aleatoric noise while rewarding epistemic exploration, robots can learn to navigate complex environments without wandering into high-risk, unpredictable zones.
Biological Parallels
Interestingly, this computational challenge mirrors how the human brain operates. Research by Shuo Zhang and Haiyan Wu in The neural correlates of novelty and variability in human decision-making suggests that humans use distinct neural circuits to calculate the value of seeking new information versus managing environmental randomness. This suggests that the separation of "resolvable ambiguity" from "inherent randomness" is a fundamental requirement for any intelligent system—biological or artificial—navigating an uncertain world.