In reinforcement learning (RL), agents are often driven by "curiosity"—a reward for visiting states where the agent’s internal model fails to predict the outcome. However, this simple heuristic leads to a classic failure mode known as the Noisy TV problem. Imagine an agent in a maze that encounters a television screen flickering with random static. Because the static is inherently unpredictable, the agent’s prediction error remains high, and it becomes "trapped" by the screen, wasting its life watching useless noise instead of exploring the environment.
To solve this, a growing body of research focuses on uncertainty disentanglement: the mathematical separation of inherent environmental randomness from a lack of knowledge.
The Two Faces of Uncertainty
The core of this research direction is the distinction between two types of uncertainty, a concept formalized in Bayesian deep learning by Depeweg et al.:
- Aleatoric Uncertainty: This is the irreducible randomness inherent in the environment. Examples include the result of a coin flip, the movement of a noisy background in a video (Wan et al.), or the "sticky actions" in Atari games where a command might randomly fail (Jarrett et al.).
- Epistemic Uncertainty: This is the "reducible" uncertainty that stems from a lack of data. It represents what the agent could know but hasn't learned yet. As the agent gathers more experience, this uncertainty should vanish.
As Nikolov et al. argue, agents that cannot distinguish between these two often struggle with efficient exploration, repeatedly visiting highly stochastic states that offer no new information about the optimal policy.
Learning to Ignore the Noise
To prevent the Noisy TV trap, agents must learn to "explain away" aleatoric noise so they can focus on epistemic gaps. Several mechanisms have been proposed to achieve this:
Hindsight and Reconstruction: Jarrett et al. propose Curiosity in Hindsight, where the agent uses information about the actual future outcome to reconstruct the transition. If the transition is unpredictable even with hindsight, it is classified as aleatoric noise and ignored. Similarly, Hou et al. introduce Learning Progress Monitoring (LPM), which rewards agents based on the improvement of their world model rather than the absolute magnitude of their surprise.
Distributional Filtering: Instead of predicting a single mean value, distributional RL agents model the entire range of possible returns. Mavrin et al. introduce Decaying Left Truncated Variance (DLTV), which mathematically separates vanishing estimation error (epistemic) from persistent environmental noise (aleatoric). By applying a decay to the exploration bonus, the agent eventually filters out the noise to focus on areas where knowledge is actually lacking. Liu et al. further this by using mutual information to ensure optimism is not the sole pursuit in noisy environments.
Aleatoric Mapping: Mavor-Parker et al. propose Aleatoric Mapping Agents (AMAs), which use a double-headed neural network. One head predicts the next state, while the other explicitly estimates the variance (noise) of that state. By subtracting the predicted noise from the total prediction error, the agent isolates the learnable novelty.
Risk-Aware Planning and Safety
Disentanglement is not just for exploration; it is critical for safety. In high-stakes robotics, an agent must know if it is uncertain because a situation is inherently dangerous or simply because it is new.
Chua demonstrates that by using ensembles of probabilistic neural networks (PETS), agents can capture both types of uncertainty to avoid catastrophic planning failures. Vlastelica et al. argue that this separation allows a system to avoid unpredictable regions while actively exploring unknown but potentially beneficial states. Without this distinction, as Eriksson et al. note, agents cannot be specifically tuned for "epistemic risk-aversion"—the caution required when the agent's internal model of the world might be fundamentally incorrect.
Toward Introspective Agents
The ultimate goal of this research is "introspective" agents that predict their own internal learning states. White et al. suggest that by representing confidence measures as General Value Functions (GVFs), agents can autonomously determine when to trust their own knowledge. Whether through information-theoretic stopping criteria for rollouts (Frauenknecht) or evidential deep learning for off-road navigation (Cai et al.), the ability to decompose uncertainty is transforming RL from a "black box" into a system capable of nuanced, self-aware decision-making.
Key Foundational Work
Introspective Agents: Confidence Measures for General Value Functions (2016) — contributed 2 concept elements to this direction.
Active Advice Seeking for Inverse Reinforcement Learning (2015) — contributed 2 concept elements to this direction.