Disentangling Aleatoric and Epistemic Uncertainty in Reinforcement Learning

This direction is merged into another direction: uncertainty conflation. Its last version stays readable and citable here.

Version 1 of 144 papers47 conceptsfrozen 2026-10-04

In reinforcement learning (RL), an agent’s progress is often stalled by a fundamental inability to distinguish between two types of "unknowns." Imagine an agent in a room with a television flickering with static. To a standard curiosity-driven agent, that static looks like a goldmine of new information because it is unpredictable. The agent might stare at the screen forever, trapped by a phenomenon known as the Noisy TV problem.

The core of this research direction is the mathematical and architectural disentanglement of aleatoric uncertainty (inherent environmental noise) from epistemic uncertainty (the agent's lack of knowledge). By separating these two, researchers are building agents that can ignore irreducible randomness while aggressively exploring learnable dynamics.

The Two Faces of Uncertainty

To understand why this distinction matters, we must define the two sources of "risk" or "surprise" an agent encounters:

  1. Aleatoric Uncertainty: This is the "noise" inherent in the system. If you flip a fair coin, you don't know the outcome, but no amount of flipping will make the outcome more predictable. In RL, this manifests as stochastic transitions or noisy rewards.
  2. Epistemic Uncertainty: This is "ignorance." If you have never seen a coin before, you don't know how it works. However, after enough observations, this uncertainty vanishes. This is the "learnable" part of the environment.

As Borislav Mavrin and colleagues point out in their work on Decaying Left Truncated Variance (DLTV), failing to separate these causes agents to waste resources on unpredictable transitions that offer no useful information.

Distributional RL as a Diagnostic Tool

Traditional RL focuses on the expected value (the average) of future rewards. However, an average hides the structure of uncertainty. Distributional RL instead models the entire probability distribution of returns.

Recent breakthroughs like Value Flows by Perry Dong, Benjamin Eysenbach, and others use flow-matching generative models to capture complex, multi-modal return structures. This allows the agent to see not just that a state is "uncertain," but how it is uncertain. If the distribution is wide because of environmental noise, the agent should treat it differently than if the distribution is wide because the agent has only visited that state once.

Alberto Maria Metelli and his team proposed Wasserstein Q-Learning, which uses Wasserstein barycenters to propagate estimation uncertainty across the Bellman update. This ensures that the agent's "lack of knowledge" is tracked as a distinct signal that decays as more data is gathered.

Solving the Noisy TV Problem

The most immediate benefit of this disentanglement is more efficient exploration. If an agent can "explain away" aleatoric noise, it can focus its curiosity on epistemic gaps.

Daniel Jarrett and Remi Munos introduced Curiosity in Hindsight (2023), a framework where the agent learns to predict the past. By using hindsight representations, the agent identifies which parts of a transition were truly unpredictable (aleatoric) and discounts them from its intrinsic reward signal. Similarly, Jinyi Liu developed OVD-Explorer, which uses a noise-aware metric to ensure that "optimism" is only applied to epistemic uncertainty, preventing the agent from being lured by the "false promise" of random noise.

Tactical Optimism and Risk-Aversion

The separation of uncertainties also enables more sophisticated safety and risk-management strategies. In high-stakes environments like robotics, an agent should be optimistic about what it doesn't know (to encourage learning) but pessimistic about inherent noise (to avoid accidents).

The RAZER framework by Marin Vlastelica and Georg Martius implements this "tactical" approach. By applying pessimism toward aleatoric noise and optimism toward epistemic ignorance, RAZER achieves higher success rates in noisy robotic tasks than methods that treat all uncertainty as a single "risk" value.

In the model-based realm, the PETS algorithm by Kurtland Chua and Sergey Levine uses ensembles of probabilistic neural networks to capture both types of uncertainty. This prevents the agent from confidently following "hallucinated" trajectories in regions where the model is inaccurate, a common failure mode in model-based RL.

Mechanisms for Disentanglement

How do these agents actually perform the split? Several recurring mechanisms appear across the literature:

  • Ensembles and Normalizing Flows: Christian Schroeder de Witt and colleagues use Bayesian Exploration Networks (BEN) to model both noise and model uncertainty directly within the Bellman operator using normalizing flows.
  • Subjective Logic: Fabio Massimo Zennaro and Audun Jøsang apply Subjective Logic to multi-armed bandits, providing a formal logic for an agent to express "I don't know" versus "This is random."
  • Multi-Agent Disentanglement: In cooperative settings, Kyunghwan Son and his team developed DRIMA, which separates environmental stochasticity from the unpredictability of teammates. This allows agents to be optimistic about their peers' potential to cooperate while remaining realistic about environmental hazards.

By treating uncertainty not as a single number, but as a structured relationship between the agent's current knowledge and the world's inherent randomness, this research direction is moving RL toward agents that are both more curious and more cautious.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_disentangling_aleatoric_and_epistemic_un_163b9d5d,
  title = {Disentangling Aleatoric and Epistemic Uncertainty in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disentangling-aleatoric-and-epistemic-uncertainty-in-reinforcement-learning/rdr_163b9d5d56ea4aeeb9ca9a85c67bf880},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_disentangling_aleatoric_and_epistemic_un_163b9d5d_v1,
  title = {Disentangling Aleatoric and Epistemic Uncertainty in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disentangling-aleatoric-and-epistemic-uncertainty-in-reinforcement-learning/rdr_163b9d5d56ea4aeeb9ca9a85c67bf880/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}