In unsupervised reinforcement learning (RL), an agent must explore its environment without the guidance of an external reward signal. A classic approach to this problem is curiosity-driven exploration, where the agent is rewarded for seeking novelty or encountering states that violate its forward-world model's predictions.
However, curiosity-driven agents suffer from a fundamental flaw known as the Noisy TV problem. Imagine an agent placed in a room with a television screen displaying pure static. Because the transitions between frames of white noise are completely random and unpredictable, a curiosity-driven agent will become permanently transfixed by the screen, continually collecting rewards for its inability to predict the next frame.
To bypass this trap, a growing body of research shifts the intrinsic objective from novelty to controllability. Instead of asking "What is surprising?", a control-motivated agent asks, "What can I reliably influence?" By prioritizing states where its actions have a distinct, predictable impact on the environment, the agent naturally ignores uncontrollable background noise and focuses on learning meaningful, goal-directed behaviors.
The Mechanics of Empowerment
How do we mathematically formalize the intuitive concept of "control"? The foundational metric used across this paradigm is empowerment, an information-theoretic measure defined as the mutual information between an agent's actions (or sequences of actions) and its future states.
$$\mathcal{I}(A; S') = \mathcal{H}(S') - \mathcal{H}(S' \mid A)$$
Maximizing this mutual information means maximizing the entropy of future states $\mathcal{H}(S')$ while minimizing the conditional entropy $\mathcal{H}(S' \mid A)$. Intuitively, high empowerment requires two things:
- Diversity of Choice: The agent can reach a wide variety of distinct future states.
- Reliability of Execution: The mapping from actions to those future states is highly deterministic, meaning the agent can precisely choose which state it lands in.
Historically, computing empowerment was computationally prohibitive because it required integrating over all possible future trajectories. This bottleneck was broken when Shakir Mohamed et al. introduced a scalable variational lower bound in Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. By utilizing neural networks to approximate the reverse mapping from outcomes back to the actions that caused them, the authors enabled agents to compute empowerment and learn complex manipulation behaviors directly from high-dimensional pixel inputs.
Expanding on this variational framework, Karol Gregor et al. introduced Variational Intrinsic Control (VIC) in Variational Intrinsic Control. Instead of looking only at immediate primitive actions, VIC maximizes the mutual information between abstract "skills" (or options) and their final environmental outcomes. This ensures that the agent's exploration budget is spent entirely on features of the environment it can actively manipulate, successfully insulating it from unpredictable but uncontrollable stimuli like visual noise.
Filtering the Environment via Causal Attribution
By anchoring exploration to controllability, agents inherently gain a powerful representational filter. If a component of the environment changes randomly or independently of the agent's policy, the mutual information between the agent's actions and that component drops to zero.
To make this filtering explicit, Raul Vicente et al. proposed the Controlled Effect Network (CEN) in Did I do that? Blame as a means to identify controlled effects in reinforcement learning. CEN implements a counterfactual "blame" mechanism that compares what actually occurred in the environment against a baseline model of a world where the agent took no action. This allows the agent to isolate its own causal footprint, providing a precise intrinsic exploration signal that ignores environmental noise and excels in sparse-reward environments like Montezuma's Revenge.
Similarly, the authors of Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards developed a Relational Transition Model (RTM) to maximize control over specific environment entities. By rewarding the agent for forcing changes in object states rather than seeking visual novelty, Yuhang Song et al. achieved human-level performance on Atari games without ever exposing the agent to a game score.
This representational filtering is equally effective in model-based reinforcement learning. In Information Prioritization through Empowerment in Visual Model-based RL, Homanga Bharadhwaj et al. demonstrated that empowerment can serve a dual purpose: it acts as an intrinsic reward to guide the policy toward high-control zones, and it acts as a loss function filter that forces the world model to ignore complex, noisy video backgrounds that the agent cannot influence. When planning in real-time, integrating these control-centric curiosity signals directly into the planning loop—such as via the Curiosity Cross-Entropy Method proposed by Cornelius Weber et al. in Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning—proves to be the primary driver for sample efficiency gains in complex continuous control tasks.
Why Empowerment Works: The Physics of Unstable Equilibria
One of the most elegant properties of empowerment is that maximizing it does not just lead to random flailing within controllable zones; it naturally uncovers complex, stabilizing behaviors.
Consider a physical system like a pole balancing on a cart, or a humanoid robot standing upright. If the robot falls over, its physical options are severely restricted; it is trapped in a low-entropy state where most actions result in remaining on the floor. Conversely, when standing upright, the robot maintains a state of high potential energy and maximum operational flexibility—it can step forward, backward, turn, or jump.
Because standing upright preserves the widest variety of future action trajectories, the state of highest empowerment frequently corresponds to an unstable equilibrium.
Low Empowerment (Fallen) High Empowerment (Upright)
[ Restricted ] [ Maximum Options ]
Actions lead to Can step, turn,
the same state. or maintain balance.
│ ▲
▼ │
┌───────────┐ ┌───────────┐
│ __ __ │ │ O │
│ \ \_/ / │ │ /│\ │
└───┴───┴───┘ └───/\──────┘
In Efficient Empowerment Estimation for Unsupervised Stabilization, Ruihan Zhao et al. leveraged this insight for unsupervised stabilization. Without any human-engineered reward function telling a system to stay upright, agents maximizing empowerment naturally discover how to balance complex dynamical systems purely out of a mathematical drive to keep their future options open.
This connection between control and internal reward is not unique to artificial agents. In cognitive neuroscience, Wen Wen et al. demonstrated in The Readiness Potential Reflects the Reliability of Action Consequence that the human brain's readiness potential—the neural signal that builds up before motor execution—is significantly stronger when the outcome of an action is highly reliable. The biological brain treats the verification of environmental controllability as an inherent reward mechanism, allocating more preparatory resources to actions that yield predictable control.
Structuring Exploration and Skill Hierarchies
In long-horizon, sparse-reward domains, an agent cannot rely on random exploration to stumble upon a distant goal. It must break the environment down into a hierarchy of manageable subgoals or skills. Controllability provides an ideal metric for discovering these structures.
Instead of defining subgoals by visual novelty, Michael Dann et al. showed in Deriving Subgoals Autonomously to Accelerate Learning in Sparse Reward Domains that measuring the policy effort required to deviate from a baseline trajectory isolates true bottleneck states. This allows agents to ignore environmental changes they cannot control and focus on reachable subgoals that represent actual physical progress.
To navigate across these subgoals, Lorenzo Steccanella et al. designed a manager-worker architecture in Hierarchical Representation Learning for Markov Decision Processes that partitions the state space into abstract regions. Low-level workers are intrinsically rewarded for reaching boundary states that facilitate transitions between these regions, prioritizing states that offer high controllability over partition transitions.
When interacting with objects, this control-centric framing prevents the agent from wasting computational resources on immutable properties. In Granger Causal Interaction Skill Chains, Caleb Chuck et al. constructed skill chains by prioritizing exploration based on an agent's ability to exert Granger-causal influence over specific environmental factors (like an object's velocity or position).
To expand this control efficiently, Silviu Pitis developed Maximum Entropy Goal Achievement (MEGA) in Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning. MEGA maximizes empowerment off-policy by using a history of past experiences to sample goals specifically at the "frontier" of the agent's current capabilities, ensuring the agent steadily pushes the boundaries of its control. When environmental goals shift or external rewards vanish, this persistent drive for empowerment acts as a behavioral catalyst, as shown by Evgenii Aleksandrovich Dzhivelikian et al. in Hierarchical intrinsically motivated agent planning behavior with dreaming in grid environments, breaking repetitive behavioral loops and forcing the agent to resume productive exploration.
Blending Control with External Objectives
While empowerment functions as a powerful unsupervised objective, it can be seamlessly integrated into frameworks with explicit human preferences, safety constraints, or external tasks.
Task Acceleration
Rather than running empowerment and reward maximization sequentially, Felix Leibfried et al. derived a unified Bellman optimality principle in A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment. By embedding empowerment directly into the value function as a state-dependent intrinsic bonus, the agent is naturally pulled toward high-control regions early in training, preventing it from getting trapped in restrictive areas and significantly accelerating actor-critic learning on complex robotics tasks.
Safe Reinforcement Learning
Controllability can also serve as a mathematical shield for safety. In ROSARL: Reward-Only Safe Reinforcement Learning, Geraud Nangue Tasse et al. utilized environmental controllability to scale safety penalties. In regions where the environment's transitions are highly stochastic or uncontrollable, the system applies harsher penalties to risky actions. This allows the agent to learn to avoid unsafe states via a single scalar reward function, bypassing the need for complex, human-tuned constrained optimization.
Value-Aligned Assistance
Finally, empowerment offers a robust philosophy for human-robot collaboration. Traditional assistive agents try to infer a human's exact goal, which is highly error-prone. In AvE: Assistance via Empowerment, Yuqing Du et al. flipped this objective: the assistive agent's reward function is augmented to maximize the human's future empowerment.
By taking actions that keep the human's future options open and prevent them from entering low-control states, the assistive agent provides meaningful, robust help without needing to read the user's mind.
Key Foundational Work
Empowerment for continuous agent - environment systems (2011) — contributed 2 concept elements to this direction.
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015) — contributed 1 concept elements to this direction.