Unsupervised Discovery of Decision States for Transfer in Reinforcement Learning

Nirbhay Modhe, Prithvijit Chattopadhyay, Mohit Sharma, Abhishek Das, Devi Parikh, Dhruv Batra, Ramakrishna Vedantam

Abstract

We propose a novel framework to identify sub-goals useful for exploration in sequential decision making tasks under partial observability. We utilize the variational intrinsic control framework (Gregor et.al., 2016) which maximizes empowerment -- the ability to reliably reach a diverse set of states and show how to identify sub-goals as states with high necessary option information through an information theoretic regularizer. Despite being discovered without explicit goal supervision, our sub-goals provide better exploration and sample complexity on challenging grid-world navigation tasks compared to supervised counterparts in prior work.

Introduction

In reinforcement learning (RL), decomposing complex tasks into simpler sub-goals is a powerful strategy for improving sample efficiency and enabling task reuse. For instance, navigating to a doorway is a logical sub-goal before searching for an object in a new room. However, traditional sub-goal discovery often relies on explicit supervision or dense rewards, which are frequently unavailable in real-world scenarios. This paper introduces Information Regularized Variational Intrinsic Control (IR-VIC), an unsupervised framework that identifies "decision states"—structurally significant bottlenecks like doorways or intersections—purely through intrinsic motivation.

Methodology: IR-VIC

IR-VIC builds upon Variational Intrinsic Control (VIC), where an agent learns latent "options" ($\Omega$) to maximize empowerment. Empowerment is defined as the agent's ability to reach a diverse set of final states reliably.

The key innovation in IR-VIC is the introduction of an information-theoretic bottleneck. By minimizing the mutual information $I(\Omega, A_t | S_t)$ between the option and the action given the current state, the agent is forced to ignore the option $\Omega$ unless it is strictly necessary for action selection. States where this mutual information remains high are identified as "decision states" or sub-goals. Mathematically, the authors demonstrate that the sum of this conditional mutual information across a trajectory provides an upper bound on empowerment, justifying IR-VIC as a principled relaxation of empowerment maximization.

Experimental Results

The method was evaluated on grid-world navigation tasks from the MiniGrid suite, including 4-Room, Maze, and MultiRoom environments. Pre-training for sub-goal discovery was conducted on fixed layouts, while transfer evaluation was performed on randomly generated, novel layouts to test generalization.

Qualitative Analysis

Visualizing the mutual information $I(\Omega, A_t | S_t)$ reveals that IR-VIC successfully identifies structural bottlenecks. As shown in the heatmaps, the information spikes at critical points such as doorways and intersections, where the agent must "decide" which path to take based on its high-level option. This suggests that "necessary option information" captures kinematic and policy-specific bottlenecks more effectively than human-defined geometric landmarks.

Qualitative visualization of discovered sub-goals (decision states) in Four Room and Maze environments.

Quantitative Transfer Performance

To test the utility of these discovered sub-goals, the agent was transferred to novel, sparse-reward environments. An intrinsic exploration bonus was provided for visiting the discovered decision states. In the challenging MultiRoomN6S25 task (6 rooms, size 25), IR-VIC achieved a 92.9% success rate, outperforming standard count-based exploration (86.8%), the unsupervised skill-discovery baseline DIAYN (0.1%), and even the supervised InfoBot baseline (90.9%).

Performance comparison of IR-VIC against various baselines on the MultiRoom transfer task.

Discussion and Conclusion

IR-VIC demonstrates that unsupervised sub-goal discovery can match or even exceed the performance of supervised methods. By identifying states where option information is most critical, the agent learns a robust set of landmarks that facilitate exploration in novel environments. While the method currently requires tuning of the bottleneck parameter $\beta$ and relies on discrete options, it opens promising avenues for scaling to continuous option spaces and more complex hierarchical RL tasks.

Summary updated

Abstract

We propose a novel framework to identify sub-goals useful for exploration in sequential decision making tasks under partial observability. We utilize the variational intrinsic control framework (Gregor et.al., 2016) which maximizes empowerment -- the ability to reliably reach a diverse set of states and show how to identify sub-goals as states with high necessary option information through an information theoretic regularizer. Despite being discovered without explicit goal supervision, our sub-goals provide better exploration and sample complexity on challenging grid-world navigation tasks compared to supervised counterparts in prior work.
BibTeX
@inproceedings{modhe2020unsupervised,
  author = {Nirbhay Modhe and Prithvijit Chattopadhyay and Mohit Sharma and Abhishek Das and Devi Parikh and Dhruv Batra and Ramakrishna Vedantam},
  title = {{Unsupervised Discovery of Decision States for Transfer in Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=AD3u6e_d5u},
}
Figure