Abstract
We propose a novel framework to identify sub-goals useful for exploration in sequential decision making tasks under partial observability. We utilize the variational intrinsic control framework (Gregor et.al., 2016) which maximizes empowerment -- the ability to reliably reach a diverse set of states and show how to identify sub-goals as states with high necessary option information through an information theoretic regularizer. Despite being discovered without explicit goal supervision, our sub-goals provide better exploration and sample complexity on challenging grid-world navigation tasks compared to supervised counterparts in prior work.
Introduction
Reinforcement Learning (RL) agents frequently encounter environments characterized by "sparse rewards," where meaningful feedback is rare. A standard approach to mitigate this is hierarchical RL, which decomposes complex tasks into manageable sub-goals. However, traditional sub-goal discovery often relies on human intuition or explicit supervision, limiting scalability. This paper introduces Information Regularized Variational Intrinsic Control (IR-VIC), an unsupervised framework designed to discover "decision points"—critical states like doorways or junctions—that serve as effective sub-goals for exploration and transfer learning.
Methodology: The Information Bottleneck
IR-VIC extends the Variational Intrinsic Control (VIC) framework. While standard VIC encourages agents to learn diverse "options" (macro-actions) to maximize empowerment, IR-VIC introduces an information-theoretic bottleneck between the high-level option ($\Omega$) and the low-level action ($A_t$).
The core intuition is that an agent should be penalized for relying on its high-level goal when the environment's structure makes the next move obvious (e.g., walking down a straight hallway). Conversely, at "decision points" where multiple paths diverge, the agent must utilize information from $\Omega$ to choose the correct action.
By regularizing this information flow, IR-VIC identifies states where the "necessary option information" remains high. These states are designated as sub-goals. During transfer to new tasks, the agent receives an internal reward for visiting these discovered states, facilitating faster exploration in novel environments.
Experimental Evaluation
The authors evaluated IR-VIC using the MiniGrid suite, focusing on 4-Room, Maze, and procedural MultiRoom layouts. The primary test of the method's utility is transfer learning: pre-training the agent in an unsupervised manner on simple layouts and then testing its performance on complex, unseen environments with sparse rewards.
Qualitative Results
Heatmaps of the mutual information $I(\Omega, A_t | S_t)$ demonstrate that IR-VIC accurately pinpoints doorways and junctions. As shown in the comparison below, IR-VIC produces much sharper and more meaningful sub-goal distributions than existing unsupervised methods like DIAYN (Diversity is All You Need).
Quantitative Performance
In transfer tasks such as MultiRoomN6S25 (6 rooms of size 25), IR-VIC achieved a remarkable 92.9% success rate. This significantly outperformed:
- DIAYN: Which failed to solve the hardest transfer tasks (0% success).
- InfoBot: A supervised baseline whose "decision states" were less effective for transfer (approx. 10% success in re-implementation).
- Count-based Exploration: IR-VIC demonstrated superior sample efficiency, reaching high success rates with fewer interactions.
Discussion and Limitations
While IR-VIC shows strong performance, it is sensitive to the regularization weight $\beta$. High $\beta$ values can overly restrict information, while low values fail to produce sparse, distinct sub-goals. Additionally, while the theoretical framework is general, current experiments are limited to discrete grid-worlds. Future work will explore scaling these concepts to continuous control and high-dimensional visual inputs.
Conclusion
IR-VIC provides a principled, unsupervised method for discovering sub-goals by identifying information bottlenecks in an agent's policy. By focusing on states where high-level intent is most critical, the framework enables robust transfer learning and efficient exploration in complex, sparse-reward environments.
Summary updated