TL;DR
InfoBot uses the Information Bottleneck principle to train reinforcement learning agents that rely on "habits" and only consult their goal at critical "decision states." This approach identifies natural subgoals, leading to significantly better policy transfer and more efficient exploration in environments where rewards are sparse.
What Problem Does This Paper Address?
A major hurdle in deep reinforcement learning (RL) is sparse rewards. In many complex environments, an agent might wander for thousands of steps without receiving any feedback, making it nearly impossible to learn. While standard exploration strategies like injecting noise or rewarding "curiosity" help, they are often task-agnostic—they don't leverage what the agent has learned from previous, similar tasks.
The authors introduce the concept of decision states: critical junctions in a state space (like a hallway intersection) where the agent's next move depends entirely on its specific goal. In contrast, most other states allow for "default" behaviors or habits (like walking straight down a hallway). The challenge is identifying these decision states automatically and using them to guide exploration in new, unseen environments.
Methods
The core of the paper is the InfoBot architecture, which applies the Information Bottleneck (IB) to a goal-conditioned policy $\pi(A | S, G)$.
-
The Information Bottleneck: The policy is split into an encoder that compresses the goal $G$ into a latent representation $Z$, and a decoder that chooses an action $A$ based on the current state $S$ and $Z$. Figure 1: Policy architecture.
-
Regularization: The agent is trained to maximize reward while minimizing the mutual information between the action and the goal, $I(A; G | S)$. This forces the agent to ignore the goal whenever possible, relying instead on a "default policy" (habits).
- Identifying Decision States: States where the agent cannot ignore the goal—where the KL divergence between the goal-conditioned policy and the default policy is high—are identified as decision states.
- Exploration Bonus: When the agent moves to a new, larger environment, it uses the learned encoder to calculate a "decision state bonus." It is rewarded for reaching states that its prior experience suggests are critical junctions, effectively treating them as subgoals for exploration.
Datasets
The authors evaluated InfoBot across a diverse range of benchmarks:
-
MiniGrid: Partially observable grid worlds including
MultiRoom(navigating through connected rooms) andFindObj(locating specific colored objects). Figure 2: MultiRoom and FindObj environments. -
MiniPacMan: A navigation task with complex branching and dead ends.
- Continuous Control: OpenAI Gym environments like
Humanoid-v1,Walker2d-v1, andHopper-v1. - Atari Games: Pong, Qbert, Seaquest, and Breakout.
- Puddle World: A spatial reasoning task where goals are provided via natural language instructions.
Key Insights and Ideas
- Information as a Proxy for Importance: The amount of information an agent needs to extract from its goal is a powerful signal for state importance. If an agent can act successfully without knowing its goal, it is likely in a "habitual" region of the state space.
- Parsimonious Subgoals: Traditional "bottleneck states" are often defined by environment geometry alone. InfoBot's "decision states" are task-relevant; a junction is only a decision state if different goals require different actions there.
- Transferable Exploration: Exploration strategies shouldn't start from scratch. By learning the "sensory cues" of decision states in small environments, agents can hunt for similar junctions in much larger, more complex ones.
- Cognitive Parallelism: The architecture mirrors theories in cognitive science regarding the distinction between automatic (habitual) and controlled (goal-directed) action selection in the human brain.
Key Results
-
Superior Generalization: In
MultiRoomtasks, InfoBot trained on 2-room maps solved 10-room maps with near 100% success, whereas standard A2C baselines failed to reach 50% success. Figure 3: Policy generalization on MultiRoom. -
Effective Exploration: In a 5-room maze, InfoBot achieved an 85% success rate, vastly outperforming state-of-the-art curiosity-based methods (54%) and count-based exploration (0%).
- MiniPacMan Success: InfoBot reached the goal 64% of the time in large mazes, compared to just 37% for Feudal RL and 47% for Curiosity-Driven Learning.
- Cross-Game Transfer: An exploration bonus learned on the Atari game Seaquest was successfully transferred to Pong and Qbert, helping the agent learn those new games faster.
Limitations and Future Work
- Variational Approximation: The method relies on a variational upper bound for mutual information, which requires choosing a prior distribution (like a unit Gaussian) that may not always be optimal.
- Task Family Dependence: The exploration bonus is effective for tasks within the same "family." Transferring a decision-state encoder between completely unrelated domains (e.g., from Chess to a driving simulator) remains an open challenge.
- Hyperparameter Sensitivity: The tradeoff parameter $\beta$, which controls the strength of the information bottleneck, requires careful tuning to balance habit formation with goal-directed behavior.
Summary updated