InfoBot: Transfer and Exploration via the Information Bottleneck

Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, Sergey Levine

TL;DR

InfoBot uses the Information Bottleneck principle to train reinforcement learning agents that rely on "habits" and only consult their goal at critical "decision states." This approach identifies natural subgoals, leading to significantly better policy transfer and more efficient exploration in environments where rewards are sparse.

What Problem Does This Paper Address?

A major hurdle in deep reinforcement learning (RL) is sparse rewards. In many complex environments, an agent might wander for thousands of steps without receiving any feedback, making it nearly impossible to learn. While standard exploration strategies like injecting noise or rewarding "curiosity" help, they are often task-agnostic—they don't leverage what the agent has learned from previous, similar tasks.

The authors introduce the concept of decision states: critical junctions in a state space (like a hallway intersection) where the agent's next move depends entirely on its specific goal. In contrast, most other states allow for "default" behaviors or habits (like walking straight down a hallway). The challenge is identifying these decision states automatically and using them to guide exploration in new, unseen environments.

Methods

The core of the paper is the InfoBot architecture, which applies the Information Bottleneck (IB) to a goal-conditioned policy $\pi(A | S, G)$.

  1. The Information Bottleneck: The policy is split into an encoder that compresses the goal $G$ into a latent representation $Z$, and a decoder that chooses an action $A$ based on the current state $S$ and $Z$. Figure 1: Policy architecture.

  2. Regularization: The agent is trained to maximize reward while minimizing the mutual information between the action and the goal, $I(A; G | S)$. This forces the agent to ignore the goal whenever possible, relying instead on a "default policy" (habits).

  3. Identifying Decision States: States where the agent cannot ignore the goal—where the KL divergence between the goal-conditioned policy and the default policy is high—are identified as decision states.
  4. Exploration Bonus: When the agent moves to a new, larger environment, it uses the learned encoder to calculate a "decision state bonus." It is rewarded for reaching states that its prior experience suggests are critical junctions, effectively treating them as subgoals for exploration.
Figure 1: Policy architecture.

Datasets

The authors evaluated InfoBot across a diverse range of benchmarks:

  • MiniGrid: Partially observable grid worlds including MultiRoom (navigating through connected rooms) and FindObj (locating specific colored objects). Figure 2: MultiRoom and FindObj environments.

  • MiniPacMan: A navigation task with complex branching and dead ends.

  • Continuous Control: OpenAI Gym environments like Humanoid-v1, Walker2d-v1, and Hopper-v1.
  • Atari Games: Pong, Qbert, Seaquest, and Breakout.
  • Puddle World: A spatial reasoning task where goals are provided via natural language instructions.
Figure 2: MultiRoomNXSY and FindObjSY MiniGrid environments. See text for details.

Key Insights and Ideas

  • Information as a Proxy for Importance: The amount of information an agent needs to extract from its goal is a powerful signal for state importance. If an agent can act successfully without knowing its goal, it is likely in a "habitual" region of the state space.
  • Parsimonious Subgoals: Traditional "bottleneck states" are often defined by environment geometry alone. InfoBot's "decision states" are task-relevant; a junction is only a decision state if different goals require different actions there.
  • Transferable Exploration: Exploration strategies shouldn't start from scratch. By learning the "sensory cues" of decision states in small environments, agents can hunt for similar junctions in much larger, more complex ones.
  • Cognitive Parallelism: The architecture mirrors theories in cognitive science regarding the distinction between automatic (habitual) and controlled (goal-directed) action selection in the human brain.

Key Results

  • Superior Generalization: In MultiRoom tasks, InfoBot trained on 2-room maps solved 10-room maps with near 100% success, whereas standard A2C baselines failed to reach 50% success. Figure 3: Policy generalization on MultiRoom.

  • Effective Exploration: In a 5-room maze, InfoBot achieved an 85% success rate, vastly outperforming state-of-the-art curiosity-based methods (54%) and count-based exploration (0%).

  • MiniPacMan Success: InfoBot reached the goal 64% of the time in large mazes, compared to just 37% for Feudal RL and 47% for Curiosity-Driven Learning.
  • Cross-Game Transfer: An exploration bonus learned on the Atari game Seaquest was successfully transferred to Pong and Qbert, helping the agent learn those new games faster.
Figure 3: Policy generalization on MultiRoom.

Limitations and Future Work

  • Variational Approximation: The method relies on a variational upper bound for mutual information, which requires choosing a prior distribution (like a unit Gaussian) that may not always be optimal.
  • Task Family Dependence: The exploration bonus is effective for tasks within the same "family." Transferring a decision-state encoder between completely unrelated domains (e.g., from Chess to a driving simulator) remains an open challenge.
  • Hyperparameter Sensitivity: The tradeoff parameter $\beta$, which controls the strength of the information bottleneck, requires careful tuning to balance habit formation with goal-directed behavior.

Summary updated

Abstract

A central challenge in reinforcement learning is discovering effective policies for tasks where rewards are sparsely distributed. We postulate that in the absence of useful reward signals, an effective exploration strategy should seek out {\it decision states}. These states lie at critical junctions in the state space from where the agent can transition to new, potentially unexplored regions. We propose to learn about decision states from prior experience. By training a goal-conditioned policy with an information bottleneck, we can identify decision states by examining where the model actually leverages the goal state. We find that this simple mechanism effectively identifies decision states, even in partially observed settings. In effect, the model learns the sensory cues that correlate with potential subgoals. In new environments, this model can then identify novel subgoals for further exploration, guiding the agent through a sequence of potential decision states and through new regions of the state space.
BibTeX
@inproceedings{goyal2019infobot,
  author = {Anirudh Goyal and Riashat Islam and Daniel Strouse and Zafarali Ahmed and Matthew Botvinick and Hugo Larochelle and Yoshua Bengio and Sergey Levine},
  title = {{InfoBot: Transfer and Exploration via the Information Bottleneck}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure