The Variational Bandwidth Bottleneck: Stochastic Evaluation on an Information Budget

Anirudh Goyal, Yoshua Bengio, Matthew M. Botvinick, Sergey Levine

Introduction

In traditional machine learning, models are typically fed every piece of available data. The Information Bottleneck (IB) method attempts to optimize this by compressing the input, discarding "noise" to retain only task-relevant features. However, standard IB has a critical flaw for certain applications: to determine what to discard, the model must still process the entire input first.

The Variational Bandwidth Bottleneck (VBB) is a novel framework designed to address this inefficiency. It allows machine learning models—particularly reinforcement learning (RL) agents—to decide whether they even need to look at "privileged" or costly information before accessing it. By learning to operate on an "information budget," these agents generalize better to new environments and significantly reduce computational and communication overhead.

The Cost of Information

In many real-world scenarios, inputs are not created equal. Some are cheap and readily available, while others are "privileged" or "costly." Consider a navigating robot: it can easily use its basic camera sensors (standard input), but running a complex, power-hungry path-planning algorithm (privileged input) drains its battery. Similarly, in multi-agent systems, agents must pay a "communication cost" to share information with one another.

Standard variational bottlenecks cannot avoid the initial cost of processing this privileged data. VBB bridges this gap by creating a bottleneck that decides if the extra information is worth the "price," basing its decision solely on the cheap, standard observations.

The VBB Architecture

The VBB architecture splits the world into two distinct inputs:

  1. Standard Input ($S$): Always available and computationally "cheap" (e.g., a robot's current camera view).
  2. Privileged Input ($G$): Potentially "expensive" or complex (e.g., the final goal location or a message from another agent).
The VBB architecture uses a Channel Capacity Network to decide whether to access privileged information or sample from a prior.

The core innovation is the Channel Capacity Network, denoted as $B(S)$. This network looks only at the standard input $S$ and outputs a value $d_{cap}$, which acts as a stochastic switch:

  • With probability $d_{cap}$, the model accesses the privileged input $G$.
  • With probability $1 - d_{cap}$, the model ignores $G$ and instead samples from a simple "prior" distribution (essentially relying on habit or a baseline guess).

To make this discrete "yes/no" decision trainable via backpropagation, the authors derived a tractable KL divergence objective. This formulation allows the model to learn the optimal probability of accessing information by treating the "information cost" as a penalty in the loss function.

Key Insights: Generalization and Natural Bottlenecks

Evaluating VBB across various RL and supervised learning environments (including MiniGrid mazes, multi-agent communication tasks, and Sequential MNIST) revealed several fascinating behaviors.

Generalization through Scarcity: By forcing the model to solve tasks without privileged information most of the time, it learns more robust, general features from the standard input. It is actively prevented from "overfitting" to the privileged data.

Identifying "Natural Bottlenecks": In maze navigation experiments, VBB agents naturally learned to request the expensive goal location only when they reached critical "decision points."

Table showing the frequency of expensive inference queries based on agent location.

As shown in the table above, agents queried the expensive inference algorithm predominantly near junctions (72% of the time) rather than in straight hallways (28%). While inside a room or hallway, they realized they didn't need the extra info and saved their budget.

Performance Highlights

VBB demonstrated strong performance improvements across multiple domains while drastically reducing resource usage:

  • Better Navigation: In a complex 12-room maze, VBB achieved a 60% success rate, outperforming the standard Variational Information Bottleneck (55%) and baseline models that always had access to the goal (49%).
  • Reduced Communication: In a 10-agent coordination task, VBB achieved a better average distance to landmarks (5.22 vs. 5.44 baseline) while accessing privileged info only 34% of the time.
  • Efficient Memory: On Neural Turing Machine (NTM) copying tasks, VBB slightly outperformed a standard NTM while accessing external memory only 32% of the time.
  • Lower Error: On "Cluttered MNIST," VBB achieved a 3.8% error rate, beating the standard Recurrent Attention Model (4.3%).

Limitations and Future Directions

The primary limitation of the current VBB framework is its independence assumption. The mathematical formulation assumes that the standard input and privileged input are statistically independent. While this didn't hinder performance in the tested environments, real-world inputs are often highly correlated.

Future work will focus on relaxing this independence assumption to make the model more flexible. Additionally, researchers suggest applying VBB to complex "adaptive computation" scenarios, where models could dynamically decide how many layers of a deep neural network to activate based on the difficulty of the current input.

Summary updated

Abstract

In many applications, it is desirable to extract only the relevant information from complex input data, which involves making a decision about which input features are relevant. The information bottleneck method formalizes this as an information-theoretic optimization problem by maintaining an optimal tradeoff between compression (throwing away irrelevant input information), and predicting the target. In many problem settings, including the reinforcement learning problems we consider in this work, we might prefer to compress only part of the input. This is typically the case when we have a standard conditioning input, such as a state observation, and a ``privileged'' input, which might correspond to the goal of a task, the output of a costly planning algorithm, or communication with another agent. In such cases, we might prefer to compress the privileged input, either to achieve better generalization (e.g., with respect to goals) or to minimize access to costly information (e.g., in the case of communication). Practical implementations of the information bottleneck based on variational inference require access to the privileged input in order to compute the bottleneck variable, so although they perform compression, this compression operation itself needs unrestricted, lossless access. In this work, we propose the variational bandwidth bottleneck, which decides for each example on the estimated value of the privileged information before seeing it, i.e., only based on the standard input, and then accordingly chooses stochastically, whether to access the privileged input or not. We formulate a tractable approximation to this framework and demonstrate in a series of reinforcement learning experiments that it can improve generalization and reduce access to computationally costly information.
BibTeX
@inproceedings{goyal2020variational,
  author = {Anirudh Goyal and Yoshua Bengio and Matthew M. Botvinick and Sergey Levine},
  title = {{The Variational Bandwidth Bottleneck: Stochastic Evaluation on an Information Budget}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure