The Advantage of Cross Entropy over Entropy in Iterative Information Gathering

Johannes Kulick, Robert Lieck, Marc Toussaint

Abstract

Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the information contained in some distribution of interest is its entropy. Greedily minimizing the expected entropy is therefore a standard method for choosing samples in order to gain strong beliefs about the underlying random variables. We show that this approach is prone to temporally getting stuck in local optima corresponding to wrongly biased beliefs. We suggest instead maximizing the expected cross entropy between old and new belief, which aims at challenging refutable beliefs and thereby avoids these local optima. We show that both criteria are closely related and that their difference can be traced back to the asymmetry of the Kullback-Leibler divergence. In illustrative examples as well as simulated and real-world experiments we demonstrate the advantage of cross entropy over simple entropy for practical applications.

Introduction

In fields like robotics and machine learning, data acquisition is often the primary bottleneck. Whether it involves physical robot interactions or expensive human labeling, the goal of active learning is to select the most informative data points. Traditionally, this is achieved by minimizing the entropy of the model's posterior distribution—essentially reducing uncertainty.

However, as Johannes Kulick and colleagues demonstrate, this standard approach has a significant flaw: it is inherently confirmatory. If an agent develops an early, incorrect belief due to noisy data, entropy minimization encourages it to seek data that further refines this wrong belief rather than data that might challenge it. This leads to agents getting stuck in "local optima" of belief, where they become overconfident in a false hypothesis.

Methodology: From Entropy to Cross Entropy

The core of the proposed solution lies in changing the objective function for data selection. The authors compare two primary strategies:

  1. Maximum Negative Entropy (Standard): This method selects actions to minimize the expected entropy of the future belief. Mathematically, this corresponds to maximizing the KL-divergence from the posterior to the prior: $D_{KL}(p_{post} || p_{prior})$.
  2. Maximum Cross Entropy (MaxCE): The proposed method selects actions to maximize the expected cross entropy between the current belief and the future belief. This is equivalent to maximizing the KL-divergence from the prior to the posterior: $D_{KL}(p_{prior} || p_{post})$.
Comparison of objective functions: Cross Entropy, Negative Entropy, and Change of Entropy as a function of posterior belief.

Because the KL-divergence is asymmetric, this swap fundamentally changes the agent's behavior. While entropy minimization seeks to "sharpen" the current belief, MaxCE seeks data that causes the most significant change in belief. This "exploratory" nature allows the agent to recover from misleading initial data by actively looking for evidence that could refute its current model.

Heatmaps illustrating the relationship between prior and posterior beliefs for Cross Entropy versus Negative Entropy.

Experimental Evaluation

The researchers validated MaxCE across three distinct domains:

  • Synthetic 3D Tasks: Using Gaussian Processes, agents had to identify relevant input dimensions. MaxCE reached near-zero model entropy in approximately half the samples required by standard Bayesian experimental design.
  • CT Slices Dataset: In this high-dimensional real-world task, MaxCE demonstrated superior robustness. When the true model was not perfectly represented in the hypothesis set, standard methods struggled to converge, whereas MaxCE remained stable.
  • Robotic Manipulation: A PR2 robot was tasked with identifying the "joint dependency structure" of furniture (e.g., which key unlocks which drawer). Standard entropy methods often failed because they became too certain the drawer was simply stuck, never attempting to use the key. MaxCE successfully identified these complex dependencies by challenging its initial "stuck" hypothesis.

Key Insights

A critical theoretical finding is that conditional entropy is not submodular. This explains why greedy "best next step" strategies in standard active learning fail to reach global optima. Furthermore, the authors highlight that while most active learning focuses on reducing predictive variance, MaxCE is significantly superior for model selection—deciding which high-level hypothesis is correct.

The authors also found that a hybrid approach, mixing MaxCE for model identification with standard uncertainty sampling for predictive accuracy, yielded the best overall performance across all metrics.

Conclusion and Future Work

By reframing information gathering as a challenge to current beliefs rather than a search for confirmation, MaxCE provides a more robust framework for active learning. Future research will focus on extending these methods to continuous hyperparameter spaces and developing adaptive mixing strategies that allow agents to autonomously balance model-challenging exploration with predictive fine-tuning.

Summary updated

Abstract

Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the information contained in some distribution of interest is its entropy. Greedily minimizing the expected entropy is therefore a standard method for choosing samples in order to gain strong beliefs about the underlying random variables. We show that this approach is prone to temporally getting stuck in local optima corresponding to wrongly biased beliefs. We suggest instead maximizing the expected cross entropy between old and new belief, which aims at challenging refutable beliefs and thereby avoids these local optima. We show that both criteria are closely related and that their difference can be traced back to the asymmetry of the Kullback-Leibler divergence. In illustrative examples as well as simulated and real-world experiments we demonstrate the advantage of cross entropy over simple entropy for practical applications.
BibTeX
@misc{kulick2014advantage,
  author = {Johannes Kulick and Robert Lieck and Marc Toussaint},
  title = {{The Advantage of Cross Entropy over Entropy in Iterative Information Gathering}},
  year = {2014},
  eprint = {1409.7552},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1409.7552},
}
Figure