Abstract
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the information contained in some distribution of interest is its entropy. Greedily minimizing the expected entropy is therefore a standard method for choosing samples in order to gain strong beliefs about the underlying random variables. We show that this approach is prone to temporally getting stuck in local optima corresponding to wrongly biased beliefs. We suggest instead maximizing the expected cross entropy between old and new belief, which aims at challenging refutable beliefs and thereby avoids these local optima. We show that both criteria are closely related and that their difference can be traced back to the asymmetry of the Kullback-Leibler divergence. In illustrative examples as well as simulated and real-world experiments we demonstrate the advantage of cross entropy over simple entropy for practical applications.
Introduction
In experimental design and robotics, active learning is the art of choosing the most informative data points to sample next. The gold standard has long been to minimize the expected entropy of the posterior distribution. Intuitively, this means choosing samples that make a model as "certain" as possible. However, the authors identify a critical flaw: this greedy approach is prone to "confirmation bias." If early, noisy data points suggest a wrong hypothesis, minimizing entropy will lead the agent to pick samples that further confirm this wrong belief because they offer a path to high certainty. The agent gets stuck in a local optimum of "wrong but certain."
The MaxCE Criterion
The authors propose a shift in perspective. Instead of looking for the lowest future entropy, we should look for the greatest change in belief. They introduce MaxCE (Maximum Cross Entropy). While traditional methods maximize the expected KL-divergence from the posterior to the prior (which simplifies to minimizing entropy), MaxCE maximizes the expected KL-divergence from the prior to the posterior.
Mathematically, the difference is a simple swap of arguments in the KL-divergence formula, but the behavioral implications are profound. Entropy minimization seeks data that makes the current favorite hypothesis even more certain, whereas MaxCE seeks data that has the highest potential to change the agent's mind.
Key Results and Insights
The researchers evaluated their method across three distinct environments: synthetic 3D data, a high-dimensional CT slices dataset, and a real-world robotic joint dependency task using a PR2 robot.
- Faster Convergence: In synthetic tasks, MaxCE reached near-zero entropy (perfect model identification) in roughly half the samples required by standard Bayesian experimental design.
- Superior Model Selection: In the CT slice experiment, MaxCE and a hybrid method (combining MaxCE with uncertainty sampling) were the only ones to consistently identify the best-fitting kernel.
- Robotic Success: In the PR2 robot experiment, only the MaxCE strategy successfully uncovered the joint dependency structure. The standard entropy-minimization strategy got stuck, failing to pick the "challenging" actions necessary to see if one joint actually affected another.
The "aha moment" here is realizing that $D_{KL}(P||Q) \neq D_{KL}(Q||P)$. By simply flipping the direction of the divergence, the objective changes from "confirming" to "challenging." Furthermore, the authors provide a formal proof that conditional entropy is not submodular, explaining why the standard greedy approach fails so often in practice without the theoretical safety net that submodularity usually provides.
Limitations and Future Work
The paper primarily focuses on discrete sets of hypotheses. While the authors suggest the method could extend to continuous hyperparameters, doing so would require complex integration techniques to keep the computations tractable. Future work could also involve developing more principled, adaptive ways to balance exploration (model identification) and exploitation (predictive accuracy) during the learning process.
Summary updated