Icebreaker: Element-wise Efficient Information Acquisition with a Bayesian Deep Latent Gaussian Model

Wenbo Gong, Sebastian Tschiatschek, Sebastian Nowozin, Richard E. Turner, José Miguel Hernández-Lobato, Cheng Zhang

TL;DR

Icebreaker is a framework designed to train machine learning models from scratch when data is scarce and every individual feature measurement carries a cost. By using a Bayesian Deep Latent Gaussian Model (BELGAM), it identifies the most informative data elements to "buy" during training, achieving high performance with significantly less data than traditional methods.

The "Ice-Start" Problem

In many real-world scenarios, data isn't just sitting in a database waiting to be used; it is expensive to acquire. For example, in healthcare, every lab test for a patient costs money, time, and potential health risks.

Standard Active Learning typically assumes you have all the features and just need to pay for the "label" (the outcome). However, the authors identify a more granular challenge they call the "ice-start" problem: a situation where you have little to no initial data, and you must decide which specific feature elements (e.g., a specific patient's glucose level) are worth the cost of acquisition to build an effective model. Existing methods often require a well-trained model to start with, making them unsuitable for this "from-scratch" scenario.

Methods: The Icebreaker Framework

The authors introduce Icebreaker, which tackles the ice-start problem using a novel generative modeling approach.

BELGAM Graphical Model

It consists of two main components:

  1. BELGAM (Bayesian Deep Latent Gaussian Model): Unlike standard Variational Autoencoders (VAEs) that use point estimates for neural network weights, BELGAM is fully Bayesian. It treats the global network weights ($\theta$) as distributions, allowing the model to quantify epistemic uncertainty—essentially, the model knows what it doesn't know. This is crucial when dealing with partially observed data ($\mathbf{X}$).
  2. PA-BELGAM (Partial Amortized Inference): To make this computationally feasible, the authors combine two techniques. They use amortized inference (a fast, encoder-based approach) for local latent variables ($\mathbf{Z}$) and Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) to sample the global network weights. This ensures the model remains accurate even when data is extremely sparse.
Icebreaker Acquisition Loop

The framework uses an iterative active learning loop. Using novel acquisition functions, it decides which data to query next. For imputation tasks, it seeks elements that reduce overall model uncertainty. For prediction tasks, it uses a balanced objective that trades off "exploration" (learning about the model's general environment) and "exploitation" (learning to predict the specific target variable).

Key Insights and Ideas

  • Element-wise vs. Row-wise: The paper demonstrates that it is often "wasteful" to acquire an entire row of data (all features for one person). Acquiring specific elements across many individuals allows the model to discover correlations more efficiently.
  • The Power of "I Don't Know": Standard VAEs tend to overfit when data is scarce because they rely on point estimates. Because BELGAM is Bayesian, it leverages its own uncertainty to guide its "curiosity," picking data points that fill the largest gaps in its knowledge.
  • Long-Tail Selection: When analyzing selection patterns, Icebreaker exhibits a "long-tail" distribution. It tends to exploit correlations within certain rows while simultaneously spreading its queries across the dataset to explore new territory.
  • Clinical Relevance: In experiments using the MIMIC-III healthcare dataset, the model naturally prioritized the Glasgow Coma Scale (GCS)—a metric clinicians already know is a primary indicator of mortality—without being explicitly told to do so.

Key Results

The model was evaluated across three distinct domains: standard UCI benchmarks (Boston Housing, Concrete, etc.), the MovieLens-1M recommender dataset, and the MIMIC-III clinical dataset.

  • Superiority in Scarcity: BELGAM significantly outperformed standard VAE-based models (like P-VAE) when the training set was small.
  • Efficiency: On the Boston Housing dataset, Icebreaker achieved lower error rates with fewer observed feature values compared to random acquisition.
  • Healthcare Robustness: In mortality prediction (MIMIC-III), Icebreaker was more robust and had smaller error bars than baselines, which is critical for high-stakes medical AI.
  • Active Prediction: The model achieved a better Area Under the Information Curve (AUIC), meaning it could make more accurate predictions at test-time with fewer queries than models trained on random data.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Variable Types: The current model is optimized for continuous data; future work could better incorporate "mixed-type" variables (a combination of categorical and continuous data).
  • Streaming Data: The framework currently operates in a pool-based setting. Adapting it for a pure "streaming" environment—where data arrives and decisions must be made instantly—is a suggested next step.
  • Computational Overhead: While the authors claim the cost is similar to a standard VAE, the use of SGHMC for weight sampling is inherently more complex than simple backpropagation, which may impact scalability for extremely large neural networks.

Summary updated

Abstract

In this paper we introduce the ice-start problem, i.e., the challenge of deploying machine learning models when only little or no training data is initially available, and acquiring each feature element of data is associated with costs. This setting is representative for the real-world machine learning applications. For instance, in the health-care domain, when training an AI system for predicting patient metrics from lab tests, obtaining every single measurement comes with a high cost. Active learning, where only the label is associated with a cost does not apply to such problem, because performing all possible lab tests to acquire a new training datum would be costly, as well as unnecessary due to redundancy. We propose Icebreaker, a principled framework to approach the ice-start problem. Icebreaker uses a full Bayesian Deep Latent Gaussian Model (BELGAM) with a novel inference method. Our proposed method combines recent advances in amortized inference and stochastic gradient MCMC to enable fast and accurate posterior inference. By utilizing BELGAM's ability to fully quantify model uncertainty, we also propose two information acquisition functions for imputation and active prediction problems. We demonstrate that BELGAM performs significantly better than the previous VAE (Variational autoencoder) based models, when the data set size is small, using both machine learning benchmarks and real-world recommender systems and health-care applications. Moreover, based on BELGAM, Icebreaker further improves the performance and demonstrate the ability to use minimum amount of the training data to obtain the highest test time performance.
BibTeX
@inproceedings{gong2019icebreaker,
  author = {Wenbo Gong and Sebastian Tschiatschek and Sebastian Nowozin and Richard E. Turner and José Miguel Hernández-Lobato and Cheng Zhang},
  title = {{Icebreaker: Element-wise Efficient Information Acquisition with a Bayesian Deep Latent Gaussian Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure