Hybrid Models with Deep and Invertible Features

Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, Balaji Lakshminarayanan

Introduction

Most deep learning models are strictly discriminative, focusing on mapping an input $x$ to a label $y$ to estimate $p(y|x)$. While effective, these models often lack awareness of the underlying data distribution $p(x)$, leading to overconfidence on out-of-distribution (OOD) samples. To address this, researchers have introduced the Deep Invertible Generalized Linear Model (DIGLM), a hybrid architecture that integrates predictive power with exact density estimation in a single feed-forward pass.

Methodology: The DIGLM Architecture

The DIGLM architecture leverages invertible "normalizing flows" (such as Glow or RealNVP) as feature extractors. Unlike standard neural networks, these transformations are bijective, allowing for exact density computation via the change-of-variables formula.

The DIGLM architecture combines an invertible mapping for feature extraction and a GLM for prediction, enabling simultaneous computation of $p(x)$ and $p(y|x)$.

The process involves two primary stages:

  1. Feature Extraction: The input $x$ is transformed into a latent representation $z$ through a series of invertible layers. Because the mapping is bijective, the model can calculate the exact density of $x$ by evaluating the density of $z$ and the Jacobian determinant of the transformation.
  2. Prediction: A Generalized Linear Model (GLM) sits on top of the latent space $z$ to predict the label $y$.

The model is trained to maximize a weighted joint likelihood: $\log p(y|x) + \lambda \log p(x)$. The parameter $\lambda$ allows for a controlled trade-off between pure predictive accuracy and generative modeling quality.

Experimental Evaluation

The authors evaluated DIGLM across several regimes, including simulated 1D regression, large-scale regression (the 5-million record Flight Delay dataset), and image classification (MNIST, SVHN).

Comparison of Gaussian Process and B-DIGLM on a 1D regression task. B-DIGLM provides both predictive means (b) and input density estimation (c).

Key Results

  • Superior Uncertainty: On the Flight Delay dataset, the DIGLM achieved a Negative Log-Likelihood (NLL) of 5.07, significantly outperforming the previous state-of-the-art of 6.91. This suggests the model is better at "knowing what it doesn't know" when data distributions shift.
  • Semi-Supervised Gains: On MNIST, using only 1,000 labeled examples, the error rate dropped from 6.61% (supervised only) to 0.99% when the model incorporated the structure of unlabeled data via the $p(x)$ component.
  • Built-in Rejection: Because the model computes $p(x)$, it can implement a "natural rejection" rule. If an input has a lower probability than anything seen during training, the model can refuse to make a prediction, reducing errors on anomalous data.

Discussion and Limitations

DIGLM offers a memory-efficient alternative to ensembles or dual-model systems by sharing almost all parameters between the generative and predictive components. Furthermore, the Jacobian determinant required for density estimation acts as a natural regularizer, potentially improving robustness against small input variations.

However, challenges remain. Generative models can sometimes assign higher likelihoods to OOD data than to training data (the "likelihood gap"). Additionally, normalizing flows require specific invertible architectures, which can be more memory-intensive and constrained than standard residual networks. Future work may explore alternative training objectives, such as Wasserstein distance, to further improve generative feature quality.

Summary updated

BibTeX
@inproceedings{nalisnick2019hybrid,
  author = {Eric T. Nalisnick and Akihiro Matsukawa and Yee Whye Teh and Dilan Görür and Balaji Lakshminarayanan},
  title = {{Hybrid Models with Deep and Invertible Features}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=rJNReoZO-B},
}
Figure