Introduction
Most deep learning models are strictly discriminative, focusing on mapping an input $x$ to a label $y$ to estimate $p(y|x)$. While effective, these models often lack awareness of the underlying data distribution $p(x)$, leading to overconfidence on out-of-distribution (OOD) samples. To address this, researchers have introduced the Deep Invertible Generalized Linear Model (DIGLM), a hybrid architecture that integrates predictive power with exact density estimation in a single feed-forward pass.
Methodology: The DIGLM Architecture
The DIGLM architecture leverages invertible "normalizing flows" (such as Glow or RealNVP) as feature extractors. Unlike standard neural networks, these transformations are bijective, allowing for exact density computation via the change-of-variables formula.
The process involves two primary stages:
- Feature Extraction: The input $x$ is transformed into a latent representation $z$ through a series of invertible layers. Because the mapping is bijective, the model can calculate the exact density of $x$ by evaluating the density of $z$ and the Jacobian determinant of the transformation.
- Prediction: A Generalized Linear Model (GLM) sits on top of the latent space $z$ to predict the label $y$.
The model is trained to maximize a weighted joint likelihood: $\log p(y|x) + \lambda \log p(x)$. The parameter $\lambda$ allows for a controlled trade-off between pure predictive accuracy and generative modeling quality.
Experimental Evaluation
The authors evaluated DIGLM across several regimes, including simulated 1D regression, large-scale regression (the 5-million record Flight Delay dataset), and image classification (MNIST, SVHN).
Key Results
- Superior Uncertainty: On the Flight Delay dataset, the DIGLM achieved a Negative Log-Likelihood (NLL) of 5.07, significantly outperforming the previous state-of-the-art of 6.91. This suggests the model is better at "knowing what it doesn't know" when data distributions shift.
- Semi-Supervised Gains: On MNIST, using only 1,000 labeled examples, the error rate dropped from 6.61% (supervised only) to 0.99% when the model incorporated the structure of unlabeled data via the $p(x)$ component.
- Built-in Rejection: Because the model computes $p(x)$, it can implement a "natural rejection" rule. If an input has a lower probability than anything seen during training, the model can refuse to make a prediction, reducing errors on anomalous data.
Discussion and Limitations
DIGLM offers a memory-efficient alternative to ensembles or dual-model systems by sharing almost all parameters between the generative and predictive components. Furthermore, the Jacobian determinant required for density estimation acts as a natural regularizer, potentially improving robustness against small input variations.
However, challenges remain. Generative models can sometimes assign higher likelihoods to OOD data than to training data (the "likelihood gap"). Additionally, normalizing flows require specific invertible architectures, which can be more memory-intensive and constrained than standard residual networks. Future work may explore alternative training objectives, such as Wasserstein distance, to further improve generative feature quality.
Summary updated