Robustly Train Normalizing Flows via KL Divergence Regularization

Kun Song, Ruben Solozabal, Hao Li, Martin Takác, Lu Ren, Fakhri Karray

Abstract

Normalizing Flows (NFs) are powerful density estimators but suffer from sensitivity to outliers and small sample sizes, particularly in high-dimensional spaces. We present a KL divergence regularization technique that stabilizes NF training by effectively incorporating "virtual" samples. This method addresses the low-rank failure mode of empirical covariance matrices in NFs. Our results demonstrate significant improvements in Out-of-Distribution (OoD) detection, with performance gains of up to 30% across various image benchmarks.

The Sensitivity Problem in Normalizing Flows

Normalizing Flows learn invertible mappings between complex data and simple base distributions. However, they face a critical "sensitivity problem." In high-dimensional spaces with sparse data, the estimated covariance matrix often becomes low-rank or singular. This makes the flow's transformation—which relies on the inverse of this matrix—unstable. Furthermore, even a single outlier can drastically warp the estimated density manifold, causing the model to overfit to noise.

Figure 1: Outliers can significantly shift the estimated data distribution, leading to poor generalization and sensitivity in high-dimensional spaces.

Deep NFs can be viewed as collections of local linear projections. If an outlier is isolated, the model may treat it as its own local neighborhood with only one sample, triggering a low-rank failure that biases the entire global density estimation.

Methods: KL Divergence Regularization

To combat this, the authors propose KL Divergence Regularization on the Jacobian matrix of the NF.

  • The Theory: For a linear NF, adding a KL divergence term between the model's transformation matrix and a prior positive definite matrix $G$ is mathematically equivalent to adding "virtual" training samples whose covariance is $G$. By choosing $G$ to be the identity matrix, they effectively "fill in" the missing ranks of the empirical covariance matrix.
  • Practical Implementation: Calculating the full Jacobian for deep NFs is computationally expensive. The authors instead derive an upper bound, regularizing the Frobenius norm of weights in coupling layers and the scaling factors produced by internal networks. This provides a computationally efficient shortcut to achieve robustness.

Experimental Results and Impact

The method was evaluated on standard benchmarks like CIFAR-10/100 against various Out-of-Distribution (OoD) datasets, including Rot-RGB, Noisy Cifar, QuickDraw, and Tiny-ImageNet.

Figure 6: OoD detection scores across different datasets showing consistent improvement as the regularization strength α increases.

Key findings include:

  • Significant OoD Detection Boost: ROC-AUC for detecting "QuickDraw" samples improved by 36.5% using the Information Bottleneck objective.
  • Noise Robustness: Regularized models maintained high accuracy even when training data was corrupted with Gaussian noise, while unregularized models saw performance collapse.
  • Data Efficiency: A regularized NF trained on only 20% of a dataset outperformed an unregularized NF trained on the full dataset.

Conclusion

KL Divergence Regularization offers a theoretically grounded and practically efficient way to stabilize Normalizing Flows. By treating mathematical regularization as "virtual data," this approach solves the inherent sensitivity of NFs to outliers and small sample sizes, making them far more reliable for real-world anomaly detection tasks.

Summary updated

Abstract

In this paper, we find that the training of Normalizing Flows (NFs) are easily affected by the outliers and a small number (or high dimensionality) of training samples. To solve this problem, we propose a Kullback–Leibler (KL) divergence regularization on the Jacobian matrix of NFs. We prove that such regularization is equivalent to adding a set of samples whose covariance matrix is the identity matrix to the training set. Thus, it reduces the negative influence of the outliers and the small sample number on the estimation of the covariance matrix, simultaneously. Therefore, our regularization makes the training of NFs robust. Ultimately, we evaluate the performance of NFs on out-of-distribution (OoD) detection tasks. The excellent results obtained demonstrate the effectiveness of the proposed regularization term. For example, with the help of the proposed regularization, the OoD detection score increases at most 30% compared with the one without the regularization.
BibTeX
@inproceedings{song2024robustly,
  author = {Kun Song and Ruben Solozabal and Hao Li and Martin Takác and Lu Ren and Fakhri Karray},
  title = {{Robustly Train Normalizing Flows via KL Divergence Regularization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i13.29426},
}
Figure