Knowledge-Based Regularization in Generative Modeling

Naoya Takeishi, Yoshinobu Kawahara

Abstract

Modern generative models like Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) are powerful but often ignore existing domain knowledge, requiring large datasets to learn basic feature relationships. This paper introduces a novel regularization method that incorporates prior domain knowledge about "relative dependence" into these models. By using a kernel-based dependence measure, the Hilbert-Schmidt Independence Criterion (HSIC), the authors enforce constraints such as "Feature A should be more related to B than to C." This approach improves model generalization and sample efficiency across diverse datasets, from industrial sensors to solar energy production, without requiring problem-specific architecture changes.

Introduction

Generative modeling faces a persistent trade-off between the interpretability of graphical models and the flexibility of deep neural networks. Graphical models allow experts to hard-code conditional independencies but are labor-intensive and require a complete understanding of the data's structure. Conversely, general-purpose neural networks are easy to deploy but often fail to respect known physical or logical constraints, leading to poor generalization in data-scarce regimes.

This research addresses this gap by introducing a "soft" constraint mechanism. Instead of redesigning the network architecture, the authors propose a regularizer that nudges standard models to respect partial domain insights. The focus is on relative dependence—a practical way for experts to specify that certain features are more closely related than others based on physical proximity or functional connection.

Methodology: Relative Dependence and HSIC

The core of the proposed method is a regularizer that translates qualitative domain knowledge into a differentiable loss term.

Defining Knowledge

Prior knowledge is represented as a set of triples $(J_{ref}, J^+, J^-)$, signifying that the features in set $J_{ref}$ should be more dependent on those in $J^+$ than those in $J^-$. This is often easier for experts to define than absolute correlation values. For instance, in an industrial plant, sensors within the same processing unit are naturally more related than those in distant units.

Translating domain knowledge from physical units (a) to relative feature dependencies (b).

The HSIC Regularizer

The authors utilize the Hilbert-Schmidt Independence Criterion (HSIC) to measure dependence. The regularizer penalizes the model when the HSIC between the "more dependent" pair is not significantly larger than the HSIC of the "less dependent" pair. Because HSIC gradients are computable via standard backpropagation, this regularizer can be integrated into any training loop for VAEs or GANs. This method effectively imposes probabilistic constraints, forcing the model to pass statistical tests of relative dependence during the generation process.

Experimental Evaluation

The method was evaluated across four distinct benchmarks:

  1. Toy Cylinder Data: A 3D dataset with known coordinate dependencies.
  2. CCMNIST (Constrained Concatenated MNIST): A synthetic benchmark where stacked digits share labels, creating a controlled environment for testing dependency learning.
  3. PLANT (Plant Sensor Data): Simulated data from the Tennessee Eastman process, where knowledge is derived from the physical plant layout.
  4. SOLAR (Solar Energy Production): Real-world data from 137 solar plants, where geographical proximity serves as the basis for relative dependence.
Samples from the CCMNIST dataset where top and middle digits are constrained to be similar.

Key Results and Insights

The experiments yielded several critical insights:

  • Improved Generalization: On the CCMNIST dataset, the regularized VAE significantly outperformed standard versions, approaching the performance of models specifically hand-designed for that data structure.
  • Enhanced Reconstruction: In the SOLAR and PLANT datasets, the proposed method achieved lower reconstruction errors than standard L2 regularization, demonstrating that respecting domain knowledge leads to better latent representations.
  • Mitigating Mode Collapse: In GAN training, the regularizer helped the generator capture the full geometry of the data distribution, a common failure point for standard GANs.
  • Scalability of Knowledge: Performance improved monotonically as more knowledge triples were added, suggesting the model effectively "absorbs" provided information.

Conclusion and Future Work

This work demonstrates that incorporating relative dependence constraints via HSIC is a powerful, architecture-agnostic way to improve generative models. While the current approach is effective, it faces computational challenges as the number of knowledge triples grows, scaling linearly with the size of the knowledge base. Future research may explore higher-order dependencies involving more than three sets of variables and further optimize the computational efficiency of the HSIC gradients for massive datasets.

Summary updated

Abstract

Prior domain knowledge can greatly help to learn generative models. However, it is often too costly to hard-code prior knowledge as a specific model architecture, so we often have to use general-purpose models. In this paper, we propose a method to incorporate prior knowledge of feature relations into the learning of general-purpose generative models. To this end, we formulate a regularizer that makes the marginals of a generative model to follow prescribed relative dependence of features. It can be incorporated into off-the-shelf learning methods of many generative models, including variational autoencoders and generative adversarial networks, as its gradients can be computed using standard backpropagation techniques. We show the effectiveness of the proposed method with experiments on multiple types of datasets and generative models.
BibTeX
@inproceedings{takeishi2020knowledge,
  author = {Naoya Takeishi and Yoshinobu Kawahara},
  title = {{Knowledge-Based Regularization in Generative Modeling}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/331},
}
Figure