Abstract
Prior domain knowledge can greatly help to learn generative models. However, it is often too costly to hard-code prior knowledge as a specific model architecture, so we often have to use general-purpose models. In this paper, we propose a method to incorporate prior knowledge of feature relations into the learning of general-purpose generative models. To this end, we formulate a regularizer that makes the marginals of a generative model to follow prescribed relative dependence of features. It can be incorporated into off-the-shelf learning methods of many generative models, including variational autoencoders and generative adversarial networks, as its gradients can be computed using standard backpropagation techniques. We show the effectiveness of the proposed method with experiments on multiple types of datasets and generative models.
Introduction
In fields such as engineering and physics, researchers often possess significant "prior knowledge" regarding how data should behave. For example, in a manufacturing plant, sensors physically connected within the same unit are expected to exhibit higher correlation than those located on opposite sides of the facility. Incorporating this domain expertise into modern generative models like Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) typically presents a difficult choice: either manually design a custom, labor-intensive architecture or rely on a general-purpose model that requires vast amounts of data to "re-learn" these known relationships.
Bridging the Gap with Knowledge-Based Regularization
This paper introduces a novel approach to "nudge" general-purpose models to respect domain-specific rules without redesigning their underlying architecture. The authors propose a Knowledge-Based Regularization method that modifies the model's loss function rather than its structure.
The core idea is to translate physical domain knowledge into relative dependence triples $(J_{ref}, J^+, J^-)$. This encodes a simple rule: a reference feature $J_{ref}$ should be more dependent on feature $J^+$ than it is on feature $J^-$.
To measure these dependencies, the authors utilize the Hilbert-Schmidt Independence Criterion (HSIC). Unlike standard correlation, HSIC is a kernel-based method capable of capturing complex, non-linear relationships. By adding an HSIC-based penalty term to the training loss, the model is incentivized to generate data that adheres to the prescribed feature relationships. This approach is model-agnostic and can be integrated into any gradient-based learning framework via standard backpropagation.
Experimental Evaluation
The proposed method was evaluated across four distinct datasets:
- Toy Data: A synthetic cylinder-shaped dataset used for geometric visualization.
- CCMNIST (Constrained Concatenated MNIST): A benchmark where stacked digits have specific label dependencies.
- PLANT (Tennessee Eastman Process): A chemical engineering simulator where knowledge is derived from physical piping layouts.
- SOLAR: Real-world solar production data where geographic proximity dictates feature dependence.
Key Results and Insights
The experiments demonstrated that even partial domain knowledge can significantly improve model performance and generalization. Across all datasets, the regularized models achieved lower reconstruction errors and better cross-entropy scores compared to standard L2 regularization.
Stabilizing GANs
A particularly notable result occurred with the toy dataset. Standard GANs often suffer from "mode collapse," failing to capture the full geometry of the data distribution. By incorporating the knowledge-based regularizer, the authors stabilized the GAN training, allowing it to successfully capture the entire cylinder geometry, whereas the unregularized version failed to do so.
Performance on CCMNIST
On the CCMNIST task, the regularized VAE significantly outperformed a standard VAE. While it did not quite reach the performance of a "dedicated" decoder manually designed for the task, it achieved remarkably similar results without the need for a custom architecture. Furthermore, the method proved to be computationally efficient, adding only a negligible amount of time to the total training process.
Conclusion
By formulating domain knowledge as probabilistic constraints via HSIC regularization, this work provides a practical way to integrate expert expertise into general-purpose generative models. This approach bridges the gap between flexible black-box models and rigid hand-crafted architectures, leading to better generalization and more stable training with less data. Future research may explore extending this framework to higher-order dependencies and assessing its robustness against noisy or incorrect prior knowledge.
Summary updated