Knowledge-Based Distant Regularization in Learning Probabilistic Models

Naoya Takeishi, Kosuke AkimotoarXiv 2018 · arXiv

Abstract

This paper introduces a framework for "distant regularization," allowing researchers to bake domain knowledge from knowledge graphs into probabilistic models. By linking data attributes to entities in a graph and using their embeddings to define parameter priors, the authors improve model generalization—especially when training data is scarce or direct relationships between variables are unknown.

Introduction

In statistical machine learning, "inductive bias"—the assumptions we make to help a model learn—is the secret sauce for good performance. Usually, we use generic regularization like Tikhonov (keeping weights small) or sparsity (keeping most weights at zero). However, we often have rich domain knowledge that doesn't fit these molds.

For example, in a factory, we might know that "Sensor A is attached to Pump B," which is "connected to Tank C." This is "distant" knowledge: it tells us about the system's structure, but it doesn't directly tell us the statistical correlation between Sensor A's voltage and Tank C's pressure. Standard methods like graph Laplacian regularization require knowing those direct correlations upfront. This paper bridges that gap, allowing models to leverage structured, symbolic knowledge graphs (KG) to guide the learning of continuous probabilistic models.

Methods: Knowledge-Based Distant Regularization

The authors propose a Knowledge-Based Distant Regularization framework. The process works in three main steps:

  1. Entity Mapping: Data attributes (like specific sensors or variables) are mapped to entities in a Knowledge Graph.
  2. Embedding Learning: The KG is converted into continuous-valued vectors (embeddings) using a method like DistMult. These embeddings capture the relationships between entities (e.g., "is-part-of" or "is-neighbor-of") in a high-dimensional space.
  3. Parameter Prior Specification: Instead of using a standard normal distribution as a prior for model parameters, the authors use the KG embeddings. Specifically, for a model like Factor Analysis (FA), they define the rows of the factor loading matrix as a function of the corresponding entity's embedding.
Conceptual overview of bridging a symbolic knowledge graph with sub-symbolic probabilistic modeling.

To keep the optimization tractable, the authors simplify the prior to a "point mass," essentially forcing the model's local parameters to be a direct transformation of the KG embeddings. The entire system—the probabilistic model and the KG embeddings—can be learned simultaneously by maximizing a joint objective function.

Experimental Setup and Datasets

The researchers evaluated their approach using a real-world global rainfall dataset:

  • Precipitation Data: Monthly mean rainfall measurements from 1901 to 2015 across 227 countries and regions.
  • Knowledge Graph: A custom-built graph containing 278 entities and 1,167 "positive tuples" (facts). It encoded geographic relationships such as is-inside (e.g., Norway is inside Europe) and is-neighbor-of (e.g., Norway neighbors Sweden).

Key Results

The authors compared their KG-regularized Factor Analysis against a standard version across two scenarios: Random data partitioning and Shift partitioning (to simulate real-world distribution changes).

Performance comparison (test NLL) across different training data proportions and KG tuple volumes for random (a) and shift (b) partitioning.
  • Improved Generalization: In both scenarios, the model using the Knowledge Graph (w/ KG) achieved a lower average test Negative Log-Likelihood (NLL) than the baseline. Lower NLL indicates a better fit to unseen data.
  • Robustness to Data Scarcity: The regularization acts as a guide. When the model has very little data to look at, it relies more on the "wisdom" encoded in the knowledge graph to make sensible predictions.
  • Performance vs. Knowledge Volume: There was a clear trend: as the proportion of the Knowledge Graph used for regularization increased, the test NLL decreased, demonstrating the value of the symbolic information.
  • Handling Distribution Shifts: In the "Shift" scenario, where training and testing data differ significantly, the KG-regularized model showed much more stable performance, whereas the baseline struggled as training data decreased.

Limitations and Future Work

While promising, the framework is in its early stages. The authors noted that the theoretical reasons why generalization improves so consistently need deeper mathematical analysis. Future work will explore more complex, non-linear mappings (like neural networks) between embeddings and parameters, as well as feedback loops where the probabilistic model helps refine or correct the embeddings in the knowledge graph. Additionally, scaling this approach to deep learning models with millions of parameters remains a significant challenge.

Summary updated

Abstract

Exploiting the appropriate inductive bias based on the knowledge of data is essential for achieving good performance in statistical machine learning. In practice, however, the domain knowledge of interest often provides information on the relationship of data attributes only distantly, which hinders direct utilization of such domain knowledge in popular regularization methods. In this paper, we propose the knowledge-based distant regularization framework, in which we utilize the distant information encoded in a knowledge graph for regularization of probabilistic model estimation. In particular, we propose to impose prior distributions on model parameters specified by knowledge graph embeddings. As an instance of the proposed framework, we present the factor analysis model with the knowledge-based distant regularization. We show the results of preliminary experiments on the improvement of the generalization capability of such model.
BibTeX
@misc{takeishi2018knowledge,
  author = {Naoya Takeishi and Kosuke Akimoto},
  title = {{Knowledge-Based Distant Regularization in Learning Probabilistic Models}},
  year = {2018},
  eprint = {1806.11332},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.11332},
}
Figure