Latent Wishart Processes for Relational Kernel Learning

Wu-Jun Li, Zhihua Zhang, Dit-Yan YeungAISTATS 2009 · OpenReview

Abstract

In this paper, we propose a novel relational kernel learning model based on latent Wishart processes (LWP) to learn the kernel function for relational data. This is done by seamlessly integrating the relational information and the input attributes into the kernel learning process. Through extensive experiments on diverse real-world applications, we demonstrate that our LWP model can give very promising performance in practice.

Introduction

Traditional machine learning models often treat data points as independent and identically distributed (i.i.d.). However, real-world data is frequently relational: web pages are connected by hyperlinks, scientific papers by citations, and proteins by interactions. These links often contain more information than the raw attributes of the objects themselves. The challenge lies in combining these two distinct views—attributes and relationships—into a single mathematical framework. Standard kernel methods are highly sensitive to the choice of the kernel function; if a kernel only considers attributes, it ignores the rich structural information of the network.

Methodology: Latent Wishart Processes

The core of our approach is the Latent Wishart Process (LWP), which provides a probabilistic framework for Relational Kernel Learning (RKL). The model follows a structured Bayesian approach:

  1. The Attribute Prior: We define a scale matrix $\Sigma$ based on a standard kernel (e.g., linear) applied to the input attributes. This represents our initial similarity estimate.
  2. The Wishart Process: We model the target kernel matrix $A$ as following a Wishart distribution $W_n(q, \Sigma)$. This allows $A$ to be decomposed into $BB^T$, where $B$ represents a set of $q$ latent features.
  3. The Link Likelihood: The existence of a link between two points is treated as a Bernoulli random variable, with the probability determined by the learned similarity in the latent space $B$ via a logistic link function.
  4. Learning: The model uses Maximum A Posteriori (MAP) estimation to find the optimal latent matrix $B$. This process effectively nudges the attribute-based similarities to align with the observed network topology.

Crucially, this kernel learning phase is unsupervised—it discovers the underlying data structure using only attributes and the link graph, without requiring class labels.

Key Insights and Visualization

LWP serves as a generalization of existing models like Relational Gaussian Processes (RGP). While RGP typically learns a single latent process ($q=1$), LWP learns $q$ independent latent processes, offering significantly more flexibility.

One of the most compelling features of LWP is its visualization power. By learning a latent representation $B$, the model can project data into a lower-dimensional space. Even when starting from a noisy initial state, the LWP learning process clearly separates different classes in 2D space without ever seeing the class labels.

Comparison of latent space representations between KPCA and LWP, demonstrating superior class separation in LWP.

Furthermore, the model exhibits remarkable robustness to hyperparameters. Experiments show that performance remains stable across a wide range of values for $q$ (the dimensionality of the latent space), capturing the essential data structure even with a small number of latent features.

Average AUC vs. latent dimensionality $q$, showing LWP's consistent performance across different settings.

Experimental Results

We evaluated LWP on three benchmark relational datasets: WebKB (university web pages), Cora (citation network), and Political Books (Amazon co-purchase network). LWP consistently outperformed standard Gaussian Process Classifiers (GPC) and other relational models (RGP, XGP):

  • WebKB: On the Texas subset, LWP achieved an AUC of 0.960, compared to 0.799 for a standard GPC.
  • Cora: LWP achieved near-perfect AUCs between 0.986 and 0.998, significantly outperforming benchmarks which typically hovered around 0.86–0.91.
  • Political Books: LWP matched state-of-the-art performance with an AUC of 0.98, proving effective even on smaller datasets.

Conclusion

Latent Wishart Processes provide a robust and flexible framework for learning kernels from relational data. By treating links as a primary source of information to refine attribute-based priors, LWP achieves superior classification performance and provides highly discriminative latent representations. Future work will extend this model to handle directed graphs and antagonistic (negative) relationships.

Summary updated

Abstract

In this paper, we propose a novel relational kernel learning model based on latent Wishart processes (LWP) to learn the kernel function for relational data. This is done by seamlessly integrating the relational information and the input attributes into the kernel learning process. Through extensive experiments on diverse real-world applications, we demonstrate that our LWP model can give very promising performance in practice.
BibTeX
@inproceedings{li2009latent,
  author = {Wu-Jun Li and Zhihua Zhang and Dit-Yan Yeung},
  title = {{Latent Wishart Processes for Relational Kernel Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2009},
  url = {https://openreview.net/forum?id=xUNP6-K665s},
}
Figure