Relational Learning with Variational Bayes

Kuang-Hung LiuICLR 2022

Overview

This paper introduces Variational Relational Learning (VRL), an unsupervised framework designed to learn abstract relationships between data points—such as "same-shape" or "rotated-by-90-degrees"—while ignoring the specific features of the objects themselves. By combining a probabilistic graphical model with a clever data augmentation strategy, the authors demonstrate that AI can learn to perceive "relational properties" as distinct from "absolute properties," a hallmark of human-like cognition.

The Challenge of Relational Learning

In psychology, "relational learning" is the ability to recognize how two things relate regardless of what those things are. For example, a child understands that two circles are "the same shape" in the same way two squares are, even though circles and squares look nothing alike.

In the machine learning world, "relational learning" usually refers to learning from structured data like graphs or databases. Most current unsupervised models (like Variational Autoencoders or contrastive learners) struggle to separate the relationship from the object. If you ask a standard model to learn the relationship between two images, it often gets distracted by the objects' colors, textures, or shapes.

The authors identify a specific gap: how do we build a model that learns relationships purely on the basis of relational properties rather than absolute ones, and do so without any human-provided labels?

Methods: Variational Relational Learning (VRL)

The authors propose VRL, which rests on two foundational pillars:

1. The VRL-PGM (Probabilistic Graphical Model)

They define the relationship as a latent variable $z$. In their model, the first object $a$ and the relationship $z$ are independent, but together they generate the second object $b$.

The VRL Probabilistic Graphical Model

This structure is designed to satisfy the constraint that the relationship shouldn't depend on the nature of the first object. The model is trained end-to-end using a variational lower bound (ELBO), similar to a standard VAE, but adapted for pairs of data.

2. Relation-Preserving Data Augmentation (RPDA)

This is the "secret sauce" that prevents the model from cheating. The authors identify a failure mode called the information-shortcut, where the model ignores the relationship and simply tries to memorize the features of object $b$. RPDA disrupts this by applying transformations (like rotating both images by the same random angle) that change the objects' appearance but keep their relationship identical. This forces the model to focus on the delta between the objects rather than the objects themselves.

Key Insights and Generalization

  • Independence is Key: For a relationship to be truly "abstract," it must be statistically independent of the objects it connects. VRL explicitly builds this assumption into the math of the graphical model.
  • The "Shortcut" Problem: Relational learning is hard because neural networks are inherently "lazy." If a network can predict object $b$ just by looking at $b$ (the shortcut), it will never bother learning the relationship $z$. RPDA provides a necessary nudge to break this habit.
  • Generalization to the Unseen: Because VRL learns the concept of the relationship (like "rotation"), it can generalize. A model trained on the relationship between the digits '1' and '2' can successfully identify the same relationship between '8' and '9', even if it has never seen those digits in that context before.

Key Results

The researchers tested VRL across several domains to prove its versatility, including MNIST and Omniglot (rotated characters), the Yale Face Database (facial expressions and illumination changes), and RAVDESS (audio waveforms of emotional speech).

VRL significantly outperformed established baselines like Spatial Transformer Networks (STN), standard VAEs, and self-supervised methods like BYOL in unsupervised clustering tasks.

Clustering and generation of learned relationships
  • Accuracy: On MNIST and Omniglot rotation tasks, VRL achieved near-perfect clustering accuracy (~99%), whereas most baselines struggled to cross 60% or failed entirely to decouple the relationship from the digit identity.
  • Robustness: In a "coupled" experiment where specific rotations were only ever seen with specific digits during training, VRL was still able to successfully identify those rotations on a test set with different digits.
  • High-Level Perception: The model successfully clustered complex human traits, such as "happy-to-surprised" facial transitions and "calm-to-angry" speech transitions, without ever being told what a "face" or "emotion" is.

Limitations and Future Work

While highly effective, the authors acknowledge a few limitations:

  • The Directional Assumption: The current PGM assumes a directed relationship (from $a$ to $b$). While the authors show this can be bypassed by "swapping" data during training, a truly undirected formulation might be more elegant.
  • Coupling Limits: If a relationship is perfectly coupled with an absolute property in the real world (i.e., a certain relationship only ever exists between specific types of objects), VRL can only provide a "partial view" of the data.
  • RPDA Knowledge: Currently, the user must define the type of augmentation (e.g., "we are looking for rotation-like relationships"). Future work could involve automatically discovering which augmentations are relevant to the underlying data structure.

Summary updated

BibTeX
@inproceedings{liu2022relational,
  author = {Kuang-Hung Liu},
  title = {{Relational Learning with Variational Bayes}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
}

Loading related content...

Figure