Parameter Tying Across Relational Structures

29 concepts24 papers1 proposal

When applying machine learning to structured domains—such as relational databases, knowledge graphs, physical systems, or molecular structures—we face a fundamental challenge: the data does not fit into a neat, fixed-dimensional vector. If we attempt to model every individual entity, relation grounding, or interaction with its own unique parameter, we quickly encounter an explosion of parameters. This leads to severe overfitting, an inability to generalize to new or larger graphs, and a failure to capture the underlying laws of the domain.

To solve this, researchers across statistical relational learning and deep learning utilize relational parameter tying (or relational weight sharing). The core intuition is simple: bind model parameters to abstract templates, schemas, or logical rules rather than to specific data instances.

By forcing different parts of a network or graph to share the same underlying parameters, the model treats recurring structural patterns as repeated observations of the same underlying process. This acts as a powerful inductive bias, analogous to how convolutional neural networks share weights across space to achieve translation invariance.


The Core Mechanism: From Groundings to Templates

To understand why relational parameter tying works, consider a traditional propositional model where every variable or interaction is independent. In a relational domain, a single logical rule or schema can manifest in thousands of different concrete "groundings" (specific instances involving specific entities).

Instead of learning a separate parameter for every grounding, parameter tying maps these diverse local structures to a common set of templates. As pioneered by Koller et al. in their work on noisy first-order rules, every occurrence of a general rule is treated as an independent observation of the same parameter. This allows a model to aggregate evidence from highly diverse network topologies to estimate a single, robust shared value.

Without Tying (Parameter Explosion):
(Alice, FriendsWith, Bob)   --> Parameter W_1
(Charlie, FriendsWith, Dan) --> Parameter W_2

With Relational Tying (Template-Based):
(X, FriendsWith, Y)         --> Shared Parameter W_Friends

This concept forms the bedrock of Probabilistic Relational Models (PRMs). As shown by Friedman et al. in Learning Probabilistic Relational Models, all instances belonging to the same entity class share a single set of probabilistic parameters. This design enables models to reason about complex dependencies without losing information by "flattening" data into simple tables. Similarly, in directed probabilistic logical models, Ramon et al. utilized relational probability trees to map logical predicates to shared distribution parameters, ensuring a single learned dependency is reused across all similar object pairs (Ramon et al.).


Why It Works: Borrowing Statistical Strength

A persistent issue in multi-relational data, such as knowledge graphs, is data sparsity. A few relations occur frequently, while the vast majority appear only a handful of times. Traditional models that treat each relation type independently fail to exploit semantic similarities, meaning rare relations cannot learn effectively (Jenatton et al.).

Relational parameter tying directly mitigates this sparsity by allowing rare relations to borrow statistical strength from common ones. This can be achieved through several mechanisms:

  • Shared Latent Factors: Jenatton et al. demonstrated that decomposing relation-specific matrices into sparse combinations of shared latent factors allows patterns learned in common relations to improve predictions for rare, semantically similar ones (Jenatton et al.).
  • Shared Projection Pools: Instead of unique parameters per relation, the ITransF model proposed by Shieh et al. utilizes a shared pool of projection matrices, mapping entities into shared conceptual subspaces (Shieh et al.).
  • Gated Architectures: The TransGate model, introduced by Yuan et al., shares a single set of gating parameters across all relations, creating a common relational foundation that prevents parameter explosion (Yuan et al.).
  • Contextual Parameter Generation: Rather than using independent lookup tables, Stoica et al. generated model parameters through a shared mapping function (like an MLP), leveraging similarities between context types to accelerate training convergence (Stoica et al.).

By forcing the model to learn a shared foundation—such as sharing hidden layers across all relations in LogicENN (Yazdi et al.)—the learning algorithm captures general entity interactions before specializing for individual relation types.


Enforcing Domain Symmetries and Generalization

Beyond data efficiency, parameter tying provides a crucial structural guarantee: invariance to the size and topology of the graph.

When weights are tied to abstract rules or schemas rather than specific coordinates or nodes, the model's total parameter count remains constant, regardless of how many objects exist in a specific problem instance.

From Logic to Neural Networks

This property bridges symbolic logic and deep learning. In Lifted Relational Neural Networks (LRNNs), developed by Šourek et al., first-order logic rules serve as templates to dynamically construct example-specific neural architectures (Šourek et al., Šourek et al.). Because weights are bound to the abstract rules, the network enforces domain symmetries and performs a form of relational regularization.

Similarly, the NNRPT framework automatically learns rules via relational random walks, enforcing weight sharing across identical relational sub-structures to keep the model compact and focused on underlying logic rather than specific data instances (Kaur et al.).

Generalizing Policies and Physics

This exact mechanism enables generalized reinforcement learning and physical reasoning:

  • Action Schema Networks: In automated planning, Toyer et al. applied weight sharing across all ground actions and propositions that share the same underlying schema. This allows a policy learned on a small problem instance to generalize directly to much larger problem sizes (Toyer et al.).
  • Interaction Networks: For physical systems, Battaglia et al. implemented a shared relational module that processes pairs of objects to compute their mutual influence. Because the same parameters are applied to every interaction, the model learns universal physical laws that generalize across entirely different system configurations (Battaglia et al.).

Balancing Flexibility: Tied vs. Untied Models

While tying parameters provides massive statistical advantages, it introduces a strict constraint: it assumes that every grounding of a rule behaves identically. In the real world, this assumption can be too rigid. For instance, a rule like FriendsWith(X, Y) -> HasSimilarTaste(X, Y) might hold true with varying degrees of probability depending on the specific individuals involved.

Standard statistical relational models, like Markov Logic Networks (MLNs), traditionally rely on strict weight tying to maintain computational efficiency (Sarkhel et al.). However, this limits their ability to represent nuanced, real-world scenarios. To relax this constraint without suffering from parameter explosion, researchers have introduced hybrid approaches:

  1. Hierarchical Clustering as Regularization: Al Farabi et al. proposed using a non-relational learner to group similar weights together in "untied" MLNs. Forcing groundings with similar non-relational characteristics to share weights acts as a form of regularization, mitigating overfitting while preserving expressiveness (Al Farabi et al.).
  2. Tensor Quantization: To make inference scalable when weights are untied, Sarkhel et al. utilized tensor quantization to dynamically cluster similar weights during optimization, achieving order-of-magnitude speedups (Sarkhel et al.).
  3. Class-Based Edge Priors: In structure learning, Mansinghka et al. demonstrated a hierarchical Bayesian framework where edge probabilities are determined by the class assignments of the connected nodes (e.g., diseases vs. symptoms) rather than being fully tied or fully independent, allowing accurate network learning from significantly smaller datasets (Mansinghka et al.).

Ultimately, whether enforced strictly via first-order logic rules (Lowd et al.; Riguzzi et al.) or softly via shared neural layers, relational parameter tying provides the mathematical framework necessary to generalize from a single observed network (Neville et al.) to universal relational principles.

Key Foundational Work

Learning Probabilities for Noisy First-Order Rules (1997) — contributed 2 concept elements to this direction.
Learning Probabilistic Relational Models (1999) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_relational_parameter_tying_31b02420,
  title = {relational parameter tying},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relational-parameter-tying/rdr_31b02420a0d8490facae3305e4423c9d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_relational_parameter_tying_31b02420_v1,
  title = {relational parameter tying},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relational-parameter-tying/rdr_31b02420a0d8490facae3305e4423c9d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake