Learning Invariants through Soft Unification

Nuri Cingillioglu, Alessandra Russo

Neural networks are exceptional at pattern recognition, but they often struggle with a fundamental aspect of human reasoning: the ability to use variables. While a human can easily learn a general rule like "if someone goes somewhere, they are now there" and apply it to any person or place, standard deep learning models typically require thousands of specific examples (e.g., John going to the Kitchen, Mary going to the Garden) to generalize.

Existing symbolic AI models use variables, but these are usually pre-engineered by humans. This paper addresses a critical challenge: Can a machine learn the concept of a variable and how to use it solely from examples, without human intervention? To solve this, the authors introduce Unification Networks, an end-to-end differentiable architecture that learns to "lift" specific data examples into general, symbolic-like patterns called invariants.

The Soft Unification Pipeline

The core of this approach is Soft Unification, a differentiable mechanism that mimics the logical process of matching expressions by substituting variables. The network learns which symbols act as variables and how to bind them to values by minimizing prediction error. The process follows five key steps:

  1. Pick an Invariant Example ($G$): The model selects a representative example from the dataset to serve as a template.
  2. Lift the Example: A learnable "variableness" function ($\psi$) determines which symbols in the template should act as variables (e.g., converting the specific entity "Bernhard" into a variable "V:Bernhard").
  3. Compute Unifying Features: The model learns features ($\phi_U$) that capture the role or structural position of a symbol rather than its specific identity.
  4. Soft Unification: Using an attention mechanism, the model maps the variables in the invariant template to the specific entities in a new, unseen example ($K$).
  5. Predict: An upstream predictor network ($f$) processes this "unified" representation to produce the final answer.
The Soft Unification pipeline, showing how an invariant example is lifted into variables and unified with a new example to make a prediction.

Because the entire pipeline is differentiable, it acts as a strong architectural prior. It forces the network to look for underlying principles that remain constant even when specific entities change.

Key Insights and Results

The researchers evaluated Unification Networks across five diverse domains, including fixed-length sequences, grid data, logical reasoning, sentiment analysis, and the bAbI natural language reasoning benchmark.

Data Efficiency is the Killer App: By learning a general rule (an invariant) rather than memorizing instances, the model becomes incredibly data-efficient. Unification Networks (UMLP, UCNN, URNN) consistently outperformed their "plain" counterparts, particularly in low-data regimes. In many cases, the model could solve tasks with as few as 50 examples where standard models completely failed.

Test accuracy across MLP, CNN, and RNN architectures, demonstrating that Unification Networks (orange) vastly outperform baselines (blue), especially in low-data regimes of 50 examples (dashed lines).

Interpretability through Sparsity: By applying a sparsity penalty to the variableness function, the model is encouraged to pick the minimum number of variables needed to solve a task. This results in "clean" invariants that humans can actually read and understand. Furthermore, in synthetic tasks where the "true" rule was known, the model successfully recovered the exact intended invariant up to 90% of the time.

Strong Performance on Benchmarks: On bAbI task 16 (basic induction), the model achieved a mean error of just 5.1% under strong supervision, outperforming several complex memory network baselines.

Limitations and Future Directions

While Unification Networks represent a significant step toward neuro-symbolic reasoning, some limitations remain:

  • Predictor Dependency: The system's success is capped by the upstream predictor network ($f$). If the base model cannot solve the task even with perfect unification, the overall system will fail.
  • The Weak Supervision Gap: Performance drops significantly under "weak supervision" (where the model isn't explicitly told which sentences are relevant to the query). The model struggles to find the correct chain of reasoning to unify in noisier contexts.
  • Scaling: The current work often relies on a single or a few invariant examples. Scaling this architecture to handle hundreds of complex, overlapping rules remains an open challenge.

Future work may extend this approach to multi-modal tasks, such as learning invariants that link visual objects in a video directly to variables in a textual description, further bridging the gap between raw perception and symbolic reasoning.

Summary updated

Abstract

Human reasoning involves recognising common underlying principles across many examples. The by-products of such reasoning are invariants that capture patterns such as "if someone went somewhere then they are there", expressed using variables "someone" and "somewhere" instead of mentioning specific people or places. Humans learn what variables are and how to use them at a young age. This paper explores whether machines can also learn and use variables solely from examples without requiring human pre-engineering. We propose Unification Networks, an end-to-end differentiable neural network approach capable of lifting examples into invariants and using those invariants to solve a given task. The core characteristic of our architecture is soft unification between examples that enables the network to generalise parts of the input into variables, thereby learning invariants. We evaluate our approach on five datasets to demonstrate that learning invariants captures patterns in the data and can improve performance over baselines.
BibTeX
@inproceedings{cingillioglu2020learning,
  author = {Nuri Cingillioglu and Alessandra Russo},
  title = {{Learning Invariants through Soft Unification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=LcJtv9vIlEsX},
}
Figure