Learning Invariants through Soft Unification

Nuri Cingillioglu, Alessandra Russo

Introduction

Bridging the gap between the pattern-recognition capabilities of deep learning and the symbolic reasoning of classical AI remains a central challenge in machine learning. While humans can effortlessly abstract general principles from a handful of examples—understanding that if "Bernhard is a frog" and "Bernhard is green," then other frogs may also be green—traditional neural networks typically require massive datasets to approximate such logic. These models often fail to generalize when specific entities, such as names or locations, change in the test set. Conversely, symbolic AI handles variables with ease but requires labor-intensive manual engineering of rules.

This paper introduces Unification Networks, an end-to-end differentiable framework designed to learn the concept of a variable and discover invariants directly from data. By employing a process called soft unification, these networks "lift" specific instances into general rules, enabling them to solve complex reasoning tasks with significantly higher data efficiency than standard architectures.

The Unification Framework

The core of the proposed approach is a five-step differentiable process that allows a model to bind specific input values to abstract templates.

  1. Invariant Selection: The model identifies a single "template" example ($G$) representative of the task.
  2. Lifting: A learnable function ($\psi$) determines which symbols in the template should act as variables. For example, in the sentence "Bernhard is a frog," the model might designate "Bernhard" as a variable ($V:bernhard$).
  3. Unifying Features: The model computes features ($\phi_U$) that capture the functional role of a symbol rather than its literal identity. This allows the model to recognize that "Lily" in a new input plays the same role as "Bernhard" in the template.
  4. Soft Unification: Using an attention mechanism, the model replaces variables in the template with specific values from the new input, effectively binding the data to the abstract rule.
  5. Downstream Prediction: An upstream network ($f$) utilizes this unified representation to generate the final output.
The lifting and soft unification process for a sequence task.

To ensure the model does not over-generalize by turning every symbol into a variable, the authors introduce a sparsity constraint. This penalty forces the model to find the minimal set of variables necessary to represent the underlying principle, preserving the meaningful structure of the template.

Experimental Evaluation

The authors evaluated Unification Networks across four architectures—UMLP, UCNN, URNN, and Unification Memory Networks (UMN)—on five diverse datasets ranging from synthetic grid tasks and logical reasoning programs to the bAbI natural language benchmark and real-world sentiment analysis.

A primary finding is the extreme data efficiency of the unification approach. In "low-data" regimes, Unification Networks frequently achieve near-perfect accuracy with as few as 50 training examples, whereas standard architectures fail to converge or require thousands of examples to reach comparable performance.

Test accuracy comparison across different architectures and training sizes.

On the bAbI benchmark, specifically Task 16 (basic induction), the UMN achieved perfect scores by learning a single invariant: "if someone is the same thing as someone else, they have the same color." Similarly, on logical reasoning tasks, the model successfully navigated multi-hop chains of transitivity and negation.

Discussion and Future Directions

Beyond performance gains, Unification Networks offer a degree of interpretability. By examining the "V:" prefixes assigned during the lifting process, researchers can see exactly which symbols the model has abstracted into variables. This provides a transparent window into the model's reasoning process, showing, for instance, how it identifies sentiment-bearing words like "silly" as high-variableness symbols in movie reviews.

However, the framework has limitations. Its performance is inherently bounded by the capabilities of the "upstream" network; if the base architecture cannot handle a specific reasoning type (like complex pathfinding), soft unification cannot compensate for that structural deficit. Additionally, while the model can handle multiple invariants, selecting the correct pattern for highly complex, multi-modal tasks remains an area for future research. The authors suggest that moving toward context-sensitive variables and applying these techniques to visual question answering represent promising next steps in the evolution of neuro-symbolic integration.

Summary updated

BibTeX
@inproceedings{cingillioglu2020learning,
  author = {Nuri Cingillioglu and Alessandra Russo},
  title = {{Learning Invariants through Soft Unification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=LcJtv9vIlEsX},
}
Figure