Learning Language Representations with Logical Inductive Bias

Jianshu Chen

Abstract

Transformer architectures have achieved remarkable success in natural language processing by learning strong representations from large-scale unlabeled text. However, these models primarily rely on a relational inductive bias—modeling pairwise connections between tokens. In this paper, we explore a more structured approach: the logical inductive bias. We introduce FOLNet (First-Order Logic Network), a novel neural architecture that treats language representation as a differentiable process of first-order logic reasoning. By constructing neural logic operators as learnable Horn clauses, FOLNet performs forward-chaining deduction to build deep linguistic representations. Our theoretical analysis reveals that the standard self-attention mechanism is actually a specialized case of these broader logical operations. Experimental results demonstrate that FOLNet outperforms strong Transformer-based baselines on benchmarks like GLUE, SQuAD 2.0, and FOLIO, exhibiting superior systematic generalization and transfer capabilities.

Introduction

The evolution of Large Language Models (LLMs) has been driven by the Transformer's ability to capture complex dependencies through self-attention. While effective, this mechanism is often criticized for its lack of systematicity—the ability to apply learned rules consistently across different contexts or sequence lengths. Traditional logic, specifically First-Order Logic (FOL), offers a formal methodology for reaching conclusions from facts through rigorous rules.

We propose bridging the gap between deep learning and formal logic by embedding a logical inductive bias directly into the model architecture. FOLNet is designed to mimic the process of logical deduction, where input tokens are treated as constants and their embeddings represent predicates. By stacking layers that act as differentiable logic operators, the model "deduces" higher-level representations step-by-step.

The FOLNet Architecture

Logic as Architecture

In FOLNet, the network layers represent a forward-chaining process. We define a set of neural logic operators—such as join, assoc, mu, and prod—that are implemented as learnable Horn clauses (if-then rules). These operators allow the model to derive new properties (predicates) from existing ones.

Dual-Branch Design

Unlike the single-branch structure of standard Transformers, FOLNet maintains two interacting branches to preserve the distinction between individual entities and their relationships:

  1. Unary Branch: Encodes properties of individual tokens (e.g., lexical category or semantic role).
  2. Binary Branch: Encodes relations between pairs of tokens (e.g., subject-verb-object dependencies).

This dual-branch approach ensures that relational information is not lost or compressed into a single vector space as the network grows deeper, leading to more robust and interpretable representations.

The Transformer Connection

A significant theoretical finding of this work is the relationship between FOLNet and the Transformer. We prove that the standard self-attention mechanism can be reconstructed using a combination of FOLNet’s join and assoc operators. This suggests that Transformers are essentially a simplified, "degenerated" version of a more general logic network, providing a new theoretical lens through which to understand their success and limitations.

Experimental Evaluation

General Language Understanding

We evaluated FOLNet across several rigorous benchmarks. In pretraining experiments using Wikipedia and BookCorpus, a FOLNet-Base model (138M parameters) outperformed BERT-Large (340M parameters) on the GLUE benchmark. This indicates that the logical inductive bias allows for higher parameter efficiency.

Logical Reasoning and Systematic Generalization

To test the model's reasoning depth, we utilized the FOLIO dataset and the CLUTRR benchmark:

  • FOLIO: FOLNet-Large achieved 70.6% accuracy, surpassing RoBERTa-Large (67.7%) and even significantly larger models like Megatron-3.9B.
  • CLUTRR: This task requires models to infer family relationships over varying reasoning chain lengths. FOLNet demonstrated a remarkable ability to generalize to longer reasoning chains than those seen during training, a scenario where standard Transformers typically fail.

Zero-Shot Transfer

FOLNet’s "logic deduction skills" proved highly transferable. In zero-shot evaluations on GLUE, FOLNet-Large outperformed RoBERTa-Large on 7 out of 8 tasks, with an average improvement of 7.2 points, suggesting that the learned logical patterns are more universal than standard associative patterns.

Discussion and Future Work

FOLNet introduces a powerful framework for incorporating formal reasoning into neural networks without sacrificing the efficiency of gradient-based learning. By treating self-attention as a subset of logical operations, we provide a path toward more systematic and interpretable AI.

Currently, FOLNet is limited to unary and binary predicates. Future work will explore extending the architecture to handle ternary relations (3-way interactions) and developing specialized pretraining objectives that specifically target the binary relational branch. Furthermore, scaling FOLNet to the 100B+ parameter range remains an exciting frontier for achieving human-level logical consistency in large-scale language modeling.

Summary updated

Abstract

Transformer architectures have achieved great success in solving natural language tasks, which learn strong language representations from large-scale unlabeled texts. In this paper, we seek to go further beyond and explore a new logical inductive bias for better language representation learning. Logic reasoning is known as a formal methodology to reach answers from given knowledge and facts. Inspired by such a view, we develop a novel neural architecture named FOLNet (First-Order Logic Network), to encode this new inductive bias. We construct a set of neural logic operators as learnable Horn clauses, which are further forward-chained into a fully differentiable neural architecture (FOLNet). Interestingly, we find that the self-attention module in transformers can be composed by two of our neural logic operators, which probably explains their strong reasoning performance. Our proposed FOLNet has the same input and output interfaces as other pretrained models and thus could be pretrained/finetuned by using similar losses. It also allows FOLNet to be used in a plug-and-play manner when replacing other pretrained models. With our logical inductive bias, the same set of ``logic deduction skills'' learned through pretraining are expected to be equally capable of solving diverse downstream tasks. For this reason, FOLNet learns language representations that have much stronger transfer capabilities. Experimental results on several language understanding tasks show that our pretrained FOLNet model outperforms the existing strong transformer-based approaches.
BibTeX
@inproceedings{chen2023learning,
  author = {Jianshu Chen},
  title = {{Learning Language Representations with Logical Inductive Bias}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=rGeZuBRahju},
}

Loading related content...

Figure