Learning Language Representations with Logical Inductive Bias

Jianshu Chen

Introduction

Transformer architectures have achieved massive success in natural language processing by learning strong language representations from large-scale unlabeled text. However, this paper introduces FOLNet (First-Order Logic Network), a novel neural architecture that seeks to go beyond standard statistical correlations by baking a "logical inductive bias" directly into the learning process. By treating the computation of language features as a differentiable logic reasoning process, FOLNet achieves superior performance and stronger transfer capabilities compared to standard Transformer models like BERT and RoBERTa.

While Transformer models have revolutionized NLP, their success largely stems from a "relational inductive bias"—the ability to model pairwise interactions between tokens via self-attention. However, human language isn't just about statistical correlations; it follows structured, logical rules.

The authors argue that current models lack a formal logical inductive bias. They seek to bridge the gap between deep learning and symbolic logic by viewing language representation not just as a series of matrix multiplications, but as a step-by-step logic reasoning process. The goal is to create a model that learns "logic deduction skills" during pretraining, which can then be more effectively applied to diverse downstream tasks.

Methods: Building the FOLNet Architecture

The core innovation of this paper is FOLNet, a principled architecture built on the foundations of First-Order Logic (FOL).

  • Neural Logic Operators: The authors define a set of learnable "neural logic operators" (such as join, assoc, mu, and prod) that act as differentiable versions of Horn clauses (if-then rules).
  • Dual-Branch Architecture: Unlike the single-branch structure of a standard Transformer, FOLNet maintains two interacting branches throughout the network:
  • Unary Branch: Models properties of individual tokens (e.g., "is this word a noun?").
  • Binary Branch: Explicitly models relations between pairs of tokens (e.g., "is word A the subject of word B?").
  • Forward-Chaining: These operators are stacked into layers, mimicking the "forward-chaining" process in logic where simple facts are combined to deduce more complex "advanced" properties.
FOLNet Architecture

Interestingly, the authors mathematically demonstrate that the standard self-attention mechanism is actually a special, "degenerated" case of their join and assoc operators. This suggests that FOLNet is a more general and powerful evolution of the Transformer.

Key Insights and Ideas

  • Logic as a First-Class Citizen: By forcing the model to follow the principles of first-order logic, the learned representations are more structured and transferable.
  • Explicit Binary Relations: Maintaining a dedicated branch for binary relations (token-to-token) throughout the network—rather than just computing them on-the-fly in attention heads—is a major performance driver.
  • Unified Positional Encoding: The framework provides a new way to understand positional encodings. Popular methods like Relative Positional Encoding (RPE) are shown to be simplified versions of FOLNet’s logic operators.
  • Zero-Shot Power: Because the model learns "how to reason" rather than just "how to predict the next word," it shows remarkably strong performance in zero-shot settings where no task-specific finetuning is provided.

Key Results

The models were pretrained on standard large-scale corpora (Wikipedia + BookCorpus for base models, and an extended 160GB dataset for large models) and evaluated across rigorous benchmarks including GLUE, SQuAD 2.0, and FOLIO.

Main Results Table

FOLNet consistently outperformed strong Transformer baselines across the board:

  • GLUE Benchmark: FOLNetBase (138M parameters) achieved an average score of 87.7, significantly beating BERTBase (83.1) and RoBERTaBase (83.3).
  • Large-Scale Performance: FOLNetLarge (437M) outperformed the much larger Megatron 3.9B model on GLUE and SQuAD 2.0.
  • Zero-Shot Transfer: On the GLUE development set, FOLNetLarge outperformed RoBERTaLarge by an average of 7.2 points in a zero-shot setting.
  • Reasoning Capability: On the logic-heavy FOLIO task, FOLNetBase achieved 64.2% accuracy, outperforming RoBERTaBase (57.8%) and even the 340M-parameter BERTLarge (62.3%).

Limitations and Future Work

While FOLNet presents a significant step forward, it does have limitations. The current implementation focuses on unary (1-ary) and binary (2-ary) predicates. Moving to higher-order predicates (3-ary or more) would lead to exponential increases in memory and computation costs. Additionally, while the model uses standard losses like Masked Language Modeling (MLM), the authors suggest that new pretraining objectives specifically designed for the binary predicate branch could unlock even more potential. Finally, while the results at 437M parameters are impressive, testing the architecture at the multi-billion parameter scale remains an exciting task for future research.

Summary updated

Abstract

Transformer architectures have achieved great success in solving natural language tasks, which learn strong language representations from large-scale unlabeled texts. In this paper, we seek to go further beyond and explore a new logical inductive bias for better language representation learning. Logic reasoning is known as a formal methodology to reach answers from given knowledge and facts. Inspired by such a view, we develop a novel neural architecture named FOLNet (First-Order Logic Network), to encode this new inductive bias. We construct a set of neural logic operators as learnable Horn clauses, which are further forward-chained into a fully differentiable neural architecture (FOLNet). Interestingly, we find that the self-attention module in transformers can be composed by two of our neural logic operators, which probably explains their strong reasoning performance. Our proposed FOLNet has the same input and output interfaces as other pretrained models and thus could be pretrained/finetuned by using similar losses. It also allows FOLNet to be used in a plug-and-play manner when replacing other pretrained models. With our logical inductive bias, the same set of ``logic deduction skills'' learned through pretraining are expected to be equally capable of solving diverse downstream tasks. For this reason, FOLNet learns language representations that have much stronger transfer capabilities. Experimental results on several language understanding tasks show that our pretrained FOLNet model outperforms the existing strong transformer-based approaches.
BibTeX
@inproceedings{chen2023learning,
  author = {Jianshu Chen},
  title = {{Learning Language Representations with Logical Inductive Bias}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=rGeZuBRahju},
}

Loading related content...

Figure