Abstract
Transformer architectures have achieved remarkable success in natural language processing by learning strong representations from large-scale unlabeled text. However, these models primarily rely on a relational inductive bias—modeling pairwise connections between tokens. In this paper, we explore a more structured approach: the logical inductive bias. We introduce FOLNet (First-Order Logic Network), a novel neural architecture that treats language representation as a differentiable process of first-order logic reasoning. By constructing neural logic operators as learnable Horn clauses, FOLNet performs forward-chaining deduction to build deep linguistic representations. Our theoretical analysis reveals that the standard self-attention mechanism is actually a specialized case of these broader logical operations. Experimental results demonstrate that FOLNet outperforms strong Transformer-based baselines on benchmarks like GLUE, SQuAD 2.0, and FOLIO, exhibiting superior systematic generalization and transfer capabilities.
Introduction
The evolution of Large Language Models (LLMs) has been driven by the Transformer's ability to capture complex dependencies through self-attention. While effective, this mechanism is often criticized for its lack of systematicity—the ability to apply learned rules consistently across different contexts or sequence lengths. Traditional logic, specifically First-Order Logic (FOL), offers a formal methodology for reaching conclusions from facts through rigorous rules.
We propose bridging the gap between deep learning and formal logic by embedding a logical inductive bias directly into the model architecture. FOLNet is designed to mimic the process of logical deduction, where input tokens are treated as constants and their embeddings represent predicates. By stacking layers that act as differentiable logic operators, the model "deduces" higher-level representations step-by-step.
The FOLNet Architecture
Logic as Architecture
In FOLNet, the network layers represent a forward-chaining process. We define a set of neural logic operators—such as join, assoc, mu, and prod—that are implemented as learnable Horn clauses (if-then rules). These operators allow the model to derive new properties (predicates) from existing ones.
Dual-Branch Design
Unlike the single-branch structure of standard Transformers, FOLNet maintains two interacting branches to preserve the distinction between individual entities and their relationships:
- Unary Branch: Encodes properties of individual tokens (e.g., lexical category or semantic role).
- Binary Branch: Encodes relations between pairs of tokens (e.g., subject-verb-object dependencies).
This dual-branch approach ensures that relational information is not lost or compressed into a single vector space as the network grows deeper, leading to more robust and interpretable representations.
The Transformer Connection
A significant theoretical finding of this work is the relationship between FOLNet and the Transformer. We prove that the standard self-attention mechanism can be reconstructed using a combination of FOLNet’s join and assoc operators. This suggests that Transformers are essentially a simplified, "degenerated" version of a more general logic network, providing a new theoretical lens through which to understand their success and limitations.
Experimental Evaluation
General Language Understanding
We evaluated FOLNet across several rigorous benchmarks. In pretraining experiments using Wikipedia and BookCorpus, a FOLNet-Base model (138M parameters) outperformed BERT-Large (340M parameters) on the GLUE benchmark. This indicates that the logical inductive bias allows for higher parameter efficiency.
Logical Reasoning and Systematic Generalization
To test the model's reasoning depth, we utilized the FOLIO dataset and the CLUTRR benchmark:
- FOLIO: FOLNet-Large achieved 70.6% accuracy, surpassing RoBERTa-Large (67.7%) and even significantly larger models like Megatron-3.9B.
- CLUTRR: This task requires models to infer family relationships over varying reasoning chain lengths. FOLNet demonstrated a remarkable ability to generalize to longer reasoning chains than those seen during training, a scenario where standard Transformers typically fail.
Zero-Shot Transfer
FOLNet’s "logic deduction skills" proved highly transferable. In zero-shot evaluations on GLUE, FOLNet-Large outperformed RoBERTa-Large on 7 out of 8 tasks, with an average improvement of 7.2 points, suggesting that the learned logical patterns are more universal than standard associative patterns.
Discussion and Future Work
FOLNet introduces a powerful framework for incorporating formal reasoning into neural networks without sacrificing the efficiency of gradient-based learning. By treating self-attention as a subset of logical operations, we provide a path toward more systematic and interpretable AI.
Currently, FOLNet is limited to unary and binary predicates. Future work will explore extending the architecture to handle ternary relations (3-way interactions) and developing specialized pretraining objectives that specifically target the binary relational branch. Furthermore, scaling FOLNet to the 100B+ parameter range remains an exciting frontier for achieving human-level logical consistency in large-scale language modeling.
Summary updated