Abstract
This paper introduces the Probabilistic Transformer, a neural architecture derived from first principles using traditional syntactic modeling. By applying Mean Field Variational Inference (MFVI) to a Conditional Random Field (CRF) based on dependency structures, we demonstrate that the resulting message-passing computation graph is mathematically equivalent to the self-attention mechanism. The Probabilistic Transformer is significantly more parameter-efficient than standard models, using 1/5 to 1/2 the parameters, while achieving superior performance on syntactic generalization tasks and competitive results on standard NLP benchmarks.
Introduction
Since their inception, Transformers have become the standard for natural language processing. However, their design remains largely heuristic, relying on empirical success rather than formal mathematical derivation from linguistic theory. While these models are known to capture syntactic structures, they were not explicitly designed to do so. This research bridges the gap between classical probabilistic NLP and modern deep learning by deriving a Transformer-like architecture starting from a formal model of syntax.
The Probabilistic Model
The foundation of this work is a Conditional Random Field (CRF) that explicitly models the syntactic and semantic properties of language through two sets of variables: discrete Label Variables ($Z$), representing word properties, and Dependency Head Variables ($H$), specifying the parent-child relationships between words in a sentence.
To perform inference in this complex model, the authors utilize Mean Field Variational Inference (MFVI). This iterative process refines the distributions of these variables by passing messages across the graph. A key mathematical breakthrough of this paper is the discovery that the update equations for these messages map directly onto the components of a Transformer. Specifically, dependency head distributions correspond to self-attention scores, and latent label distributions correspond to contextual embeddings.
Architectural Equivalence
When the MFVI updates are unrolled into a neural network, the resulting "Probabilistic Transformer" shares a striking structural similarity with standard Transformer variants, such as the Pre-LN Transformer. However, there are critical differences: the Probabilistic Transformer shares parameters across its $T$ iterations and notably lacks the large Feed-Forward Networks (FFNs) that typically account for two-thirds of a standard Transformer's parameters.
Experimental Results
The model was evaluated across several benchmarks, including Masked Language Modeling (MLM), sequence labeling (POS tagging and NER), and text classification (SST-2/5).
- Syntactic Generalization: On the COGS dataset, which tests structural understanding, the Probabilistic Transformer outperformed the standard Transformer (84.60% vs 82.05% accuracy), suggesting that its principled syntactic inductive bias aids in generalizing to novel sentence forms.
- Parameter Efficiency: The model is remarkably lean, maintaining competitive performance on POS tagging and NER while using a fraction of the parameters of its heuristic counterparts.
- Performance Trade-offs: While competitive in most areas, the model showed higher perplexity in MLM tasks. This is likely due to the absence of high-capacity FFN layers, which the authors suggest may be responsible for capturing global features like topic information in standard models.
Discussion and Conclusion
The Probabilistic Transformer provides a theoretical grounding for the success of self-attention, showing it to be an approximation of probabilistic inference in a dependency-based world model. While the current iterative nature of the model results in slower training speeds (approximately 3x slower than standard Transformers) and challenges in scaling to massive datasets, it offers a more interpretable and parameter-efficient path forward. Future work will explore moving from discrete latent labels to continuous vector spaces to further bridge the performance gap on large-scale language tasks.
Summary updated