Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation

Haoyi Wu, Kewei Tu

Abstract

This paper introduces the Probabilistic Transformer, a neural architecture derived from first principles using traditional syntactic modeling. By applying Mean Field Variational Inference (MFVI) to a Conditional Random Field (CRF) based on dependency structures, we demonstrate that the resulting message-passing computation graph is mathematically equivalent to the self-attention mechanism. The Probabilistic Transformer is significantly more parameter-efficient than standard models, using 1/5 to 1/2 the parameters, while achieving superior performance on syntactic generalization tasks and competitive results on standard NLP benchmarks.

Introduction

Since their inception, Transformers have become the standard for natural language processing. However, their design remains largely heuristic, relying on empirical success rather than formal mathematical derivation from linguistic theory. While these models are known to capture syntactic structures, they were not explicitly designed to do so. This research bridges the gap between classical probabilistic NLP and modern deep learning by deriving a Transformer-like architecture starting from a formal model of syntax.

The Probabilistic Model

The foundation of this work is a Conditional Random Field (CRF) that explicitly models the syntactic and semantic properties of language through two sets of variables: discrete Label Variables ($Z$), representing word properties, and Dependency Head Variables ($H$), specifying the parent-child relationships between words in a sentence.

The factor graph of the proposed probabilistic model, illustrating the relationship between dependency head variables (H) and label variables (Z).

To perform inference in this complex model, the authors utilize Mean Field Variational Inference (MFVI). This iterative process refines the distributions of these variables by passing messages across the graph. A key mathematical breakthrough of this paper is the discovery that the update equations for these messages map directly onto the components of a Transformer. Specifically, dependency head distributions correspond to self-attention scores, and latent label distributions correspond to contextual embeddings.

Architectural Equivalence

When the MFVI updates are unrolled into a neural network, the resulting "Probabilistic Transformer" shares a striking structural similarity with standard Transformer variants, such as the Pre-LN Transformer. However, there are critical differences: the Probabilistic Transformer shares parameters across its $T$ iterations and notably lacks the large Feed-Forward Networks (FFNs) that typically account for two-thirds of a standard Transformer's parameters.

Comparison of the standard Transformer, Pre-LN Transformer, and the proposed Probabilistic Transformer architecture.

Experimental Results

The model was evaluated across several benchmarks, including Masked Language Modeling (MLM), sequence labeling (POS tagging and NER), and text classification (SST-2/5).

  • Syntactic Generalization: On the COGS dataset, which tests structural understanding, the Probabilistic Transformer outperformed the standard Transformer (84.60% vs 82.05% accuracy), suggesting that its principled syntactic inductive bias aids in generalizing to novel sentence forms.
  • Parameter Efficiency: The model is remarkably lean, maintaining competitive performance on POS tagging and NER while using a fraction of the parameters of its heuristic counterparts.
  • Performance Trade-offs: While competitive in most areas, the model showed higher perplexity in MLM tasks. This is likely due to the absence of high-capacity FFN layers, which the authors suggest may be responsible for capturing global features like topic information in standard models.

Discussion and Conclusion

The Probabilistic Transformer provides a theoretical grounding for the success of self-attention, showing it to be an approximation of probabilistic inference in a dependency-based world model. While the current iterative nature of the model results in slower training speeds (approximately 3x slower than standard Transformers) and challenges in scaling to massive datasets, it offers a more interpretable and parameter-efficient path forward. Future work will explore moving from discrete latent labels to continuous vector spaces to further bridge the performance gap on large-scale language tasks.

Summary updated

Abstract

Syntactic structures used to play a vital role in natural language processing (NLP), but since the deep learning revolution, NLP has been gradually dominated by neural models that do not consider syntactic structures in their design. One vastly successful class of neural models is transformers. When used as an encoder, a transformer produces contextual representation of words in the input sentence. In this work, we propose a new model of contextual word representation, not from a neural perspective, but from a purely syntactic and probabilistic perspective. Specifically, we design a conditional random field that models discrete latent representations of all words in a sentence as well as dependency arcs between them; and we use mean field variational inference for approximate inference. Strikingly, we find that the computation graph of our model resembles transformers, with correspondences between dependencies and self-attention and between distributions over latent representations and contextual embeddings of words. Experiments show that our model performs competitively to transformers on small to medium sized datasets. We hope that our work could help bridge the gap between traditional syntactic and probabilistic approaches and cutting-edge neural approaches to NLP, and inspire more linguistically-principled neural approaches in the future.
BibTeX
@inproceedings{wu2023probabilistic,
  author = {Haoyi Wu and Kewei Tu},
  title = {{Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=hC2_8rD8Hq},
}
Figure