Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation

Haoyi Wu, Kewei Tu

TL;DR

Researchers have derived a "Probabilistic Transformer" from first principles of classical syntactic modeling using Conditional Random Fields (CRFs) and Variational Inference. This work reveals a striking mathematical equivalence between traditional probabilistic dependency parsing and modern self-attention, achieving competitive performance with significantly fewer parameters.

What Problem Does This Paper Address?

For decades, natural language processing (NLP) relied on explicit syntactic structures—like dependency trees—to understand language. The "deep learning revolution" changed this, ushering in the era of Transformers. While Transformers are incredibly successful and appear to learn syntax implicitly, their design is largely heuristic, born from engineering intuition rather than formal linguistic theory.

This paper tackles the "black box" nature of Transformers by asking: Can we derive a model for contextual word representation starting from a purely syntactic and probabilistic perspective? The goal is to bridge the gap between traditional, linguistically-principled models and cutting-edge neural architectures.

Methods

The authors propose the Probabilistic Transformer, built on two classical pillars:

  1. Conditional Random Fields (CRFs): They design a CRF that models two types of latent (hidden) variables for every word in a sentence: a discrete label ($Z_i$) representing its syntactic/semantic property, and a dependency head ($H_i$) pointing to its parent word in a syntactic tree.
  2. Mean Field Variational Inference (MFVI): Since exact inference in such a complex model is computationally impossible (NP-Hard), they use MFVI to approximate the marginal distributions of these latent variables.

The "aha!" moment comes when they tensorize the MFVI update equations. They discover that the iterative process of updating these distributions is mathematically almost identical to the self-attention mechanism in Transformers. Specifically:

  • Dependency heads correspond to self-attention scores.
  • Distributions over latent labels correspond to contextual word embeddings.
  • Multi-channel updates in the CRF correspond to multi-head attention.
Figure 2: Computation graphs for multi-head attention in transformers and multi-channel update in probabilistic transformers. See an explanation of replacing concat+linear with linear+sum in the upper part of multi-head attention in Section 3.3.

To make the model practical, they incorporate extensions like relative positional encoding (via distance-sensitive potential functions) and tensor decomposition to keep the parameter count low.

Datasets

The model was evaluated across a diverse suite of NLP benchmarks:

  • Masked Language Modeling (MLM): Penn TreeBank (PTB) and BLLIP-XS.
  • Sequence Labeling: POS tagging on PTB and Universal Dependencies (UD); Named Entity Recognition (NER) on CoNLL-2003.
  • Text Classification: Stanford Sentiment Treebank (SST-2 and SST-5).
  • Syntactic Generalization: The COGS dataset, designed to test if models can handle novel combinations of known syntactic structures.

Key Insights and Ideas

  • Theoretical Convergence: The most significant finding is that a purely probabilistic model of syntax is a Transformer in disguise. This provides a theoretical grounding for why self-attention works so well for language.
  • Parameter Efficiency: Because the model shares parameters across iterations and uses tied matrices derived from tensor decomposition, it is much leaner than a standard Transformer. The authors report using 1/5 to 1/2 the parameters of their Transformer baselines while maintaining performance.
  • Innate Syntax: Unlike standard Transformers, this model has an "innate" understanding of dependency structures. This is reflected in its superior performance on the COGS syntactic generalization test.
Full computation graph comparison

Key Results

The Probabilistic Transformer performs competitively with standard Transformers on small-to-medium datasets:

  • NER (CoNLL-2003): Achieved an F1 score of 75.47, outperforming the Transformer baseline of 74.02.
  • Syntactic Test (COGS): Reached 84.60% sentence-level accuracy, compared to 82.05% for the Transformer.
  • Sentiment Analysis (SST-5): Scored 42.77% accuracy vs. 40.13% for the Transformer.
  • Efficiency: The model achieves these results with a fraction of the parameters, though it is currently slower to train (~3x) due to the complexity of its positional encoding implementation.

Limitations and Future Work

While the model excels on smaller datasets, it currently lacks scalability. Preliminary tests on large-scale data show it underperforming compared to standard Transformers. The authors attribute this partly to the absence of a "feed-forward" structure, which in standard Transformers acts as a key-value memory.

Future directions include:

  • Integrating a feed-forward-like structure (perhaps via "global variables" representing topics).
  • Moving from discrete latent labels to continuous vectors to increase representational power.
  • Extending the model to include a decoder for generative tasks like translation.
  • Optimizing the implementation to improve training and inference speed.

Summary updated

Abstract

Syntactic structures used to play a vital role in natural language processing (NLP), but since the deep learning revolution, NLP has been gradually dominated by neural models that do not consider syntactic structures in their design. One vastly successful class of neural models is transformers. When used as an encoder, a transformer produces contextual representation of words in the input sentence. In this work, we propose a new model of contextual word representation, not from a neural perspective, but from a purely syntactic and probabilistic perspective. Specifically, we design a conditional random field that models discrete latent representations of all words in a sentence as well as dependency arcs between them; and we use mean field variational inference for approximate inference. Strikingly, we find that the computation graph of our model resembles transformers, with correspondences between dependencies and self-attention and between distributions over latent representations and contextual embeddings of words. Experiments show that our model performs competitively to transformers on small to medium sized datasets. We hope that our work could help bridge the gap between traditional syntactic and probabilistic approaches and cutting-edge neural approaches to NLP, and inspire more linguistically-principled neural approaches in the future.
BibTeX
@inproceedings{wu2023probabilistic,
  author = {Haoyi Wu and Kewei Tu},
  title = {{Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=hC2_8rD8Hq},
}
Figure