Consistent Bidirectional Language Modelling: Expressive Power and Representational Conciseness

Georgi Shopov, Stefan GerdjikovEMNLP 2024 · DOI

Introduction

Large Language Models (LLMs) typically fall into two categories: unidirectional autoregressive models (e.g., GPT) and bidirectional masked models (e.g., BERT). While unidirectional models provide a mathematically consistent joint distribution $P(x)$ via the chain rule, they lack the full contextual awareness of bidirectional models. Conversely, bidirectional models capture rich context but often lack mathematical consistency—the conditional probabilities they predict may not correspond to any valid joint distribution. This paper introduces "latent language models," a framework that bridges this gap by combining bidirectional context with the formal consistency of autoregressive generation.

Theoretical Framework: Latent Language Models

The authors propose a two-part architecture based on bisequential decompositions from automata theory. This structure ensures that "future" information is incorporated into the generation process without violating the chain rule.

  1. The Co-sequential Encoder: Scans the input from right-to-left, generating a sequence of latent features that encapsulate the "future" context.
  2. The Sequential Generator: Scans from left-to-right, using both the original input and the latent features from the encoder to predict the next token.

By defining these as Rational Language Models, the authors can represent them using finite-state transducers. This formalization allows for exact likelihood scoring and efficient ancestral sampling.

Figure 1: A simple probabilistic finite-state automaton representing a basic rational language model.

Expressive Power and Conciseness

A core contribution of this work is the proof that latent language models are strictly more expressive than standard unidirectional rational models. They can efficiently represent complex dependencies, such as those found in the $a^n b^n$ language, which are difficult for finite-state autoregressive models to track.

Furthermore, the paper demonstrates exponential conciseness. For certain classes of languages, a unidirectional model requires a number of states exponential in the sequence length ($ \Omega(|\Sigma|^n) $), whereas a bidirectional latent model can represent the same distribution with only linear complexity ($ O(n|\Sigma|) $). This is illustrated through mixture models where latent states allow for a more compact representation of dependencies.

Figure 2: An illustration of a mixture model where latent states (q0, q1) allow for more efficient representation of dependencies.

Key Results

  • Consistency Guarantee: The latent framework ensures that the model always produces a valid probability distribution, solving the "inconsistency trap" prevalent in models like BERT.
  • Decomposition Theorems: Theorems 4.1 and 5.1 prove that any rational language model can be decomposed into this bidirectional latent structure.
  • The Lookahead Index: The authors introduce a metric to quantify the amount of "future" information required for a model to remain rational, linking encoder complexity to the syntactic properties of the language.

Conclusion and Future Directions

While this research provides a rigorous mathematical foundation for consistent bidirectional modelling, several challenges remain. The current work is theoretical, focusing on formal languages rather than large-scale natural language datasets. Future work must address the learnability of these models—specifically, how to parameterize the latent encoders (perhaps using Transformers or RNNs) to handle the complexities of human language while maintaining the theoretical guarantees of the latent framework.

Summary updated

BibTeX
@inproceedings{shopov2024consistent,
  author = {Georgi Shopov and Stefan Gerdjikov},
  title = {{Consistent Bidirectional Language Modelling: Expressive Power and Representational Conciseness}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.328},
}
Figure