Arrows of Time for Large Language Models

Vassilis Papadopoulos, Jérémie Wenger, Clément Hongler

Introduction

In 1951, Claude Shannon made a profound observation about human language: while forward and backward text sequences possess the exact same mathematical entropy, humans find predicting text backward "subjectively much more difficult." Today, modern Large Language Models (LLMs) are almost exclusively autoregressive, meaning they learn by predicting the next token in a forward sequence.

This raises a fundamental question: Is there a structural asymmetry in language that makes the forward direction inherently "better" for machine learning? From a strict information-theoretic perspective, a sequence of text should be equally predictable whether read forward or backward. The bits of information are identical. Yet, this paper reveals a consistent Arrow of Time (AoT) in LLMs, bridging the gap between abstract information theory and the computational reality of machine learning.

The Arrow of Time Phenomenon

To investigate this asymmetry, researchers employed a rigorous "all else being equal" experimental setup. They trained identical model architectures on the exact same data, with one critical difference: the order of the tokens. The Forward (FW) models received standard text, while the Backward (BW) models received sequences in reverse (e.g., "dog the fed boy the").

Training loss over time for forward and backward models in English and French.

Across multiple languages and architectures, models trained to predict the next token consistently achieved lower perplexity (better performance) than those trained to predict the previous token. This suggests that natural language is fundamentally easier to compute in its natural temporal order.

Methodology and Controls

To ensure this phenomenon wasn't merely a quirk of a specific architecture or tool, the researchers implemented several strict controls:

  • Architectural Variety: They tested GPT-2 (ranging from 5M to 1.6B parameters), GRUs, and LSTMs. The asymmetry persisted across all of them.
  • Artifact Control: They ruled out tokenization bias by re-training Byte Pair Encoding (BPE) tokenizers on reversed character-level data. This confirmed the asymmetry is intrinsic to the language itself, not an artifact of how the text was pre-processed.
  • Synthetic Probing: To mathematically isolate why one direction might be harder to learn, they created "toy" languages based on prime factoring and linear algebra.

Key Insights: Computation vs. Information

The core revelation of the paper is that language acts somewhat like a one-way function. In the synthetic prime factoring experiment, the FW model easily learned to multiply ($p, q \to pq$), but the BW model struggled immensely to factor ($pq \to p, q$). Even though the underlying information is identical in both directions, the computational complexity required to extract that information is vastly different.

The authors hypothesize that human language evolved under selection pressure to be "sparse" and easily computable for forward-predicting agents (our brains). In their linear toy model, they demonstrated that the inverse of a sparse operation is typically dense, making it significantly harder for a neural network to learn.

Key Results and Scale Amplification

The "Forward Arrow of Time" appeared in every natural language tested from the CC-100 dataset, including English, French, German, Turkish, and even right-to-left languages like Hebrew and Arabic. Interestingly, the magnitude of the effect varies by language and amplifies with scale.

Performance gap between forward and backward models across different GPT model sizes.

As models grow larger and context windows increase, the performance gap between FW and BW models widens. This suggests the Arrow of Time is deeply rooted in long-range dependencies—the complex ways words relate to each other over long paragraphs.

Specific findings include:

  • Language Variations: In GPT-Medium models (405M parameters), the BW model for English had a 0.76% higher cross-entropy loss compared to its FW counterpart. For French, the gap was much wider at 2.65%.
  • Model Size Inefficiency: Larger BW models often perform worse than smaller FW models. A 405M parameter BW model can be outperformed by a significantly smaller FW model, highlighting the sheer computational inefficiency of learning language backward.

Limitations and Future Work

While the paper establishes a clear Arrow of Time in LLMs, several mysteries remain. It is not yet fully understood why certain languages (like French) exhibit a stronger directional asymmetry than others. Furthermore, it remains an open question whether a BW model could eventually catch up to a FW model if trained for an infinitely long time, approaching the theoretical information limit.

The authors suggest exciting avenues for future research, including exploring whether this Arrow of Time exists in other sequential modalities like DNA, computer code, or video. Finally, they hint at a potential deep link between the learnability of data and the physical arrow of time in thermodynamics, opening the door for fascinating interdisciplinary studies.

Summary updated

Abstract

We study the probabilistic modeling performed by Autoregressive Large Language Models (LLMs) through the angle of time directionality, addressing a question first raised in (Shannon, 1951). For large enough models, we empirically find a time asymmetry in their ability to learn natural language: a difference in the average log-perplexity when trying to predict the next token versus when trying to predict the previous one. This difference is at the same time subtle and very consistent across various modalities (language, model size, training time, ...). Theoretically, this is surprising: from an information-theoretic point of view, there should be no such difference. We provide a theoretical framework to explain how such an asymmetry can appear from sparsity and computational complexity considerations, and outline a number of perspectives opened by our results.
BibTeX
@inproceedings{papadopoulos2024arrows,
  author = {Vassilis Papadopoulos and Jérémie Wenger and Clément Hongler},
  title = {{Arrows of Time for Large Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=UpSe7ag34v},
}
Figure