Abstract
We study the probabilistic modeling performed by Autoregressive Large Language Models (LLMs) through the angle of time directionality, addressing a question first raised in (Shannon, 1951). For large enough models, we empirically find a time asymmetry in their ability to learn natural language: a difference in the average log-perplexity when trying to predict the next token versus when trying to predict the previous one. This difference is at the same time subtle and very consistent across various modalities (language, model size, training time, ...). Theoretically, this is surprising: from an information-theoretic point of view, there should be no such difference. We provide a theoretical framework to explain how such an asymmetry can appear from sparsity and computational complexity considerations, and outline a number of perspectives opened by our results.
Introduction: The Shannon Paradox
In his foundational 1951 work, Claude Shannon observed that from a purely information-theoretic perspective, the statistical structure of a language like English should be time-symmetric. Mathematically, the joint probability of a sequence of words $P(X_1, \dots, X_n)$ is identical to the probability of its reverse. This implies that the entropy—the fundamental limit of compressibility—is the same whether a sequence is read forward or backward.
However, human intuition suggests a different story. We find it significantly easier to predict the "next" word in a sentence than to guess the "previous" one. This research investigates whether this subjective human experience is reflected in the performance of Large Language Models (LLMs). If information theory dictates symmetry, why do we observe an "Arrow of Time" in how machines learn to process language?
Methodology: Comparing Forward and Backward Models
To investigate this phenomenon, the researchers conducted a systematic comparison between Forward (FW) and Backward (BW) models. The experimental setup involved training identical architectures—same hyperparameters, same data, and same training duration—with the only difference being the direction of the token sequence.
Architectures and Data
The study primarily utilized decoder-only Transformers (GPT-2 style), ranging from 5 million to 1.6 billion parameters. To ensure the results were not an artifact of the Transformer architecture, the researchers also tested recurrent models like GRUs and LSTMs.
The models were trained on the CC-100 dataset, which includes a diverse array of natural languages such as English, French, German, Vietnamese, and Greek. To control for cultural or script-based biases, the researchers specifically included right-to-left scripts (Hebrew and Arabic) and languages with varying grammatical structures (such as the verb-initial Tagalog).
Empirical Findings: The Universal Forward Bias
The results across all experiments were remarkably consistent: Forward models consistently outperform Backward models. In every natural language tested, the FW model achieved lower perplexity than its BW counterpart.
- Language Variance: In English (using GPT2-Medium), the FW loss was 2.88 compared to 2.902 for the BW model (a +0.76% difference). In French, the gap was even more pronounced at +2.65%.
- Scaling Effects: The performance gap is not static; it increases with both model size and context window length. While very small models or extremely short context windows (e.g., 16 tokens) show negligible asymmetry, the "Arrow of Time" becomes starkly visible as models gain the capacity to learn complex, long-range dependencies.
- Script Independence: The effect persisted regardless of whether the language is written left-to-right or right-to-left, proving that the asymmetry is a property of the data's underlying logical structure rather than its visual representation.
Theoretical Framework: Sparsity and Computational Complexity
If entropy is symmetric, why is one direction harder to learn? The authors propose that the answer lies in the distinction between information content and computational learnability. Even if two distributions contain the same information, the "computational path" required to extract that information may be vastly different.
The Sparsity Argument
A key insight from the paper is that inverting a sparse operation often results in a dense one. If we view natural language generation as a "sparse" process in the forward direction (where a few latent variables determine the next token), the backward process is likely to be "denser" and more complex for a neural network to approximate.
Types of Hardness
The researchers categorize the difficulty into two types:
- Type 1 (Representational): The model may lack the capacity to represent the inverse function. For example, while multiplication is easy to represent, its inverse (factoring) is computationally difficult.
- Type 2 (Optimization): The model might have the capacity to represent the function, but the training process (gradient descent) fails to find the solution because the backward direction lacks the structural cues present in the forward direction.
Synthetic Proof: Prime Factoring
To validate this, the researchers used a synthetic task involving prime numbers. They found that a model trained to predict the product of two primes (multiplication) performed near-optimally, while a model trained to predict the primes from the product (factoring) failed significantly. This demonstrates that even when the information is identical, the direction of the task dictates its learnability.
Discussion and Future Directions
The discovery of a consistent Arrow of Time in LLMs opens several new avenues for research. It suggests that the presence of such an asymmetry might be a signature of "intelligent" or biological data generation processes.
Future work may explore whether this phenomenon extends to other modalities, such as DNA sequences, computer code, or video. Furthermore, the paper hints at a potential, though currently speculative, link between the computational Arrow of Time in machine learning and the physical Arrow of Time in thermodynamics. While it remains to be seen if BW models could eventually close the gap with infinite training time, these results suggest that for the foreseeable future, language remains a fundamentally forward-leaning process.
Summary updated