This paper, authored by Y.-H. Taguchi, provides a rigorous deconstruction of recent attempts to apply stochastic thermodynamics to Large Language Models (LLMs). It argues that the "entropy production" signal previously used to identify causal ordering in text is largely an artifact of the model's training and the specific order in which text was generated, rather than a deep property of the text's causal structure.
TL;DR
The paper demonstrates that the "thermodynamic" interpretation of LLM scores is flawed because it lacks a proper reference point (a "zero"). By introducing a permutation-based null hypothesis, the author reveals that the vast majority of the observed signal comes from the model's bias toward the original generation order ("authored component"), with "causal order" contributing only a small fraction (8%–15%) of the effect.
What Problem Does This Paper Address?
Recent research (notably by Sagawa [1]) proposed that comparing the probability of a text $x$ with its reversal $\tilde{x}$ yields a measure of "entropy production" ($\Delta$) that can distinguish causally ordered prose from scrambled sentences.
Taguchi argues this analogy is physically and statistically shaky. Physically, a single fixed string is not a "trajectory" in an ensemble, so fluctuation theorems—which provide the mathematical "zero" for entropy production—don't apply. Statistically, without a principled way to define what a "zero" score looks like for a fixed text, we can't know if a high score indicates "causality" or just a mundane training bias.
Methods
The paper introduces two key methodological innovations to fix the "zero" problem:
- The Permutation Set & Midrank: Instead of just comparing a text to its single reversal, the author considers all $n!$ possible permutations of a text's sentence blocks. Because the reversal operation is an involution on this set, the distribution of scores is exactly centered at zero. The "midrank" (where the original text falls in this distribution) provides a unitless, bounded measure. A score of 0.50 represents the perfect "null" where the model is blind to sentence order.
- Assembly-Order Comparison ($\sigma$): Using Masked Language Models (MLMs) and the "any-order chain rule," the author compares scoring the same text in two different assembly directions (left-to-right vs. right-to-left). Mathematically, if a model is perfectly consistent, this difference ($\sigma$) should be identically zero for every text.
- The AR Ladder: The author evaluates these measures across an "autoregressive ladder" of five models, ranging from GPT-2 (124M parameters) to Qwen-3 (4B parameters), to see how these effects scale.
Datasets
The study re-analyzes the 14 corpora used in the original Sagawa paper, which consist of pairs of "causal" and "non-causal" (arbitrarily ordered) texts generated by various LLMs (including GPT-4, Gemini, and Claude). The author also uses a toy "exactly solvable system" (a pairwise Markov random field) to prove theoretical identities.
Key Insights and Ideas
- The "Authored" Bias: LLMs are trained to predict the next token from left to right. Naturally, they score the original "authored" order much higher than any scramble. This "authored component" is the dominant signal, not the "causal" nature of the sentences.
- Incompatibility as a Metric: For masked models, a non-zero $\sigma$ doesn't measure text properties; it measures the "evaluator's incompatibility"—the degree to which the model fails to define a consistent joint probability distribution.
- The Tail Tells a Different Story: While the median separation between causal and non-causal text doesn't move much as models get bigger, the "tail" of the distribution does. Larger models are significantly more likely to rank a causal arrangement as the absolute #1 (strict maximum) compared to non-causal arrangements.
Key Results
- Causality is a Small Slice: Only 8% to 15% of the distance from the null is attributable to causal order. The rest is simply the "authored component"—the fact that the text was produced in that order.
- Token vs. Block Reversal: The paper confirms that token-level reversal is dominated by "syntactic artifacts" (the text becomes ungrammatical gibberish), which is why block-level (sentence) reversal is the only meaningful comparison for high-level structure.
- Scaling Effects: Across the AR ladder, the median separation between label groups remains small (2–4 grid steps), but in the tail, the separation roughly doubles as models grow from 124M to 4B parameters.
- MLM Results: For masked models, the authored arrangement's midrank is highly dependent on the model itself (moving 14 grid steps between BERT and LLaDA), confirming it reports on the evaluator rather than the text.
Limitations and Future Work
- No Human Baseline: All texts in the study were generated by LLMs. The "authored component" might behave differently for human-written text, which is the most critical missing control.
- Label Validity: The "causal" and "non-causal" labels are based on generation prompts, not formal causal criteria (like interventions or counterfactuals). The author notes that "causal" is currently just the name of a prompt, not a measured property.
- Model Confounding: The AR ladder mixes model size with different training recipes and architectures, making it hard to isolate parameter count as the sole driver of trends.
Summary updated