Distinguishing between cause and effect is one of the most fundamental challenges in science. If we observe two correlated variables, $X$ and $Y$, how do we know if $X \to Y$ or $Y \to X$ without performing a randomized experiment? Traditional statistical methods often struggle here because the joint distribution $P(X, Y)$ can often be factored both ways.
A growing body of research suggests that the answer lies in entropy. The core intuition is a form of Occam’s Razor: nature’s true causal mechanisms are "simple" or "efficient." When we model a relationship in the correct causal direction, the "hidden information" (the noise) required to explain the effect is minimal. If we try to force the model in the wrong direction, the required noise becomes significantly more complex and high-entropy.
The Entropy Gap
The foundational idea, introduced in Entropic Causal Inference by Alex Dimakis, Murat Kocaoglu, and colleagues, is that a causal model $Y = f(X, E)$ is identifiable if the exogenous noise $E$ has low entropy.
In a discrete system, if $X$ causes $Y$, then $Y$ is a function of $X$ and some independent noise $E$. The researchers discovered an "entropy gap": the entropy of the noise needed to explain $Y$ from $X$ is typically much lower than the entropy of the noise needed to explain $X$ from $Y$. This is because the reverse mapping must not only account for the original noise but also "undo" the transformation $f$, which usually introduces additional uncertainty and complexity.
Innovation and Parsimony
This principle extends to time-series and stochastic processes. Amichai Painsky, Saharon Rosset, and Meir Feder frame this as finding the Innovation Representation of Stochastic Processes. They argue that the true causal direction is the one that allows the system to be represented by a sequence of independent "innovations" with the lowest possible entropy. Essentially, the most parsimonious explanation of exogenous influence aligns with the true causal structure.
Beyond the Gaussian Trap
In continuous variables, many researchers traditionally relied on additive noise models (ANM) and least-squares regression. However, Ruichu Cai and Jie Qiao point out in On the Role of Entropy-based Loss for Learning Causal Structures with Continuous Optimization that least-squares implicitly assumes Gaussian noise.
If the real-world noise is non-Gaussian (e.g., Uniform or Gumbel), variance-based methods can fail. By using entropy-based loss functions, we can identify causal directions across diverse noise distributions. Entropy acts as a more robust, distribution-agnostic indicator of independence: in the correct direction, the residuals (noise) are independent of the cause, a state that minimizes the entropy of the system.
Information Geometry and Deterministic Relations
Even when the relationship is entirely deterministic and noise-free, an asymmetry remains. Dominik Janzing, Povilas Daniušis, and colleagues developed Information Geometric Causal Inference (IGCI) to exploit this.
The intuition here is that if $X \to Y$ via a function $f$, the density of $X$ and the slope of $f$ are usually independent. However, in the reverse direction, the density of $Y$ will appear correlated with the slope of $f^{-1}$. By calculating the entropy of the variables relative to a reference manifold (like a uniform distribution), IGCI can detect which variable's distribution is the "independent source" and which is the "transformed effect."
Scaling to Complex Networks
While early work focused on pairs of variables, recent advances have scaled these entropic principles to multivariate graphs. In Entropic Causal Inference: Graph Identifiability, Spencer Compton, Kristjan Greenewald, and Murat Kocaoglu proved that the true structure of a categorical causal graph can be uniquely identified by looking for the simplest (lowest entropy) explanation.
They introduced "peeling algorithms" that identify source nodes by checking which variables have conditional entropy signatures that match their intrinsic noise entropy. As Devansh Arpit and colleagues describe in Causal Layering via Conditional Entropy, this allows researchers to recover the topological ordering (the "layers") of a causal system by monitoring how entropy changes as we condition on different sets of variables.
By shifting the focus from simple correlation to the information-theoretic cost of a mechanism, entropic causal discovery provides a powerful, mathematically rigorous way to uncover the hidden "arrows" of causality in complex data.