Equilibrium measures for anisotropic exponential interaction energies with applications to transformers

Oleksii Kachaiev, Martin Burger, Samira Kabri, Yury Korolev, Tim RoitharXiv 2026 · arXiv

This paper, recently published on arXiv, provides a rigorous mathematical foundation for understanding the equilibrium states of self-attention mechanisms in transformer architectures. By bridging the gap between particle physics, stochastic analysis, and machine learning, the authors prove long-standing conjectures about how these models "concentrate" information and provide new tools for simulating their behavior.

TL;DR

The authors prove that the equilibrium states of anisotropic self-attention models in transformers are smooth, strictly positive probability distributions on a sphere. By linking these states to "killed" anisotropic Brownian motion, they develop a novel "Walk on Spheres" sampling algorithm and a numerical approximation method using Mathieu functions.

What Problem Does This Paper Address?

Transformers rely on self-attention, a mechanism where every token in a sequence interacts with every other token to determine their relative importance. In the "mean-field" limit—where we imagine an infinite number of tokens—this system behaves like a collection of interacting particles on a sphere.

The "energy" of this system is determined by an interaction kernel $K_D(x, y) = e^{x^T D y}$. While the isotropic case (where the interaction matrix $D$ is just a multiple of the identity) is well-understood, real-world transformers use anisotropic interactions (where $D$ has different weights for different directions).

Until now, it was conjectured but not proven that for a positive definite matrix $D$, the system would always settle into a state that covers the entire sphere (full support) rather than collapsing into a few points. This paper sets out to prove this "full-support conjecture" and provide practical ways to calculate these equilibrium states.

Methods

The paper employs a sophisticated blend of stochastic analysis and partial differential equations (PDEs):

  • Probabilistic Representation: The authors show that the energy minimizer is equivalent to the exit distribution of an anisotropic Brownian motion. Imagine a random walker starting at the center of a ball. As it moves, there is a constant chance it will be "killed" (removed). The distribution of where the surviving walkers hit the boundary of the ball corresponds exactly to the energy minimizer.
  • Feynman-Kac Formula: They use this classic tool to link the stochastic process to a specific elliptic PDE. This allows them to use "elliptic regularity theory" to prove that the resulting density is smooth and never hits zero.
  • Walk on Spheres (WoS) Algorithm: Leveraging the stochastic viewpoint, they adapt the WoS algorithm—traditionally used for solving the Dirichlet problem—to sample from the equilibrium measure without needing to compute the density directly.
  • Mathieu Series Expansion: For the 2D case ($d=2$), they use Mathieu functions (which solve the wave equation in elliptical coordinates) to create a highly accurate numerical approximation of the density.

Datasets

As a theoretical mathematics and physics paper, it does not use standard machine learning benchmarks (like ImageNet or GLUE). Instead, the authors evaluate their methods using:

  • Synthetic Interaction Matrices: Randomly generated symmetric positive definite matrices $D$ in various dimensions (up to $d=20$).
  • Numerical Simulations: Comparisons between their proposed "Walk on Spheres" algorithm and traditional projected gradient descent.

Key Insights and Ideas

  • The "Killed" Path Connection: The most striking insight is that a complex variational problem (minimizing energy) can be perfectly described by a simple random walk with a "killing" rate. This transforms a difficult optimization problem into a sampling problem.
  • Resolution of the Full-Support Conjecture: They rigorously prove that if $D$ is positive definite, the equilibrium measure is "absolutely continuous" with respect to the surface measure. In plain English: the transformer's attention doesn't "blind" itself to any part of the feature space; it maintains a global, albeit weighted, view.
  • Uniqueness of the Minimizer: They show that while many "stationary" states might exist (where particles stop moving), only the global minimizer is spread out across the sphere. Every other stationary state is "singular" (concentrated on a tiny, zero-volume set).

Key Results

  • Smoothness and Positivity: Theorem 7 proves the existence of a smooth, strictly positive density $w_D$ on the sphere.
  • Sampling Efficiency: The "Walk on Spheres" algorithm was shown to converge to the true minimizer as the tolerance parameter $\epsilon$ goes to zero.
  • Numerical Convergence: In experiments with $d=20$, the sampling energy difference compared to gradient descent decreased at a rate of $1/N$, confirming the theoretical robustness of the stochastic approach.

Limitations and Future Work

  • The Eigenvector Conjecture: The authors hypothesize that the density's minimum is always found at the unit vectors corresponding to the largest eigenvalue of $D$. While their simulations support this, a rigorous proof remains elusive.
  • Non-Symmetric Matrices: Real-world transformers use $W_Q^T W_K$ as an interaction matrix, which is not necessarily symmetric or positive definite. Extending these proofs to the non-symmetric case is a major open challenge.
  • High-Dimensional Approximations: The Mathieu function approach is currently limited to two dimensions. Developing similar spectral approximations for higher dimensions would be computationally valuable.

Summary updated

Abstract

We study the minimization of anisotropic exponential interaction energies over probability measures on the sphere, motivated by mean-field models of self-attention in transformer architectures. For every symmetric positive definite interaction matrix, we relate the minimizer to a corresponding anisotropic Brownian motion and use the Feynman--Kac formula and elliptic regularity theory to show that it admits a smooth, strictly positive density on the sphere. We further show that it is the only stationary measure with full support for the dynamics considered. We use the stochastic viewpoint to derive an algorithm for sampling from the minimizer. In dimension two, we also propose a numerical scheme for approximating its density based on a Mathieu series expansion.
BibTeX
@misc{kachaiev2026equilibrium,
  author = {Oleksii Kachaiev and Martin Burger and Samira Kabri and Yury Korolev and Tim Roith},
  title = {{Equilibrium measures for anisotropic exponential interaction energies with applications to transformers}},
  year = {2026},
  eprint = {2611.19518},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19518},
}
Figure