Two-Clustering Regime of Token Dynamics in Causal Attention

Trinh T. Nguyen, Duy-Tung Pham, Hoang-Son Do, Tan Minh Nguyen, Thieu N. VoarXiv 2026 · arXiv

TL;DR

This paper provides a rigorous mathematical proof for the "two-clustering" behavior in causal attention, demonstrating that tokens in autoregressive models naturally collapse into two opposing clusters aligned with the value matrix's dominant eigenvector. By introducing a physics-inspired "instability within the flow" framework, the authors explain how high-dimensional token trajectories simplify into a predictable two-dimensional structure, even when model parameters change over time.

What Problem Does This Paper Address?

While Transformers have revolutionized AI, our theoretical understanding of their internal "token dynamics"—how representations evolve across layers—is still catching up to their empirical success. Previous research into "full attention" (where every token sees every other token) often found that tokens converge toward a single cluster, a phenomenon linked to "oversmoothing" where models lose expressive power.

However, modern generative models (like GPT-4 or Llama) use causal attention, where tokens can only attend to their predecessors. This masking breaks the symmetry of full attention, making it much harder to analyze. A recent conjecture suggested that causal attention leads to a "two-clustering" regime, but a formal proof was missing, particularly for the realistic scenario where model weights might vary across layers (time-dependent parameters). This paper aims to bridge that gap, explaining why tokens don't just collapse into one point, but rather split into two.

Methods

The authors treat the Transformer as a dynamical system, viewing individual tokens as interacting particles moving through a high-dimensional embedding space. Their approach combines several novel techniques:

  • Instability within the Flow: This is the paper's central analytical contribution. Inspired by stability theory in physics, the authors visualize token evolution as a small ball rolling on a "hill" (the attention landscape). Crucially, in deep models, this hill itself moves over time. They prove that a stationary hill forces tokens toward two extremal states, while a rapidly moving one can trap them in "interior" clusters.
  • Hierarchical Suppression: They introduce the "error-to-traverse ratio" to show how the system progressively eliminates non-dominant directions, forcing the high-dimensional trajectory to collapse onto a lower-dimensional manifold.
  • Dimensional Reduction: They prove that despite the complexity of the embedding space, token trajectories eventually settle into a two-dimensional coordinate plane governed by a few dominant directions.

Datasets

As a primarily theoretical work, the authors use a mix of synthetic simulations and established benchmarks to validate their proofs:

  • Synthetic Simulations: Token trajectories were tested in 2D, 3D, 32D, and 128D spaces to visualize clustering behavior.
  • WikiSQL and MultiNLI: The authors performed fine-tuning experiments on GPT-3 (175B) using LoRA. They compared the performance of fine-tuning different weight matrices ($Q, K, V$) to see if the theoretical importance of the Value matrix ($V$) held up in practice.

Key Insights and Ideas

  • The $V$ Matrix is the "Boss": The two clusters align perfectly with the eigenvectors associated with the largest eigenvalue ($\lambda_{max}$) of the value transformation matrix. If $\lambda_{max} \le 0$, the two-cluster regime generally fails.
  • The "Déjà Vu" Effect: The authors identify a recurrence phenomenon where, even as tokens stabilize, the final tokens are repeatedly forced to re-align with the dominant direction. This suggests that "past" dominant information is never truly forgotten in the infinite-depth limit.
  • Rigidity in High Dimensions: One of the most surprising findings is that token dynamics are "rigid." Even in a 1024-dimensional space, the long-term behavior is effectively two-dimensional. This provides a theoretical justification for why low-rank adaptation (LoRA) is so effective.

Key Results

The paper successfully resolves the two-clustering conjecture for causal attention. Key findings include:

  • Global Convergence: They prove that for almost every initial condition, tokens converge to the two clusters $\{\xi_{max}, -\xi_{max}\}$.
  • GQA Validation: The theory holds up in Group Query Attention (GQA), a popular mechanism in modern LLMs like Llama 3. Figure 1 (below) shows tokens organizing into two opposing clusters on a circle after processing.
  • Fine-tuning Superiority: In their GPT-3 experiments, fine-tuning combinations that included the $V$ matrix consistently outperformed those that only tuned $Q$ or $K$. For example, on WikiSQL, tuning $Q$ and $V$ reached 73.7% accuracy, compared to only 70.4% for $Q$ alone. This confirms that the $V$ matrix plays a critical role in shaping the attractors of token dynamics.
Comparison of theoretical analyses for causal-attention dynamics.

Limitations and Future Work

  • Structural Assumptions: The proof for time-dependent parameters relies on a specific "block structure" assumption for the Query-Key matrix. While this covers common "weight-tying" scenarios, it may not perfectly describe every pre-trained Transformer in the wild.
  • Symmetric $V$ Matrix: Much of the analysis assumes the value matrix $V$ is symmetric. Extending these proofs to non-symmetric matrices—which are common in practice—remains an open challenge.
  • Multi-Head Interactions: The current work focuses primarily on single-head dynamics. Future research could explore how multiple attention heads interact to create even more complex clustering regimes.

Summary updated

Abstract

We partially resolve Conjecture 1 of [Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024)] and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue $λ_{\max}$ of the value transformation. We prove that this behavior holds when $λ_{\max}>0$ and is generally false when $λ_{\max}\le0$. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension reduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework--instability within the flow--inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.
BibTeX
@misc{nguyen2026two,
  author = {Trinh T. Nguyen and Duy-Tung Pham and Hoang-Son Do and Tan Minh Nguyen and Thieu N. Vo},
  title = {{Two-Clustering Regime of Token Dynamics in Causal Attention}},
  year = {2026},
  eprint = {2611.18088},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18088},
}
Figure