Intrinsic and Extrinsic Organized Attention: Softmax Invariance and Network Sparsity

Oluwadamilola Fasina, Ruben V. C. Pohle, Pei-Chun Su, Ronald R. CoifmanarXiv 2025 · DOI · arXiv

Introduction

Transformer architectures have revolutionized natural language processing and computer vision, yet their internal mechanisms remain largely opaque. A primary challenge in interpreting multi-head attention is the non-linear softmax activation, which often obscures the underlying signal by "squashing" coordinates. This paper proposes a mathematical framework to organize attention heads into hierarchical structures using tensor organization and paradifferential calculus. By revealing the intrinsic and extrinsic geometry of attention, we demonstrate that softmax invariance is achievable under specific regularity conditions and that this organization exposes significant network sparsity.

Methodology: Tensor Organization and Paradifferential Calculus

The research addresses the "black box" nature of Transformers through two sophisticated mathematical toolsets.

The Questionnaire Method

To reveal hidden patterns, the authors employ the "Questionnaire Method," an algorithm that organizes 3D tensors (Query $\times$ Key $\times$ Head) into hierarchical partition trees. Instead of treating attention heads as a flat list, this method builds a coherent geometry for the Query, Key, and Head axes. By grouping similar indices, the resulting tensor becomes "regular" and "smooth," facilitating the application of signal processing techniques.

Figure 2: Comparison of an original attention head in Transformer-XL (a) and its organized version (b) using the Questionnaire Method, revealing a smoother, more structured geometry.

Softmax Invariance via Paradifferential Calculus

A core theoretical contribution is the proof that softmax activation is not necessarily a destructive operator. Using paradifferential calculus and paraproduct decompositions, the authors analyze the mixed $\alpha$-Hölder regularity of attention heads. They demonstrate that if a head is organized to meet specific regularity conditions, the structural features of the head are preserved through the softmax operation.

Figure 6: Visualization of the attention head approximation (a) versus the actual softmax output (b) for Transformer-XL, demonstrating structural invariance.

Empirical Analysis and Results

The framework was evaluated on two distinct architectures: a Transformer-XL (TXL) language model trained on WikiText-103 and a Vision Transformer (ViT B16) fine-tuned on CIFAR-10.

Qualitative Regularity and Manifold Structure

Visualizations confirm that organized attention heads are significantly smoother than their raw counterparts. Furthermore, applying Diffusion Maps to these organized heads reveals a distinct manifold structure. In the TXL model, heads exhibit a continuous relationship, while in the ViT model, they tend to cluster with specific outliers, suggesting different processing strategies for language versus vision.

Sparsity and Pruning Potential

By projecting the organized network onto a 3D Tri-Haar basis, the authors measured the $l_1$ entropy of expansion coefficients as a proxy for information processing. Key findings include:

  • Consistent Importance: Heads identified as "Top 3" or "Bottom 3" by entropy remained consistent across all test batches. This suggests that head importance is an intrinsic property of the architecture and training, rather than a function of specific input data.
  • Model Complexity: The TXL model exhibited significantly higher $l_1$ entropy (approx. 1200-2400) compared to the ViT model (approx. 2-9), indicating that the language model handles more complex, "high-frequency" transitions.

Conclusion and Future Work

This work provides a principled path toward model interpretability and automated pruning. By identifying heads with low $l_1$ entropy, researchers can remove structurally redundant components to create leaner models without sacrificing accuracy. While the current implementation faces constraints—such as requiring matrix dimensions to be powers of 2 and sensitivity to initial parameters—the proof of softmax invariance and the discovery of attention manifolds mark a significant step forward in understanding the geometry of Transformers.

Summary updated

BibTeX
@misc{fasina2025intrinsic,
  author = {Oluwadamilola Fasina and Ruben V. C. Pohle and Pei-Chun Su and Ronald R. Coifman},
  title = {{Intrinsic and Extrinsic Organized Attention: Softmax Invariance and Network Sparsity}},
  year = {2025},
  eprint = {2506.15541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.15541},
}
Figure