Introduction
Transformer architectures have revolutionized natural language processing and computer vision, yet their internal mechanisms remain largely opaque. A primary challenge in interpreting multi-head attention is the non-linear softmax activation, which often obscures the underlying signal by "squashing" coordinates. This paper proposes a mathematical framework to organize attention heads into hierarchical structures using tensor organization and paradifferential calculus. By revealing the intrinsic and extrinsic geometry of attention, we demonstrate that softmax invariance is achievable under specific regularity conditions and that this organization exposes significant network sparsity.
Methodology: Tensor Organization and Paradifferential Calculus
The research addresses the "black box" nature of Transformers through two sophisticated mathematical toolsets.
The Questionnaire Method
To reveal hidden patterns, the authors employ the "Questionnaire Method," an algorithm that organizes 3D tensors (Query $\times$ Key $\times$ Head) into hierarchical partition trees. Instead of treating attention heads as a flat list, this method builds a coherent geometry for the Query, Key, and Head axes. By grouping similar indices, the resulting tensor becomes "regular" and "smooth," facilitating the application of signal processing techniques.
Softmax Invariance via Paradifferential Calculus
A core theoretical contribution is the proof that softmax activation is not necessarily a destructive operator. Using paradifferential calculus and paraproduct decompositions, the authors analyze the mixed $\alpha$-Hölder regularity of attention heads. They demonstrate that if a head is organized to meet specific regularity conditions, the structural features of the head are preserved through the softmax operation.
Empirical Analysis and Results
The framework was evaluated on two distinct architectures: a Transformer-XL (TXL) language model trained on WikiText-103 and a Vision Transformer (ViT B16) fine-tuned on CIFAR-10.
Qualitative Regularity and Manifold Structure
Visualizations confirm that organized attention heads are significantly smoother than their raw counterparts. Furthermore, applying Diffusion Maps to these organized heads reveals a distinct manifold structure. In the TXL model, heads exhibit a continuous relationship, while in the ViT model, they tend to cluster with specific outliers, suggesting different processing strategies for language versus vision.
Sparsity and Pruning Potential
By projecting the organized network onto a 3D Tri-Haar basis, the authors measured the $l_1$ entropy of expansion coefficients as a proxy for information processing. Key findings include:
- Consistent Importance: Heads identified as "Top 3" or "Bottom 3" by entropy remained consistent across all test batches. This suggests that head importance is an intrinsic property of the architecture and training, rather than a function of specific input data.
- Model Complexity: The TXL model exhibited significantly higher $l_1$ entropy (approx. 1200-2400) compared to the ViT model (approx. 2-9), indicating that the language model handles more complex, "high-frequency" transitions.
Conclusion and Future Work
This work provides a principled path toward model interpretability and automated pruning. By identifying heads with low $l_1$ entropy, researchers can remove structurally redundant components to create leaner models without sacrificing accuracy. While the current implementation faces constraints—such as requiring matrix dimensions to be powers of 2 and sensitivity to initial parameters—the proof of softmax invariance and the discovery of attention manifolds mark a significant step forward in understanding the geometry of Transformers.
Summary updated