Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures

Charles O'NeillarXiv 2025 · DOI · arXiv

Introduction

Despite the ubiquity of Transformer architectures in modern artificial intelligence, a unified theoretical understanding of their internal mechanics remains elusive. Current research often views Transformers through disparate lenses: geometric perspectives focus on symmetry and equivariance, mechanistic interpretability focuses on "circuits," and engineering perspectives focus on the empirical success of layer stacking. This paper introduces a rigorous mathematical "skeleton" for Transformers using category theory, framing self-attention as a parametric endofunctor and the stacking of layers as a free monad.

A Unified Categorical Pipeline

The core contribution of this work is the translation of neural network components into the language of the 2-category $Para(Vect)$. In this framework, the linear components of self-attention—the Query ($Q$), Key ($K$), and Value ($V$) maps—are modeled as parametric 1-morphisms. These morphisms take both model parameters (weights) and data (input tokens) as inputs, inducing an endofunctor on the base category of vector spaces ($Vect$).

A categorical overview of the Transformer architecture, mapping positional encodings, self-attention, and stacking to algebraic structures.

As illustrated in the figure above, the architecture can be decomposed into a sequence of categorical operations:

  1. Positional Encodings: Modeled as actions of a positional monoid $M$ on the embedding space.
  2. Self-Attention: Represented as a parametric endofunctor involving $Q, K,$ and $V$ projections.
  3. Stacked Attention: Formalized as a free monad construction, providing a universal justification for deep architectures.

The Universality of Stacking

One of the paper's most significant insights is the formalization of "depth." While stacking multiple identical layers is a common heuristic in deep learning, this paper proves that it is mathematically equivalent to constructing a free monad on the attention endofunctor. This suggests that stacking is not merely an empirical trick but the "most general" or "universal" way to extend a single attention operation into a multi-layered system.

Furthermore, this framework bridges the gap with mechanistic interpretability. The "QK" and "OV" circuits identified in prior research (such as Anthropic's work) are shown to be exact compositions of parametric morphisms within this categorical structure. This alignment suggests that the "circuits" found by researchers are inherent algebraic properties of the architecture's formal definition.

Positional Encodings and Equivariance

The paper also addresses how Transformers handle sequence order. By default, the linear portion of self-attention is naturally equivariant to permutations—meaning it treats the input as an unordered set. The paper formally proves this permutation equivariance (Theorem 5.1) through the lens of group actions.

To introduce order, positional encodings are treated as affine monoid actions. While sinusoidal encodings are not strictly additive in the simplest sense, the paper argues they satisfy a universal property among injective mappings, allowing the model to distinguish positions while maintaining the necessary algebraic structure for the attention mechanism to operate.

Formal Results

The paper's findings are established through several key theorems rather than empirical benchmarks:

  • Theorem 3.1: Confirms that a single-head self-attention mechanism constitutes a valid parametric 1-morphism in $Para(Vect)$.
  • Theorem 3.2: Demonstrates that the colimit of iterated attention layers forms a free monad, providing a principled foundation for model depth.
  • Theorem 5.1: Provides a formal proof of permutation equivariance for the linear components of the mechanism.

Limitations and Future Directions

While providing a robust framework for the linear components of Transformers, the current work explicitly excludes non-linearities such as Softmax, ReLU, and Layer Normalization. Future research aims to extend this framework to more complex categories, such as smooth manifolds or differential categories, to incorporate these non-linear operations and the process of backpropagation. Additionally, exploring "higher-level 2-monads" may clarify the complex interplay between attention heads and MLP layers in standard Transformer blocks.

Summary updated

Abstract

Self-attention mechanisms have revolutionised deep learning architectures, yet their core mathematical structures remain incompletely understood. In this work, we develop a category-theoretic framework focusing on the linear components of self-attention. Specifically, we show that the query, key, and value maps naturally define a parametric 1-morphism in the 2-category $\mathbf{Para(Vect)}$. On the underlying 1-category $\mathbf{Vect}$, these maps induce an endofunctor whose iterated composition precisely models multi-layer attention. We further prove that stacking multiple self-attention layers corresponds to constructing the free monad on this endofunctor. For positional encodings, we demonstrate that strictly additive embeddings correspond to monoid actions in an affine sense, while standard sinusoidal encodings, though not additive, retain a universal property among injective (faithful) position-preserving maps. We also establish that the linear portions of self-attention exhibit natural equivariance to permutations of input tokens, and show how the "circuits" identified in mechanistic interpretability can be interpreted as compositions of parametric 1-morphisms. This categorical perspective unifies geometric, algebraic, and interpretability-based approaches to transformer analysis, making explicit the underlying structures of attention. We restrict to linear maps throughout, deferring the treatment of nonlinearities such as softmax and layer normalisation, which require more advanced categorical constructions. Our results build on and extend recent work on category-theoretic foundations for deep learning, offering deeper insights into the algebraic structure of attention mechanisms.
BibTeX
@misc{oneill2025self,
  author = {Charles O'Neill},
  title = {{Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures}},
  year = {2025},
  eprint = {2501.02931},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.02931},
}
Figure