An Introduction to Transformers

Richard E. Turner

Abstract: The transformer is a neural network component that can be used to learn useful representations of sequences or sets of data-points. The transformer has driven recent advances in natural language processing, computer vision, and spatio-temporal modelling. There are many introductions to transformers, but most do not contain precise mathematical descriptions of the architecture and the intuitions behind the design choices are often also missing. Moreover, as research takes a winding path, the explanations for the components of the transformer can be idiosyncratic. In this note we aim for a mathematically precise, intuitive, and clean description of the transformer architecture. We will not discuss training as this is rather standard. We assume that the reader is familiar with fundamental topics in machine learning including multi-layer perceptrons, linear transformations, softmax functions and basic probability.

Introduction: Beyond Idiosyncratic Explanations

While the transformer has become the backbone of modern AI—powering everything from Large Language Models (LLMs) to Vision Transformers (ViTs)—many existing pedagogical resources lack mathematical precision. Descriptions often reflect the historical, "winding path" of research, leading to idiosyncratic explanations of components like queries, keys, and values. This note aims to strip away these historical artifacts to provide a "clean" mathematical framework for understanding the transformer as an interleaving process of cross-sequence and cross-feature transformations.

Data Representation

The transformer operates on a set of $N$ data points, each represented by a $D$-dimensional feature vector. Collectively, these are represented as a matrix $X \in \mathbb{R}^{D \times N}$. In this convention, each column corresponds to a "token" (e.g., a word or an image patch), and each row corresponds to a specific feature dimension.

The input data representation as a matrix $X^{(0)}$ with $D$ features (rows) and $N$ tokens (columns).

The Transformer Block: A Two-Stage Process

A transformer model consists of a sequence of identical blocks that iteratively refine the data representation. Each block is composed of two distinct stages that alternate between processing along the sequence dimension and the feature dimension.

Stage 1: Cross-Sequence Processing (Self-Attention)

The first stage aggregates information across the sequence. This is the "horizontal" operation across the matrix $X$. The goal is to allow each token to update its representation based on its relationship with other tokens in the set.

Mathematically, this is achieved through an attention matrix $A \in \mathbb{R}^{N \times N}$, where the entry $A_{n,n'}$ represents the "affinity" or similarity between token $n$ and token $n'$. The output of this stage, $Y^{(m)}$, is a linear combination of the input features weighted by these affinities.

Cross-sequence processing: The output $Y^{(m)}$ is formed by multiplying the input $X^{(m-1)}$ by the attention matrix $A^{(m)}$.

To capture complex, asymmetric relationships (e.g., the word "caulking" should attend strongly to "tool," but "tool" might not attend to "caulking"), the transformer uses a Query ($q$) and Key ($k$) mechanism. Multi-Head Self-Attention (MHSA) further extends this by using $H$ parallel attention heads, allowing the model to attend to different types of relationships simultaneously.

Stage 2: Cross-Feature Processing (MLP)

The second stage refines the features of each token independently. This is a "vertical" operation down the columns of the matrix. A Multi-Layer Perceptron (MLP) is applied to each token's feature vector. Crucially, the same MLP parameters are shared across all $N$ tokens, ensuring the model remains permutation-equivariant (it treats the tokens as a set).

Key Architectural Insights

  • Transformers as Set Processors: By default, transformers are permutation-invariant; they do not inherently "know" the order of tokens. Sequence order is only introduced through "Position Encoding"—adding specific vectors to the input matrix $X^{(0)}$ to break symmetry.
  • TokenNorm vs. LayerNorm: To stabilize training in deep networks, transformers use normalization. The author suggests the term "TokenNorm" to describe the practice of normalizing each token's feature vector independently, which avoids the ambiguity of the term "LayerNorm" across different domains (like CNNs vs. Transformers).
  • Interleaving Motif: The core logic of the transformer—alternating between aggregating information across nodes (Stage 1) and processing features within nodes (Stage 2)—is a powerful motif also found in Graph Neural Networks (GNNs) and ResNets.

Limitations and Complexity

The primary computational bottleneck of the transformer is the $O(N^2)$ complexity of the attention matrix. As the sequence length $N$ grows, the memory and compute requirements for the $N \times N$ matrix $A$ become prohibitive. Additionally, despite their representational power, transformers can be sensitive to training hyperparameters, often requiring specific optimizers like Adam and careful learning rate scheduling to achieve stability.

Summary updated

Abstract

The transformer is a neural network component that can be used to learn useful representations of sequences or sets of data-points. The transformer has driven recent advances in natural language processing, computer vision, and spatio-temporal modelling. There are many introductions to transformers, but most do not contain precise mathematical descriptions of the architecture and the intuitions behind the design choices are often also missing. Moreover, as research takes a winding path, the explanations for the components of the transformer can be idiosyncratic. In this note we aim for a mathematically precise, intuitive, and clean description of the transformer architecture. We will not discuss training as this is rather standard. We assume that the reader is familiar with fundamental topics in machine learning including multi-layer perceptrons, linear transformations, softmax functions and basic probability.
BibTeX
@misc{turner2023introduction,
  author = {Richard E. Turner},
  title = {{An Introduction to Transformers}},
  year = {2023},
  eprint = {2304.10557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.10557},
}
Figure