Tensors—multi-dimensional arrays—serve as the foundational mathematical structure for modern machine learning, quantum computing, and data analysis. Yet, a persistent gap remains between abstract tensor calculus and the practical software and hardware ecosystems used to execute them. While mathematics treats tensors natively across arbitrary dimensions, our computational infrastructure remains fundamentally matrix-centric (2D).
This mismatch forces developers to write fragmented, error-prone code and prevents compilers from optimizing high-dimensional operations. To bridge this divide, a major research direction focuses on establishing unified index-based notations—primarily derived from Einstein summation notation—as a universal abstraction layer for tensor computations.
The Matrix-Centric Bottleneck
When software libraries handle high-dimensional arrays, they typically reduce them to flat chunks of memory and manipulate them using sequences of low-level operations like reshape, transpose, permute, and squeeze. As highlighted by Alex Rogozhnikov in the design of Einops: Clear and Reliable Tensor Manipulations with Einstein-like Notation, this imperative approach lacks self-documentation. Code quickly becomes cryptic, and accompanying comments detailing tensor shapes frequently become outdated, making debugging exceptionally difficult.
Beyond code readability, this fragmentation cripples software optimization. In a report summarizing a Dagstuhl Seminar, Bientinesi et al. point out that unlike the mature ecosystem for matrix computations, tensor computations lack foundational, low-level building blocks analogous to BLAS (Basic Linear Algebra Subprograms). Without a unified intermediate representation, different scientific communities end up building isolated, uncoordinated libraries. This replication of effort prevents general-purpose compilers from applying holistic optimizations to high-dimensional data structures.
The Intuition of Declarative Indexing
The core insight behind solving this fragmentation is to shift from imperative memory manipulation to declarative index notation. Instead of telling the computer how to move bytes around in memory, a declarative language describes the structural relationship between the input and output dimensions.
Einstein notation provides an elegant framework for this. In its traditional form, it simplifies expressions by automatically summing over repeated indices (e.g., multiplying matrix $A_{ij}$ by matrix $B_{jk}$ to produce $C_{ik}$ is written simply as $i j, j k \rightarrow i k$).
By extending this concept, modern frameworks turn Einstein notation into a comprehensive user interface and compiler target:
- Explicit Dimension Tracking: By assigning explicit names or characters to dimensions (e.g.,
b h w cfor batch, height, width, channels), the code explicitly documents the tensor's geometry. - Framework Agnosticism: Because the notation describes structural transformations rather than execution steps, the same string pattern can be compiled to run on PyTorch, NumPy, TensorFlow, or custom hardware backends.
- Primitive Unification: Rather than relying on dozens of specialized API functions for broadcasting, reduction, and indexing, operations are modeled as vectorized applications of elementary functions over named axes.
For instance, Fervers et al. introduced It's All Just Vectorization: einx, a Universal Notation for Tensor Operations to demonstrate that even complex operations like indexing and concatenation can be unified under a single vectorization logic. Similarly, in functional programming, Jansson et al. developed domain-specific languages that leverage index-based notation to preserve the rich algebraic properties of tensors, preventing them from being degraded into simple, opaque multi-dimensional arrays.
Imperative (Fragmented APIs):
x = x.transpose(1, 2)
x = x.reshape(batch, -1, channels)
# What does this actually do? The mathematical intent is obscured.
Declarative (Unified Index Notation):
x = einops.rearrange(x, 'b c h w -> b (h w) c')
# The structural transformation is immediately clear and self-documenting.
Bridging Advanced Mathematics and Computational Graphs
Standardizing on Einstein-like notation does more than clean up code; it unlocks massive computational speedups by making advanced mathematics compatible with automated compiler optimization.
In fields like differential geometry and physics, higher-order derivatives (such as Jacobians and Hessians) are traditionally formulated using complex Ricci notation. However, this notation is incompatible with standard deep learning software ecosystems. Laue et al. demonstrated that by abandoning Ricci notation in favor of standard Einstein notation, complex tensor calculus can be integrated directly into standard computational graphs.
When the underlying operations are expressed via a unified index notation, the compiler can analyze the global algebraic structure of the entire computation. This allows it to eliminate redundant intermediate representations and optimize execution paths, yielding speedups of up to three orders of magnitude for higher-order derivative calculations.
The Hardware Execution Challenge
While unified index notations provide an elegant frontend abstraction, executing these operations efficiently on physical hardware remains an active area of research.
Modern hardware accelerators, such as GPUs, are heavily optimized for standard 2D matrix multiplications (GEMM operations). When a high-order tensor operation is expressed declaratively, the underlying execution engine must lower it into a sequence of hardware-compatible tasks. As observed by Su et al., theoretically efficient tensorial operations often fail to achieve practical speedups on current hardware because accelerators lack specialized kernels designed to exploit the multi-linear parallelism of novel tensor algebra. This performance gap is echoed by Zhou et al., who note that hardware limitations create a persistent bottleneck for non-standard mathematical structures.
To drive the development of next-generation execution engines, researchers require rigorous evaluation tools. Blacher et al. addressed this by introducing the Einsum Benchmark: Enabling the Development of Next-Generation Tensor Execution Engines. By compiling a diverse suite of large-scale Einstein summation problems across quantum computing and probabilistic AI, this benchmark exposes systemic inefficiencies and memory overheads in current engines. Standardized benchmarks like this provide the necessary foundation to optimize hardware backends, ensuring that the elegance of unified tensor abstractions is matched by raw execution performance.
Key Foundational Work
Preliminary versions of the MATLAB tensor classes for fast algorithm prototyping. (2004) — contributed 3 concept elements to this direction.
A preliminary report on the development of MATLAB tensor classes for fast algorithm prototyping. (2004) — contributed 1 concept elements to this direction.
Matching Constraints and the Joint Image (1995) — contributed 1 concept elements to this direction.